CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning

## Abstract

Abstract Large language models (LLMs) have made significant progress in natural language understanding and generation, driven by scalable pretraining and advanced finetuning. However, enhancing reasoning abilities in LLMs, particularly via reinforcement learning from human feedback (RLHF), remains challenging due to the scarcity of high-quality preference data, which is labor-intensive to annotate and crucial for reward model (RM) finetuning. To alleviate this issue, we introduce CodePMP, a scalable preference model pretraining (PMP) pipeline that utilizes a large corpus of synthesized code-preference pairs from publicly available high-quality source code. CodePMP improves RM finetuning efficiency by pretraining preference models on large-scale synthesized code-preference pairs. We evaluate CodePMP on mathematical reasoning tasks (GSM8K, MATH) and logical reasoning tasks (ReClor, LogiQA2.0), consistently showing significant improvements in reasoning performance of LLMs and highlighting the importance of scalable preference model pretraining for efficient reward modeling.

## 1 Introduction

Figure: Figure 1: Compared to directly finetuning reward models, CodePMP significantly improves the sample efficiency and capability of reward models, which in turn boosts the generator’s(MetaMath-Mistral-7B) reasoning performance (Best-of-N accuracy) across both mathematical reasoning tasks (GSM8K and MATH) and logical reasoning tasks (ReClor and LogiQA2.0).
Refer to caption: extracted/5895834/images/codepmp_highlight.png

Large language models (LLMs) have made remarkable progress in natural language understanding and generation, benefiting from scalable pretraining and finetuning techniques like supervised finetuning (SFT) (Wang et al., ; ) and Reinforcement Learning from Human Feedback (RLHF) (Bai et al., ; Lightman et al., ; Bai et al., ; Gulcehre et al., ; Schulman et al., ; Rafailov et al., ). However, enhancing LLMs’ reasoning abilities, particularly in complex logical and mathematical tasks, remains a significant challenge (Wang et al., ; Zhang et al., ). Although RLHF is effective, it relies heavily on high-quality preference data, which is costly and labor-intensive to annotate (Cobbe et al., ; Zheng et al., ). This limitation impedes the scalability of reward model (RM) finetuning, which is essential for guiding LLMs toward optimal outputs.

To alleviate this issue, prior works like Anthropic’s Preference Model Pretraining (PMP) (Askell et al., ) have proposed improving reward modeling data efficiency by pretraining preference models on large-scale preference data from public sources like Reddit and Wikipedia, followed by an efficient finetuning on limited high-quality human-annotated data. However, this approach is less effective for reasoning tasks due to the scarcity of reasoning preference pairs available online. Compared to other tasks, manually annotating preference data for reasoning is inherently more challenging and difficult to scale (Zhang et al., ; Zhou et al., ), highlighting the urgent need for a scalable PMP approach for reasoning tasks.

Figure: Figure 2: An illustration of the CodePMP process. First, raw code collected from GitHub is cleaned and summarized into code prompts (descriptions). Then, for each code prompt, a weak CodeLLM generates a rejected response, while a stronger CodeLLM produces a chosen response. Finally, these <chosen, rejected> pairs, accumulated in the millions, form the pretraining dataset for the preference model. This pretraining process improves not only sample efficiency but also the performance for downstream reasoning reward model finetuning.
Refer to caption: extracted/5895834/images/codepmp_arch.png

In this paper, we propose CodePMP, a scalable preference model pretraining pipeline that enhances LLM reasoning abilities using synthesized preference pairs derived from high-quality, publicly available source code. Code, with its inherently logical and structured nature, provides rich data suitable for reasoning tasks. Recent works (Zhang et al., ; Aryabumi et al., ) also show a strong correlation between code training and reasoning improvements in LLMs. By leveraging the huge amount and diverse coverage of source code available on platforms like GitHub, CodePMP offers a scalable solution for pretraining preference models, thereby improving RM finetuning efficiency and enhancing LLMs reasoning performance.

Specifically, CodePMP generates preference pairs by synthesizing chosen and rejected code responses for a given code-related prompt or description using CodeLLMs. A strong CodeLLM produces higher-quality (chosen) responses, while a weaker model generates suboptimal or even low-quality (rejected) responses. These <chosen, rejected> pairs, accumulated in the millions, form a large-scale synthesized preference dataset. This dataset is then used to pretrain the preference model with pairwise ranking objectives (Cobbe et al., ; Charniak & Johnson, ), providing an good initialization for further fine-tuning the reward models.

We evaluate CodePMP on widely studied reasoning tasks, including mathematical reasoning tasks such as GSM8K (Cobbe et al., ) and MATH (Hendrycks et al., ), as well as logical reasoning tasks like ReClor (Yu et al., ) and LogiQA2.0 (Liu et al., ). Our experiments demonstrate that CodePMP significantly improves RM fine-tuning accuracy and Best-of-N performance in reasoning tasks, outperforming direct RM fine-tuning, as highlighted in Figure 1. Moreover, additional experimental results reveals that RMs initialized with CodePMP exhibit greater robustness across different tasks. These results indicate that code-derived preference data provides a scalable, cost-effective solution for enhancing LLM reasoning capabilities while reducing reliance on extensive preference annotation, achieving more effective reward modeling for reasoning tasks.

In summary, our main contributions are:

- 1.
We introduce CodePMP, a scalable method that uses code-derived preference pairs to pretrain preference models, improving sample efficiency and robustness for downstream RM finetuning.
- 2.
We validate that CodePMP significantly improves performance on reasoning tasks, demonstrating that a scalable PMP process positively impacts LLM reasoning abilities.
- 3.
We provide a detailed analysis of key design elements in CodePMP, offering valuable insights for future research in related areas.

## 2 Preliminaries

##### Language Modeling

Language modeling (LM) is a fundamental task in natural language processing, aimed at modeling sequences of language. This is typically achieved through Causal Language Models (Causal LM), where the model is trained to maximize the likelihood of predicting the next word $w_{t}$ given the preceding words $w_{1},w_{2},\dots,w_{t-1}$. The training process minimizes the negative log-likelihood of the predicted word sequence:

$$ $\mathcal{L}_{\text{LM}}=-\sum_{t=1}^{T}\log P(w_{t}|w_{1},w_{2},\dots,w_{t-1})$ $$

This loss function $\mathcal{L}_{\text{LM}}$ encourages the model to capture the underlying patterns in the data. Transformer architectures (Vaswani, ) have become the standard for Causal LM due to their ability to handle long-range dependencies effectively.

##### Reward Modeling

Reward modeling (RM) is crucial in reinforcement learning from human feedback (RLHF), providing a scalar reward signal that guides the learning process based on output quality. The reward model $R_{\theta}$ predicts the quality of an output $y$ given a context $x$ as $s=R_{\theta}(x,y)$, where $s$ is the scalar reward score. In preference modeling, RM predicts the relative quality of outputs by comparing pairs. A common method is the Pairwise Ranking Loss, where the model assigns higher scores to preferred (chosen) outputs:

$$ $\mathcal{L}_{\text{RM}}=-\log\left(\sigma(s_{\text{chosen}}-s_{\text{rejected} })\right)$ $$

, where $s_{\text{chosen}}=R_{\theta}(x,y_{\text{chosen}})$ and $s_{\text{rejected}}=R_{\theta}(x,y_{\text{rejected}})$, and $\sigma(\cdot)$ is the sigmoid function.

##### Best-of-N Sampling

Best-of-N (BoN) sampling improves LLM reasoning (Cobbe et al., ; Lightman et al., ). In this approach, $N$ candidate solutions $\{y_{1},y_{2},\dots,y_{N}\}$ are generated by sampling from the LLM’s output distribution for a given problem. A reward model scores each candidate and selects the highest-scoring one as the final answer:

$$ $\hat{y}=\arg\max_{y_{i}\in\{y_{1},y_{2},\dots,y_{N}\}}R_{\theta}(x,y_{i})$ $$

, where $R_{\theta}(x,y_{i})$ represents the reward score for each candidate $y_{i}$. This technique is especially effective in tasks like mathematical problem-solving and logical inference, where selecting the most plausible solution from a diverse set of outputs improves overall accuracy (Wang et al., ).

## 3 Code Preference Model Pretraining

### 3.1 Model Design

Code Preference Model Pretraining (CodePMP) is designed to enhance the sample efficiency of reward models, particularly for reasoning tasks where high-quality preference data is scarce. Traditionally, reward models are finetuned on small, curated datasets, which limits their effectiveness in complex tasks like mathematical reasoning or logical deduction. CodePMP mitigates this limitation by introducing a pretraining phase between basic language model pretraining and finetuning on domain-specific reasoning datasets. This phase leverages a large, diverse dataset of code-preference pairs, enabling the model to learn generalizable patterns and ranking strategies.

CodePMP training involves two key components: Reward Modeling (RM) and Language Modeling (LM). In RM, the model is trained on code-preference pairs, learning to assign higher scores to the chosen code through a pairwise ranking loss. In LM, only the chosen code is used for autoregressive training to maintain the model’s general capabilities. The overall loss is a combination of the RM and LM losses, ensuring the model enhances its ranking ability without sacrificing general language modeling performance: $\mathcal{L}_{\text{PMP}}=\mathcal{L}_{\text{rank}}+\mathcal{L}_{\text{LM}}$.

### 3.2 Data Construction

To enable scalable preference model pretraining, we construct a dataset sourced from GitHub, which includes a diverse range of repositories and associated metadata. The dataset consists of two primary components:
Repository Data comprises over 1.3 billion code files from GitHub repositories, while GitHub Metadata includes information such as commit histories, discussions, pull requests, and issues.

The CodePMP dataset is constructed through a systematic process. First, raw source code is processed by a description summarizer, typically an instruction-tuned CodeLLM, to generate prompts that describe the functionality of the code.

These prompts are then used by two CodeLLMs of different capabilities to generate code snippets:

- •
Chosen response: Generated by a more advanced CodeLLM (e.g., 6.7B parameters).
- •
Rejected response: Generated by a less capable CodeLLM (e.g., 1.3B parameters).

This process yields pairs of code responses—one chosen and one rejected—which are used for preference modeling. This scalable approach significantly enhances pretraining efficiency, improving performance on downstream tasks.

The steps of the CodePMP methodology are outlined systematically in Algorithm 1.

Figure: Algorithm 1 Code Preference Model Pretraining (CodePMP)

## 4 Experimental

In this section, we first outline the experimental setup, followed by the experimental results, highlighting that CodePMP is a highly scalable method.

### 4.1 Experimental Settings

#### 4.1.1 CodePMP Settings

##### Data Construction

We generate code preference pairs following Algorithm 1, using the deepseek-coder-6.7b-instruct model as the strong CodeLLM to generate chosen responses and the deepseek-coder-1.3b-instruct model as the weak CodeLLM to generate rejected responses. The constructed CodePMP dataset includes 28 million files and 19 billion tokens. The diverse datasets provide sufficiently broad prompt coverage for preference model pretraining, which is conducive to the generalization of preference models in reasoning tasks. In addition, the average lengths of the chosen and rejected responses are similar, ensuring that response length does not bias the CodePMP learning process. Details are provided in Appendix C and Table 4.

##### CodePMP Training

By default, we initialize the preference models with the publicly available Qwen models (Yang et al., ), using different model sizes, specifically Qwen2-1.5B and Qwen2-7B. Detailed hyperparameters for CodePMP training are provided in Appendix B.

#### 4.1.2 Reasoning Finetuning Settings

We validate CodePMP on reward models across two reasoning task types: mathematical and logical reasoning. The reward model is finetuned on corresponding preference datasets for each task. For mathematical reasoning, we use the MathShepherd-pair dataset, derived from MathShepherd (Wang et al., ), and evaluate the model on a holdout test set to assess RM accuracy.

Similarly, for logical reasoning, we use the ReClor-pair and LogiQA2.0-pair datasets, derived from ReClor (Yu et al., ) and LogiQA2.0 (Liu et al., ), respectively. We train reward models on these datasets, with holdout test sets used to evaluate model accuracy. Dataset construction and finetuning hyperparameters are provided in Appendix D and  B.

#### 4.1.3 Evaluation Settings

Following (Zhang et al., ), we adopt two evaluation metrics:

##### RM Accuracy

This metric measures the accuracy of the reward model in distinguishing chosen from rejected solutions on the holdout test sets. It provides insight into the model’s ability to classify individual sequences.

##### Best-of-N (BoN) Accuracy

This metric evaluates the proportion of correct solutions selected by the finetuned RM from $N$ candidate responses. It assesses the model’s group-wise ranking performance, focusing on its ability to select the correct answer from a set of candidates. We use MetaMath-Mistral-7B (Yu et al., ) as the generator for this evaluation.

For mathematical reasoning, we use the GSM8K (Cobbe et al., ) and MATH (Hendrycks et al., ) test sets. For logical reasoning, we evaluate on the ReClor (Yu et al., ) and LogiQA2.0 (Liu et al., ) test sets. Further details can be found in Appendix D.

Note that logical reasoning questions typically involve a paragraph followed by statements to be judged true or false, making Best-of-N evaluation challenging. Therefore, we use multiple-choice accuracy, where the reward model ranks four manually annotated options and selects the best one. This metric is equivalent to Best-of-4, and thus, for logical reasoning tasks, multiple-choice accuracy and Best-of-N are used interchangeably.

### 4.2 Experimental Results

#### 4.2.1 RM Accuracy Results

We first compare RM accuracy on the holdout test set with and without CodePMP initialization. As shown in Table 1, RM finetuned with CodePMP initialization achieves higher accuracy on both 1.5B and 7B models across mathematical and logical reasoning tasks, demonstrating that CodePMP enhances the model’s ability to differentiate correct from incorrect reasoning. Moreover, CodePMP exhibits strong generalization, yielding significant improvements across different reasoning tasks.

**Table 1: Comparison of RM accuracies: reward models finetuned with CodePMP initialization achieve higher accuracies on the reasoning holdout test sets, demonstrating an improved ability to distinguish chosen responses from rejected ones.**
| Model | PMP | MathShepherd-pair | Reclor-pair | LogiQA2.0-pair |
| --- | --- | --- | --- | --- |
| 1.5B | ✗ | 0.7226 | 0.758 | 0.7538 |
| ✓ | 0.8186 | 0.794 | 0.7774 |  |
| 7B | ✗ | 0.8777 | 0.862 | 0.8263 |
| ✓ | 0.9274 | 0.874 | 0.8441 |  |

Figure: (a) BoN accuracies on mathematical reasoning.
Refer to caption: extracted/5895834/images/codepmp_main_exp.png

Figure: (a) GSM8K / 1.5B
Refer to caption: extracted/5895834/images/rm_bon_gsm8k_1.png

#### 4.2.2 BoN Accuracy Results

We evaluate BoN accuracy across reasoning tasks with and without CodePMP initialization. As shown in Figure 8(a) and Table 8(b), RM finetuned with CodePMP initialization consistently achieves higher BoN accuracy across both mathematical and logical reasoning tasks for 1.5B and 7B models. This highlights CodePMP’s effectiveness in improving RM’s group-wise ranking performance.

Across different values of N, RM models initialized with CodePMP maintain their lead, showing robust improvement even as N increases to 256. In contrast, RM without CodePMP shows a sharp decline in accuracy as N increases, underscoring the stability CodePMP provides, likely due to the diverse code-preference pairs used during training.

For logical reasoning, the performance gap between CodePMP and non-PMP models is smaller, as $N$ is limited to 4, while in mathematical reasoning, $N$ reaches 256. This suggests that increasing $N$ further in logical reasoning could amplify the advantages of CodePMP in future evaluations.

#### 4.2.3 RM Sample Efficiency Comparison

One key advantage of CodePMP is its ability to improve the sample efficiency of RM finetuning. To assess this, we conduct experiments with progressively larger sample sizes for RM finetuning. As indicated by (Kaplan et al., ), optimal results are achieved when the learning rate scheduler completes its decay at the end of training. Therefore, rather than evaluating intermediate checkpoints, we retrain models with varying sample sizes for optimal results. Figure 4 and  10 (Appendix E) show that as the sample size increases, RMs with CodePMP initialization consistently outperforms others in both BoN and RM accuracy. Notably, RMs finetuned with CodePMP initialization using just 0.5k samples surpasses RMs finetuned without CodePMP initialization using 40k samples on mathematical tasks, demonstrating CodePMP’s significant advantage in sample efficiency. However, as sample size increases, this advantage diminishes slightly, suggesting that with much larger datasets, CodePMP’s benefit may become less pronounced, but the cost of manual labeling remains a key consideration.

#### 4.2.4 The Importance of Scalable PMP

A key benefit of using code data for PMP is the vast availability of publicly accessible, high-quality code-preference pairs, ensuring diversity. To validate scalability, we vary the number of training pairs for CodePMP and retrain models with different amounts of data. As shown in Figure 5, overall, increasing the number of code-preference pairs consistently improves BoN accuracy in both mathematical and logical reasoning tasks across model sizes, with no sign of diminishing returns. This indicates that further scaling the code-preference data would likely yield additional performance gains, underscoring the importance of building a scalable PMP pipeline.

Figure: (a) GSM8K / 1.5B
Refer to caption: extracted/5895834/images/pmp_scale_gsm8k_1.png

## 5 Ablation Studies

In this section, we present a detailed analysis of CodePMP design. Unless otherwise stated, all experiments used the 1B model due to resource limitations and present the results of mathematical reasoning due to page limitation. Results of logical reasoning refers to Appendix E.2.

### 5.1 Impact of Pair Construction

##### Model-Generated Pairs Comparison

We compare various pair construction methods generated by different models. In Figure 6(a), the samples before the “&” are positive, and those after are negative. “Source Code” refers to the original code snippet, while “1.3B-Des-Clip” indicates that 10% of the code description is removed before being input into a 1.3B CodeLLM to generate a rejected response. The green lines represent CodePMP’s choice. Results show that pairing positive samples from the 7B model with negative samples from the 1.5B model consistently delivers the best performance across all test sets. Given that code execution can generate reliable outputs, future work will explore incorporating execution feedback to create more accurate preference pairs.

##### Web-Crawled vs GitHub-Sourced Pairs

We also compare GitHub-sourced code with web-crawled data( Askell et al. ()) from platforms such as StackExchange and Reddit. As shown in Figure 7(a), GitHub-sourced pairs (“Source Code”) consistently outperform those from web platforms (“Webpage”), particularly as the number of solutions (N) increases. Moreover, the performance improvement of GitHub-sourced pairs shows no sign of plateauing, highlighting the importance of diverse, high-quality source code in building a scalable PMP pipeline.

Figure: (a) Different construction methods.
Refer to caption: extracted/5895834/images/res_cmp_pair_construction.png

Figure: (a) With or without EOC token.
Refer to caption: extracted/5895834/images/res_res_eoc.png

### 5.2 Impact of EOC Token

Experiments by( Askell et al. ()) show that adding an end-of-context (EOC) token to each sequence significantly improves overall performance. To explore its impact in the context of CodePMP, we compared performance with and without the EOC token. As shown in Figure 7(a), the EOC setting (“w/ EOC”) consistently underperform the setting without EOC tokens (“w/o EOC”) across different test tasks, which is opposed to ( Askell et al. ()) We attribute this discrepancy to the different model, data and evaluation settings.

### 5.3 Impact of Learning Rate Schedulers

In the CodePMP experiments, we use the warmup-stable-decay (WSD) learning rate scheduler( Hu et al. ()), which can effectively reduce the time required for scaling related experiments. Previous studies mainly employ a learning rate schedule with linear warmup followed by cosine decay, known as warmup-cosine decay (WCD). We compare the performance of WSD and WCD, as shown in Table 7(b), both schedulers yield similar results. Thus, to improve computational efficiency, we adopt the WSD scheduler for all experiments.

Figure: (a) BoN accuracies on mathematical reasoning.
Refer to caption: extracted/5895834/images/codepmp_main_exp_gemma.png

**Table 2: Performance gains on coding RM and general RM (RMBench) evaluations show that CodePMP not only improves reasoning tasks but also generalizes well across various tasks.**
|  |  |  | RMBench |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | PMP | Coding | Summary | Chat | Chat Hard | Safety | Reasoning |
| 1.5B | ✗ | 0.6841 | 0.4154 | 0.4804 | 0.5351 | 0.3665 | 0.2751 |
| ✓ | 0.758 | 0.6126 | 0.9050 | 0.4364 | 0.3698 | 0.6041 |  |
| 7B | ✗ | 0.6912 | 0.5839 | 0.4972 | 0.5022 | 0.5240 | 0.6804 |
| ✓ | 0.7619 | 0.7668 | 0.9413 | 0.5373 | 0.4906 | 0.9116 |  |

### 5.4 Validating CodePMP on Other LLMs

To further evaluate the generalizability of CodePMP, we validate its performance on the widely adopted Gemma-2B model (Team et al., ). As illustrated in Figure 8, the application of CodePMP results in significant performance gains in both mathemantical reasoning and logical reasoning evaluations. This not only underscores the robustness of CodePMP but also demonstrates its broad applicability in improving sample efficiency and overall performance across diverse LLM architectures.

### 5.5 Performance on Coding and General RM Benchmarks

We evaluate CodePMP on both code-specific and general reward modeling benchmarks. The CodeUltraFeedback_binarized test set serves as an in-domain evaluation, while RMBench provides an out-of-domain assessment. As shown in Table 2, models finetuned with CodePMP initialization consistently outperform those without CodePMP across various model sizes. These results demonstrate that CodePMP not only enhances performance in reasoning tasks but also generalizes well across a range of RM benchmarks.

## 6 Related Works

##### Reward Modeling

In the context of RLHF, reward models (RMs) have traditionally employed ranking models like Bradley-Terry and Plackett-Luce to represent human preferences (Bradley & Terry, ; Plackett, ; Cobbe et al., ; Saunders et al., ; Lightman et al., ; Wang et al., ; Uesato et al., ; Luo et al., ; Yu et al., ; Stiennon et al., ; Nakano et al., ). More recently, probability-based approaches (Zhao et al., ; Jiang et al., ) have emerged, offering more precise predictions. Additionally, models such as Critique-out-Loud (Ankner et al., ) enhance RMs by integrating natural language feedback. Generative reward models (GRMs) (Yang et al., ) further boost sample efficiency. Preference Modeling Pretraining (PMP) (Askell et al., ) introduces a novel pretraining phase, utilizing large-scale pairwise ranking data to enhance RM performance. Despite these advancements, many methods are hindered by the reliance on expensive manual annotations or limited datasets, constraining scalability. CodePMP mitigates this by automating preference data generation from code, significantly improving RM sample efficiency and reducing dependency on manual data collection.

##### Code Training

The inclusion of code in LLM pretraining has led to marked improvements in tasks such as commonsense reasoning (Madaan et al., ) and mathematical problem-solving (Liang et al., ; Shao et al., ; Yang et al., ). Furthermore, code enhances general reasoning capabilities (Muennighoff et al., ; Fu & Khot, ; Ma et al., ). Recent studies (Dong et al., ; Ma et al., ) indicate that incorporating code during supervised finetuning strengthens LLMs, particularly in complex decision-making tasks. CodePMP takes a pioneering approach by utilizing scalable, synthetically generated code preference pairs, reducing the dependence on manual annotation (Dubey et al., ; Gemini-Team et al., ; Groeneveld et al., ; Bi et al., ). This methodology enhances sample efficiency and scalability in reasoning-intensive tasks, presenting new opportunities for further improving LLM performance.

##### LLM Reasoning

Improving reasoning capabilities in LLMs remains a significant challenge, with various advanced methods being proposed. Chain of Thought (CoT) prompting (Wei et al., ; Fu et al., ) improves reasoning by generating intermediate steps, while CoT combined with supervised finetuning (SFT) further enhances performance (Cobbe et al., ; Liu et al., ; Yu et al., ). Other approaches focus on expanding inference time computation, such as problem decomposition (Zhou et al., ), search-based methods like MCTS (Xu, ), and using LLMs as verifiers (Huang et al., ; Luo et al., ). Reward models, including outcome-based (ORM) and process-based (PRM), have also shown success, with PRM delivering superior results (Lightman et al., ; Wang et al., ). Distinct from these approaches, CodePMP introduces a scalable preference model pretraining phase that can integrate seamlessly with all the aforementioned techniques.

## 7 Conclusion & Future Works

This paper introduces CodePMP, a scalable pretraining approach that leverages code-preference pairs to improve reasoning capabilities in large language models. Experimental results validate that CodePMP significantly improves sample efficiency and boosts performance on reasoning tasks.

For future work, we aim to extend CodePMP in two key directions. CodePrMP will focus on utilizing compiler and interpreter verifiability to provide low-cost process supervision signals. GenPMP will explore how to improve sample efficiency and the performance of generative reward models by integrating code data.

## Appendix A Hyperparameters

We outline key hyperparameters in this Table 3. In the tables, WSD refers to the warmup-stable-decay learning rate scheduler( (Hu et al., )), which has the benefit of reducing the time required for scaling law experiments.

**Table 3: Hyperparameters for CodePMP training, mathematical reasoning RM finetuning, and logical reasoning RM finetuning.**
|  | CodePMP |  | Mathematical RM | Logical RM |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HP | 1.5B | 7B |  | 1.5B | 7B | 1.5B | 7B |
| epoch | 1 | 1 |  | 1 | 1 | 1 | 1 |
| bs | 1024 | 1024 |  | 64 | 64 | 64 | 64 |
| lr | 3e-6 | 1e-6 |  | 1e-6 | 3e-7 | 1e-5 | 1e-5 |
| lr scheduler | WSD | WSD |  | WCD | WCD | WCD | WCD |
| warmup ratio | 0.03 | 0.03 |  | 0.03 | 0.03 | 0.25 | 0.25 |
| decay ratio | 0.1 | 0.1 |  | - | - | - | - |
| weight decay | 0.1 | 0.1 |  | 0 | 0 | 0 | 0 |
| max length | 1024 | 1024 |  | 1024 | 1024 | 1024 | 1024 |

## Appendix B Training Pipeline

Figure B presents an overview of the complete training pipeline.
The process begins with base language model (LM) pretraining on trillions of tokens from general text, followed by a preference model pretraining (PMP) phase using billions of tokens from code preference pairs. Finally, the model is finetuned on a smaller, more specialized dataset relevant to reasoning tasks, typically consisting of millions of tokens.

Figure: Figure 9: An overview of the complete training pipeline.
Refer to caption: extracted/5895834/images/codepmp_pipeline.png

## Appendix C CodePMP Dataset

As shown in Table 4, the constructed CodePMP dataset consists of a total of 28 million files, accounting for 19 billion tokens across various languages. The dataset is primarily composed of Python files, with 20 million files and 13.1 billion tokens, followed by Notebook files, contributing 3 million files and 2.1 billion tokens, and other programming languages with 5 million files and 3.8 billion tokens. This diverse dataset supports the pretraining phase, aiding the model in generalizing across multiple reasoning tasks.

The average length of chosen responses varies slightly between different languages. Python files exhibit an average response length of 170 tokens for chosen responses and 167 tokens for rejected responses, while Notebook files have slightly shorter average lengths of 158 and 155.5 tokens, respectively. Other languages show the highest average response lengths, with 213.2 tokens for chosen and 210 tokens for rejected responses. Overall, the average length of chosen responses across the dataset is 194.5 tokens, while the average length of rejected responses is 189.9 tokens, indicating minimal bias in the dataset based on response length.

**Table 4: Amount and average length statistics of CodePMP dataset.**
|  | Amount Statistics | Average Length |  |  |
| --- | --- | --- | --- | --- |
| Language | Files (M) | Tokens (B) | Chosen | Rejected |
| Python | 20 | 13.1 | 170.0 | 167.0 |
| Notebook | 3 | 2.1 | 158.0 | 155.5 |
| Other Languages | 5 | 3.8 | 213.2 | 210.0 |
| Total | 28 | 19.0 | 194.5 | 189.9 |

## Appendix D RM Finetuning Dataset

### D.1 Mathematical Reasoning

The RM finetuning for mathematical reasoning uses the MathShepherd dataset( (Wang et al., )), which contains 444k query-response samples, with some queries having multiple distinct responses. We divide the dataset into a 400k training set and a 44k test set. For RM finetuning, we construct preference pairs by selecting both correct and incorrect responses for the same query. To form the 4.3k test set, we combine one positive and negative sample for each query from the original test set.

We also create two training sets of different sizes: MathShepherd-preference-800k and MathShepherd-preference-40k. The 800k training set is built by combining multiple positive and negative samples for each query in the original training set, resulting in 800k samples. In contrast, the 40k training set randomly selects one positive-negative pair for each query, totaling 40k samples.

### D.2 Logical Reasoning

#### D.2.1 Reclor

Reclor is a human-annotated reading comprehension reasoning dataset, where each sample consists of a passage, a question, and multiple options. To create preference pairs, we combine the correct and incorrect options for the same question. This process results in a total of 14.5k preference pairs, with 14k pairs used for training and 1.5k for testing, forming the Reclor-preference dataset.

#### D.2.2 LogiQA2.0

For logical reasoning finetuning scaling analysis, we synthesize a preference dataset based on the logical reasoning dataset LogiQA2.0. LogiQA2.0 is a reading comprehension benchmark requiring discrete reasoning over passages, with 96k crowdsourced, adversarially created questions. To answer correctly, models must resolve references (which may point to multiple locations in the input) and perform discrete operations like addition, counting, or sorting.
We use four models (Qwen2-7B-Instruct(^1^11https://huggingface.co/Qwen/Qwen2-7B-Instruct), Qwen2-72B-Instruct(^2^22https://huggingface.co/Qwen/Qwen2-72B-Instruct), DeepSeek-V2-Chat(^3^33https://huggingface.co/deepseek-ai/DeepSeek-V2-Chat), and DeepSeek-Coder-V2-Instruct(^4^44https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct)) to sample queries from the LogiQA2.0 dataset multiple times, with sampling $topp=1$, and $topk\in\{0,0.01,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0\}$. Answer correctness is annotated using DeepSeek-7B-Math-Compare-Answer(^5^55https://huggingface.co/Tianqiao/DeepSeek-7B-Math-Compare-Answer). Correct and incorrect answers are combined to create preference pairs, resulting in 1,019k pairs, with 977k used for training and 42k for testing, forming the LogiQA2.0 preference dataset for RM scaling analysis.

### D.3 CodeUltraFeedback_binarized

CodeUltraFeedback_binarized(^6^66https://huggingface.co/datasets/coseal/CodeUltraFeedback_binarized) is a preference dataset in the code domain, consisting of 9.5k preference pairs. We randomly split the dataset, using 90% of the samples for finetuning training and 10% for testing RM accuracy.

## Appendix E More Experimental Results

Due to the length of the paper, we only report important experimental results in the experiment and ablation sections. In this section, we present complementary results in more detail.

### E.1 RM Sample Efficiency Comparison: RM accuracy results

Figure: (a) MathShepherd-pair / 1.5B
Refer to caption: extracted/5895834/images/rm_acc_mathsheherd_1.png

We use RM accuracy evaluation to compare the sample efficiency of RM finetuning, as shown in Figure 10. Consistent with the conclusion from the main experiment, CodePMP improves sample efficiency. Under the same sample conditions, it consistently delivers stable improvements, making RM training more efficient.

### E.2 Ablation Results on logical Reasoning

Table 5 shows the comparison of ReClor and LogiQA2.0 BoN accuracies for different Model-generated pair construction methods for CodePMP. Overall, the $6.7B\&1.3B$ setting is the best among them.

Table 6 shows the comparison of the BoN accuracy performance of ReClor and LogiQA2.0 using GitHub source code and web crawled code data to build CodePMP training pairs. The results verify that Github code is a better source for pair construction.

Table 7 shows the comparison of ReClor and LogiQA2.0 BoN accuracies for CodePMP models with and without EOC. In general, the setup without using EOC brings better results.

Table 8 shows the comparison of ReClor and LogiQA2.0 BoN accuracies for WCD and WSD lr scheduler settings. The results below show that WSD lr scheduler brings better results than WCD.

**Table 5: Comparison of ReClor and LogiQA2.0 BoN accuracies for different Model-generated pair construction methods for CodePMP. Overall, the $6.7B\&1.3B$ setting is the best among them.**
|  |  | Test Set |  |
| --- | --- | --- | --- |
| RM Finetune | Constructions Method | ReClor | LogiQA2.0 |
| ReClor-pair | Source Code & 1.3B | 0.550 | 0.4464 |
| Source Code & 1.3B-Des-Clip | 0.600 | 0.4522 |  |
| Source Code & 6.7B | 0.578 | 0.4605 |  |
| 1.3B & 1.3B-Des-Clip | 0.572 | 0.4745 |  |
| 6.7B & 1.3B-Des-Clip | 0.564 | 0.4809 |  |
| 6.7B & 1.3B | 0.608 | 0.5032 |  |
| LogiQA2.0-pair | Source Code & 1.3B | 0.708 | 0.5217 |
| Source Code & 1.3B-Des-Clip | 0.704 | 0.5268 |  |
| Source Code & 6.7B | 0.714 | 0.5198 |  |
| 1.3B & 1.3B-Des-Clip | 0.738 | 0.5402 |  |
| 6.7B & 1.3B-Des-Clip | 0.742 | 0.5274 |  |
| 6.7B & 1.3B | 0.734 | 0.5415 |  |

**Table 6: Comparison of the BoN accuracy performance of ReClor and LogiQA2.0 using GitHub source code and web crawled code data to build CodePMP training pairs. The results below show that Github code is a better source for pair construction.**
|  |  | Test Set |  |
| --- | --- | --- | --- |
| RM Finetune | Data Source | ReClor | LogiQA2.0 |
| ReClor-pair | Webpage | 0.574 | 0.4898 |
| Github | 0.582 | 0.4981 |  |
| LogiQA2.0-pair | Webpage | 0.742 | 0.5293 |
| Github | 0.752 | 0.5504 |  |

**Table 7: Comparison of ReClor and LogiQA2.0 BoN accuracies for CodePMP models with and without EOC. In general, the setup without using EOC brings better results.**
|  |  | Test Set |  |
| --- | --- | --- | --- |
| RM Finetune | Model | ReClor | LogiQA2.0 |
| ReClor-pair | w/o EOC | 0.582 | 0.4981 |
| w/ EOC | 0.596 | 0.4617 |  |
| LogiQA2.0-pair | w/o EOC | 0.752 | 0.5504 |
| w/ EOC | 0.686 | 0.4809 |  |

**Table 8: Comparison of ReClor and LogiQA2.0 BoN accuracies for WCD and WSD lr scheduler settings. The results below show that WSD lr scheduler brings better results than WCD.**
|  |  | Test Set |  |
| --- | --- | --- | --- |
| RM Finetune | Model | ReClor | LogiQA2.0 |
| ReClor-pair | WCD | 0.552 | 0.4828 |
| WSD | 0.582 | 0.4981 |  |
| LogiQA2.0-pair | WCD | 0.748 | 0.5204 |
| WSD | 0.752 | 0.5504 |  |

**Table 9: Cross-validation evaluation: ReClor and LogiQA2.0 BoN accuracies, black numbers are the results of cross-validation. In the cross-validation evaluation, CodePMP can still bring stable improvements on different test sets.**
|  |  |  | Test Set |  |
| --- | --- | --- | --- | --- |
| RM Finetune | Model | PMP | ReClor | LogiQA2.0 |
| ReClor-pair | 1.5B | ✗ | 0.534 | 0.4592 |
| ✓ | 0.582 | 0.4981 |  |  |
| 7B | ✗ | 0.724 | 0.5453 |  |
| ✓ | 0.756 | 0.5835 |  |  |
| LogiQA2.0-pair | 1.5B | ✗ | 0.710 | 0.5293 |
| ✓ | 0.752 | 0.5504 |  |  |
| 7B | ✗ | 0.748 | 0.6371 |  |
| ✓ | 0.794 | 0.6779 |  |  |

### E.3 Cross-validations on logical Reasoning

We further conduct cross-dataset validation on logical reasoning. Specifically, we evaluate the RM models finetuned on the ReClor-pairs dataset using the LogiQA2.0 test set. Similarly, we evaluate the RM models finetuned on the LogiQA2.0-pairs dataset using the ReClor test set. As shown in Table 9, CodePMP consistently improves RM evaluation performance, demonstrating that the enhancements CodePMP brings to RM training are robust and generalizable.
Note that the BoN accuracies of the RM trained with LogiQA2.0-pair on the Reclor test set are higher than those of the RM trained directly on the Reclor-pair, because the LogiQA2.0-pair dataset is three times larger than the Reclor-pair dataset.

## Appendix F Logical Reasoning Evaluation Examples

We randomly select and present examples from the Reclor test set, which consists of multiple-choice questions based on a given passage. While it is possible to have the model generate additional candidate answers to create a Best-of-N test, it becomes difficult to ensure that the original correct answer remains among the options after introducing new candidates, and to identify the new correct answer. We attempt to use GPT-4o to annotate the correct answers for 32 responses, but the consistency with manual inspection is low, as is the consistency of GPT-4o’s own multiple annotations. It can be inferred that the consistency rate would worsen if expanded to 256 responses. Therefore, after careful consideration, we decide to use RM to score only the original four manually annotated answer options, match the top-ranked option with the manually annotated correct answer, and calculate accuracy. In principle, this method is equivalent to the Best-of-4 test.

**Table 10: Examples from the Reclor test set, which consists of multiple-choice questions based on a given passage.**
| ID | Text | Options | Answer |
| --- | --- | --- | --- |
| 12824 | Mayor: Four years ago, when we reorganized the city police department in order to save money, critics claimed that the reorganization would make the police less responsive to citizens and would thus lead to more crime. The police have compiled theft statistics from the years following the reorganization that show that the critics were wrong. There was an overall decrease in reports of thefts of all kinds, including small thefts.<br>Question: Which of the following, if true, most seriously challenges the mayor’s argument? | 1.<br>In other cities where police departments have been similarly reorganized, the numbers of reported thefts have generally risen following reorganization.<br>2.<br>When city police are perceived as unresponsive, victims of theft are less likely to report thefts to the police.<br>3.<br>The mayor’s critics generally agree that police statistics concerning crime reports provide the most reliable available data on crime rates.<br>4.<br>The mayor’s reorganization of the police department failed to save as much money as it was intended to save. | 1 |
| 218 | Jupiter is a gas giant planet and the largest planet in the solar system. Its mass is 2.5 times the total mass of the other seven planets in the solar system. Observations have found that most of the more than 70 moons surrounding Jupiter are composed of water ice. Therefore, Jupiter’s atmosphere should contain a considerable amount of water.<br>Question: Which of the followings, if true, can best support the above statement? | 1.<br>After hundreds of millions of years, the satellite may slowly fall onto the planet.<br>2.<br>Many of the water in interstellar space exists in gaseous form.<br>3.<br>Uranus is also a gas giant planet, and it has been confirmed that it contains a lot of water ice.<br>4.<br>The satellite and the planets around it were formed from the same gas and dust at the same time. | 3 |
| 10376 | Lake Dali is a barrier lake on the plateau formed by volcanic eruptions. Like salmon living in the sea, Hua Zi fish-Leuciscus waleckii, which lives in a brackish lake, must migrate to the upper reaches of the Tanshui River to spawn and breed, although the four rivers currently flowing into Lake Dali are inland rivers, and none of them leads to the sea. Scientists are still convinced that the Huaziyu in Lake Dali first migrated from the ocean.<br>Question: Which of the following options, if true, provides the best explanation for scientists’ beliefs? | 1.<br>The Leuciscus waleckii that lives in the waters such as Heilongjiang is twice as big as the Leuciscus waleckii fish in Lake Dari.<br>2.<br>The caught Hua Zi fish can only survive for a day or two after being put into sea water or fresh water, and will decay quickly after death.<br>3.<br>Melting glaciers will form Lake Dali, and the overflowing lake was once connected to the Liao River, which flowed into the ocean.<br>4.<br>The researchers put the fry of Hua Zi fish in Dali Lake into Gainao thousands of miles away, and the culture was successful. | 2 |
| 13334 | It is repeatedly claimed that the dumping of nuclear waste poses no threat to people living nearby. If this claim could be made with certainty, there would be no reason for not locating sites in areas of dense population. But the policy of dumping nuclear waste only in the more sparsely populated regions indicates, at the very least, some misgiving about safety on the part of those responsible for policy.<br>Question: Which one of the following, if true, would most seriously weaken the argument? | 1.<br>Until there is no shred of doubt that nuclear dumps are safe, it makes sense to situate them where they pose the least threat to the public.<br>2.<br>There are dangers associated with chemical waste, and it, too, is dumped away from areas of dense population.<br>3.<br>In the event of an accident, it is certain that fewer people would be harmed in a sparsely populated than in a densely populated area.<br>4.<br>Dumping of nuclear waste poses fewer economic and bureaucratic problems in sparsely populated than in densely populated areas. | 3 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Is Majority voting@K comparable with reranking top K with a RM in Figure 3?
soundness: 2
strengths: - The proposed method is highly scalable, allowing it to create 28 million preferred and rejected response pairs.

- The reward model can improve two different reasoning tasks (math reasoning and logic reasoning).
summary: The paper presents CodePMP, a novel and scalable preference model pretraining (PMP) pipeline aimed at enhancing the reasoning capabilities of Large Language Models (LLMs). 

CodePMP builds its dataset in two stages: 1) generating instructions from GitHub files using a CodeLLM, and 2) generating two responses for each instruction from a strong CodeLLM and a weak CodeLLM, which are the preferred and rejected response pairs. 28 million such response pairs are constructed.

After pre-training an LLM on these 28 million pairs, CodePMP improves RM finetuning efficiency. The resulting reward model built upon CodePMP can then be leveraged to improve performance on reasoning tasks, including solving mathematical and logical problems.
weaknesses: - CodePMP creates preference pairs for coding tasks, but coding tasks are not evaluated in experiments. 

- The CodePMP data is constructed by deepseek-coder-instruct. It would be interesting to see whether CodePMP can further improve deepseek-coder-instruct on coding tasks.

- The reward model is initialized using Qwen2 models (Qwen2-1.5B and Qwen2-7B), which are more capable in math reasoning than the math generator MetaMath-Mistral-7B in Section 4.1.3. A more meaningful setting would be to examine whether the Qwen2 based reward model can further improve the performance of Qwen2 or Qwen2-Math on math reasoning tasks.

- Majority voting should be included as a baseline in Figure 3.

- Further analysis is needed to understand why the coding-based CodePMP contributes to improvements in math and logical reasoning.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Figure 4 and 5 are too small to watch.
2. Why CodePMP can help reward modeling in mathematical tasks? Please give more analysis in the paper.
3. Reference formatting issues, such as line 429.
4. Would you consider using a larger reward model to verify the scalability?
soundness: 2
strengths: 1. This work proposes an interesting idea of preference pre-training.
2. This method is automated, reducing the dependency on manually annotated preference data.
3. With best-of-N strategy, CodePMP could improve LLM's reasoning performance.
summary: The paper presents codepmp, a scalable pretraining method designed to improve the reward model by leveraging synthesized code-preference pairs. CodePMP collects a vast repository of publicly available source code from GitHub and leverages a strong and a weak model to generate synthetic preference pairs to pretrain a reward model. Utilizing this pretrained reward model, LLM's ability to solve reasoning tasks is largely improved by best of N sampling.
weaknesses: 1. Unfortunately, the effectiveness of the RM was not fully verified, e.g. by using the RM for RFT or PPO training.
2. It remains unclear whether and how code training could help reasoning tasks in natural language. It would be great if the authors could have explored more on relationship between coding and reasoning tasks for model training.
3. No mention of whether the dataset will be open sourced.
4. The models used in data construction are limited. It would be helpful to verify the generalization with more number of generators, at least the ones used to construct the training data.
5. Evalution tasks are limited. Regular coding benchmarks like humaneval and mbpp should be relevant.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In the Data Construction, a description summarizer is used to generate prompts that describe the code. There is no any detail on this part. And what is quality of constructed data pair? There is no any related analysis in this work. It is not a human-annotated dataset. So it is important to show the data quality.  

2. In Section 4.2.2, how the different number of N candidate responses are generated? Is Qwen2 model used to sample these candidate response? There is no details about this.

3. In Figure 4, the author claims that: RM with Code PMP initialization consistently outperform. However, it seems not right. Figure 4 (g) shown a counter part case. RM with Code PMP initialization is consistently  worse than RM without Code PMP initialization. Please check in detail.

4. As shown in Figure 5, increasing the number of code-preference pairs consistently improve accuracy. Look like there is still some improvement space. Do you think more preference data can be obtained without too much work?

Other questions: is the PMP data planed to be released? how many code-preference pairs? (only file number and token number are shown, e.g, Python files, with 20 million files and 13.1 billion tokens,)
soundness: 2
strengths: 1. A pipeline that create synthesize code-preference pairs is introduce in this work. It can help to solve the scarcity of high-quality preference data if it is working well.

2. Large improvements are achieved on several reasoning tasks (GSM8K, MATH, ReClor, LogiQA2.0) with these synthesize code-preference pairs data.

3. Some details of CodePMP are shown. These can be helpful to the community.
summary: This work introduce CodePMP, a pipeline that create synthesize code-preference pairs in order to have more high quality preference data. The authors show that a better reward model can be obtained with these preference pair data. And it also shows the improvements in several reasoning tasks (GSM8K, MATH, ReClor, LogiQA2.0) with these synthesize code-preference pairs data.
weaknesses: 1. The experiments details are missing or confused. It is better to clarify in the next version. Please check the following Questions section for more details. 

2. In the experiments, only two Qwen2 models are used for the evaluation. Other model family results can be used for the verification of the methods.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Given that CodePMP relies heavily on code data, how well do you expect it to generalize to reasoning tasks that do not have a structured, logical basis like code (e.g., common-sense reasoning, social interactions, or natural language inference)? Have you explored tasks beyond mathematical and logical reasoning?
soundness: 2
strengths: **1. Efficiency in Reward Model Fine-Tuning**


CodePMP significantly enhances the sample efficiency of reward model (RM) fine-tuning, as the experiments indicate that fewer samples are required to achieve high accuracy. The paper shows that RMs initialized with CodePMP outperform directly fine-tuned models, even with smaller amounts of data. This efficiency makes the method appealing for real-world applications where data collection can be expensive or time-consuming.

**2. Scalable Data Generation**


One of the major strengths of CodePMP is its ability to generate large-scale preference data using code-preference pairs. By synthesizing data from publicly available source code on platforms like GitHub, the authors address the scarcity of high-quality preference data. This reduces the reliance on costly human annotation, making the pretraining process significantly more scalable and potentially more cost-effective than traditional methods relying solely on human feedback.

**3. Leveraging Code for Reasoning Tasks**

The paper introduces an innovative use of GitHub-sourced code to enhance reasoning in LLMs. By leveraging the logical structure of code, the authors create code-preference pairs, where stronger models generate "accepted" responses and weaker models generate "rejected" ones. This approach capitalizes on the abundance of publicly available code on GitHub, using it to simulate reasoning tasks. The structured nature of code makes it an ideal dataset for pretraining models to improve problem-solving and logical reasoning capabilities, aligning with recent research showing that training on code boosts reasoning performance in LLMs.
summary: This paper presents CodePMP, a novel pipeline for preference model pretraining aimed at enhancing the reasoning capabilities of large language models (LLMs). The primary challenge addressed is the scarcity of high-quality preference data, which is crucial for reinforcement learning from human feedback (RLHF) but costly to annotate. The authors propose a solution by leveraging code-preference pairs synthesized from publicly available high-quality source code, such as from GitHub, to improve reward model finetuning.

The CodePMP pipeline generates millions of code-preference pairs by using two different code models (a stronger model and a weaker model) to create a “chosen” and “rejected” code response for a given code prompt. These pairs are then used to pretrain the preference models, leading to better sample efficiency and improved reasoning performance in downstream tasks like mathematical reasoning (GSM8K, MATH) and logical reasoning (ReClor, LogiQA2.0).
weaknesses: **1. Lack of Examples** 

The paper lacks clear and detailed examples that illustrate how CodePMP actually solves reasoning tasks using code-preference pairs. While the overall methodology is explained, it's hard for the reader to grasp how reasoning is specifically improved through the use of code data. The absence of step-by-step examples of how the code-preference pairs are used to train the reward model leaves an important gap. For instance, showing a few real examples of code, its corresponding “chosen” and “rejected” responses, and how these responses contribute to solving a reasoning task would be highly valuable. The appendix, while providing some additional technical details, does not adequately fill this gap. 


**2. Limited Baselines**

The paper’s comparison to **alternative methods** is quite limited. It focuses primarily on showing the improvements over direct RM fine-tuning without pretraining, but fails to benchmark CodePMP against other advanced reasoning techniques such as:

Chain-of-Thought (CoT) prompting, which has been shown to significantly improve reasoning performance by breaking down tasks into intermediate steps.
Search-based methods like Monte Carlo Tree Search (MCTS), which have been effective in enhancing reasoning capabilities in language models through external computation and verification steps.
A reader would expect a thorough comparison of CodePMP against these methods, especially since these are widely used in the current state-of-the-art LLM systems. Additionally, the paper should clarify:

How does CodePMP stack up against CoT and search-based approaches in the same experimental setup?
What are the strengths and weaknesses of CodePMP relative to these techniques?

**3. Challenges in Maintaining Data Quality with Evolving LLM Capabilities**

A key limitation of the CodePMP methodology is its reliance on two distinct models—one to generate "accepted" (high-quality) responses and another to produce "rejected" (low-quality) responses. While this setup works for synthesizing large quantities of training data, there is a valid concern that as LLMs improve, the gap between the two models could narrow, leading to mismatches in the data quality. For example, there is a risk of **data distribution drift:** As the strong model improves, the generated "accepted" responses may no longer reflect the kind of reasoning tasks the preference model was initially trained on, leading to a mismatch between the training data and the actual data seen in practice. The system might overfit to synthetic training data that does not generalize well to the actual tasks faced by stronger LLMs.

Meta-review: This paper introduces CodePMP, a scalable preference model pretraining pipeline that leverages synthesized code-preference pairs from publicly available, high-quality source code to improve reward model fine-tuning. I appreciate the authors added additional results during the rebuttal period. However, there are some concerns still in the current work: (1)  the evaluation does not demonstrate the results of reward models in model optimization (e.g. DPO), which should be the end goal; (2) Scalability Concerns: While CodePMP is designed to be scalable, LLM models often evolve and would need continuous iteration of preference data to adapt to new LLM capabilities.

Decision: Reject