Learning Code Preference via Synthetic Evolution

## Abstract

Abstract Large Language Models (LLMs) have recently demonstrated remarkable coding capabilities.
However, assessing code generation based on well-formed properties and aligning it with developer preferences remains challenging.
In this paper, we explore two key questions under the new challenge of code preference learning:
(i) How do we train models to predict meaningful preferences for code? and
(ii) How do human and LLM preferences align with verifiable code properties and developer code tastes?
To this end, we propose CodeFavor ,
a framework for training pairwise code preference models from synthetic evolution data,
including code commits and code critiques.
To evaluate code preferences,
we introduce CodePrefBench , a benchmark comprising 1364 rigorously curated code preference tasks to cover three verifiable properties—correctness, efficiency, and security—along with human preference.
Our evaluation shows that CodeFavor holistically improves the accuracy of model-based code preferences by up to 28.8 % percent 28.8 28.8\% .
Meanwhile, CodeFavor models can match the performance of models with 6 ∼ 9 × 6\sim 9\times more parameters
while being 34 × 34\times more cost-effective.
We also rigorously validate the design choices in CodeFavor via a comprehensive set of controlled experiments.
Furthermore, we discover the prohibitive costs and limitations of human-based code preference:
despite spending 23.4 person-minutes on each task, 15.1 ∼ 40.3 % similar-to 15.1 percent 40.3 15.1\sim 40.3\% of tasks remain unsolved.
Compared to model-based preference,
human preference tends to be more accurate under the objective of code correctness,
while being sub-optimal for non-functional objectives.

## 1 Introduction

Large Language Models (LLMs) for code (Chen et al., ; GitHub, ; Amazon Web Services, ) have become instrumental in modern software development.
Code LLMs assist developers in various scenarios, from suggesting code completions and generating functional code based on user instructions to proposing complex code changes to resolve bug reports and feature requests.

Instruction-tuned LLMs (Luo et al., ; Wei et al., ) are increasingly adept at generating functional code based on natural language instructions.
However, evaluating the quality of LLM-generated code remains challenging, particularly regarding code correctness, efficiency, security, adherence to best practices, and alignment with developer preferences.
Effectively and efficiently assessing LLM-generated code against these properties is crucial for both evaluation (Liu et al., ) and preference optimization for code LLMs (Weyssow et al., ).
Nevertheless, the subject of learning code preferences has been largely under-explored,
motivating us to study code preferences systematically and train code preference models with new data and modeling methods.

Following the established format in LLM-as-a-judge (Chiang et al., ),
we define the code preference task as follows:
Given a user query, a pair of two candidate code responses, and optionally a preference criterion,
code preference is demonstrated by choosing one response over the other.
Specifically, current approaches estimate code preference based on three proxies,
each with advantages and limitations:

- •
Code execution:
Code preference in another way can be confidently determined by execution statuses (Liu et al., ).
However, applying code execution to arbitrary programs poses challenges due to
*(i)* setup complexity,
*(ii)* code incompleteness, and
*(iii)* execution overhead.
For instance, code execution may necessitate specific hardware (*e.g.,* GPUs) and precise software versions,
which are challenging to deduce from the code and, even if inferred, are too cumbersome to set up and run.
- •
Human annotation:
Human-labeled preferences are often seen as the standard oracle in developing LLMs, such as in the RLHF for OpenAI’s GPT models (Ouyang et al., ) and LLM evaluation in Chatbot Arena (Chiang et al., ).
However, applying human labeling to code is particularly challenging and cost-intensive.
Programs are inherently abstract and complex, labeling them requires experienced developers to perform detailed analysis and testing.
Meanwhile, human preference is inherently subjective, influenced by the annotators’ code tastes and expertise, which can cause noisy preferences for code, whose quality could otherwise be concretely defined and measured.
- •
LLM-as-a-judge:
Prominent LLMs have also been employed to evaluate LLM responses (Chiang et al., ; Zheng et al., ; McAleese et al., ).
This method is more scalable than human labeling and can be generalized to a wider range of programs compared to code execution.
However, its reliability often hinges on the reasoning capabilities of high-cost proprietary LLM judges (Weyssow et al., ), subject to inherent biases (Zheng et al., ).

While scaling human- and execution-based preference for code is human-resource- and engineering-challenging(^1^11Such as hiring more annotators with domain expertise and setting up individual execution environments.),
improving model-based code preference becomes emerging and crucial, beyond directly prompting off-the-shelf models (Weyssow et al., ).
Furthermore,
how exactly human developers and prominent LLMs determine code preference remains obscure,
with little research on quantifying or analyzing their performance across various code criteria.
To this end, this work attempts to explore two critical questions in code preference learning:

- 1.
Technical question:
How can we build effective and efficient code preference models regarding modeling approaches and data sources?
- 2.
Empirical question:
What are the preferences of human annotators and LLMs, and to what extent do they align with verifiable code properties and human judgments?

CodeFavor.
We propose CodeFavor, a novel framework for training code preference models.
Specifically, CodeFavor employs pairwise modeling to predict preference within a code pair according to a user-specified criterion.
We propose two synthetic data generation methods to construct preference ranking samples from code evolution:
*(i) Commit-Instruct* transforms the pre- and post-commit code snippets to code preference pairs; and
*(ii) Critic-Evol* samples faulty code from a draft LLM and has another critic LLM to improve the broken code.
These methods allow us to curate synthetic preference data efficiently, leveraging the natural evolution of code and the capabilities of existing LLMs.

CodePrefBench.
To evaluate code preferences labeled by various approaches,
we introduce CodePrefBench, a collection of 1,364 carefully curated preference tasks.
These tasks target verifiable properties including correctness, efficiency, and security, while additionally considering general developer preferences.
Using CodePrefBench, we extensively analyze the effectiveness and cost of code preferences derived from developer agreement, general LLMs, and CodeFavor models.
Our study demystifies key insights on the pitfalls of different approaches over different coding criteria.
Our results also demonstrate that our models not only achieve top performance in effectiveness but also are significantly more cost-efficient compared to existing solutions.

We summarize our main contributions below:

- 1.
Dimension & Technique:
We propose CodeFavor, the *first* open recipe to train pairwise code preference models.
At the heart of CodeFavor is a pairwise modeling design and two complementary methods for generating synthetic preference pairs from code evolution.
- 2.
Benchmark & Code:
We present CodePrefBench, the *first* comprehensive developer preference benchmark with 1,364 labeled by three verifiable oracles (correctness, efficiency, security) and general developer preferences from 18 annotators.
We release the data and code at [https://github.com/amazon-science/llm-code-preference](https://github.com/amazon-science/llm-code-preference).
- 3.
Study & Results:
Based on CodePrefBench, we comprehensively quantify and conduct case studies on code preferences derived from human developers and LLMs.
We show that CodeFavor can significantly improve the accuracy of model-based preference by up to 28.8%.
CodeFavor models can match the preference accuracy of models that are larger by $6\sim 9\times$,
while being cheaper by $34\times$.
We also conduct extensive controlled experiments to validate our design choices.

## 2 CodeFavor : Learning Code Preference via Synthetic Evolution

Figure: Figure 1: Approach overview of CodeFavor. We train a pairwise preference model using synthetic data created from two complementary sources of code evolution: Commit-Instruct and Critic-Evol.
Refer to caption: /html/2410.03837/assets/x1.png

[Figure 1](#S2.F1) depicts the approach overview.
Our framework, CodeFavor, is designed for training pairwise models
that predict code preference by taking an instruction, a code pair, and a criterion as input.

Additionally,
CodeFavor proposes two synthetic data generation methods, Commit-Instruct and Critic-Evol, for extracting synthetic training data from code evolution.
Specifically,
Commit-Instruct creates contrasting code pairs through rephrasing and filtering massively available code commits.
Complementarily,
Critic-Evol prompts a large critic LLM to judge and revise code snippets from a smaller draft LLM,
pairing the drafted attempt and revision to create synthetic preference data.

### 2.1 Pairwise Modeling

Input.
We follow prior work in reward modeling (Zhao et al., ; Liu et al., ; Dong et al., ) and use decoder-based transformers for learning pairwise preferences.
Specifically, the model $\pi$ takes as input a prompt $x=\{i,y_{A},y_{B},c\}$,
comprised of
(i) an instruction $i$,
(ii) a pair of code candidates $\{y_{A},y_{B}\}$, and
(iii) a fine-grained criterion $c$ defining the desired preference following (Kim et al., ).
More specifically, our prompt format is shown in [Equation 1](#S2.E1):

$$ $\displaystyle x=\texttt{[INSTRUCTION]}\{i\}\texttt{[CODE_A]}\{y_{A}\}\texttt{[CODE_B]}\{y_{B}\}\texttt{[CRITERION]}\{c\}$ (1) $$

Output.
We explore two output designs for code preference modeling: classification and generation.

- 1.
*Classification:*
We train a binary classifier based on a single next-token prediction (Zhao et al., ; Liu et al., ).
Specifically, given the exact prompt format in [Equation 1](#S2.E1),
the classifier outputs either a token “A” if $y_{A}$ is preferable to $y_{B}$ for $\{i,c\}$ or “B” otherwise.
At inference time, the preference decision is determined by the next-token probability between “A” and “B”, shown in [Equation 2](#S2.E2).
- 2.
*Generation:*
We also train generative models to provide code preference decisions in natural language.
Specifically, we use a more natural version of [Equation 1](#S2.E1),
demonstrated in LABEL:lst:prompt-feedback, for prompting the generation model.
Next, we parse the code preference decision from the model-generated feedback using rules detailed in [Section A.3](#A1.SS3).

$$ $\displaystyle y_{+}=\begin{cases}y_{A}&\text{if }\mathbb{P}_{\pi}(y_{A}\succ y_{B}|x)>0.5\\ y_{B}&\text{otherwise}\\ \end{cases}=\begin{cases}y_{A}&\text{if }\mathbb{P}_{\pi}(\text{``A''}|x)>\mathbb{P}_{\pi}(\text{``B''}|x)\\ y_{B}&\text{otherwise}\\ \end{cases}$ (2) $$

The advantage of classification modeling is computing efficiency as only one token is produced.
Meanwhile, generative modeling optimizes for interpretability, with reasoning steps explicitly displayed.

### 2.2 Synthetic Code Preference from Code Evolution

Training a pairwise preference model requires a rich set of contrastive code pairs, along with the corresponding instructions and evaluation criteria.
Collecting complex (Luo et al., ) and diverse (Wei et al., ) code pairs is crucial yet challenging,
given such resources are neither readily available nor curated by prior work.
To this end, we propose to create code preference training data using synthetic code evolution,
based on code commits ([Section 2.2.1](#S2.SS2.SSS1)) and code critiques ([Section 2.2.2](#S2.SS2.SSS2)).
We argue that code evolution is a practical source for synthesizing code preferences,
not only because of its natural indication of preferences(^2^22Software iterates towards improvement, making post-evolution code oftentimes better than pre-evolution code.),
but also thanks to their general availability and diversity.
We focus on the general methodology in this section and defer the detailed prompting implementation to
[Section A.1](#A1.SS1).

#### 2.2.1 Commit-Instruct : Preference Learning from Code Commits

We propose Commit-Instruct, a synthetic data generation method transforming raw code commits into code preference training samples.
Specifically,
the workflow (middle of [Figure 1](#S2.F1)) employs a critic LLM to analyze each raw code commit and produce a training sample in a desired format [Section 2.1](#S2.SS1).
Each raw commit can be denoted by $r=(m,y_{pre},y_{post})$,
where $m$ is the commit message, and $\{y_{pre},y_{post}\}$ are the pre- and post-commit code snippets.
Commit-Instruct processes each commit in three steps:

- 1.
Reasoning:
The critic LLM is instructed to reason and explain code changes from $y_{pre}$ to $y_{post}$.
- 2.
Filtering:
Given the explanation, the critic LLM first determines whether or not the code change is meaningful.
If so, we proceed to the next step; otherwise, the commit is discarded.
This step aims to ensure the quality of synthetic data by excluding trivial or ambiguous code changes.
- 3.
Rephrasing:
Based on the commit $r$ and its explanation,
the critic LLM synthesizes a preference sample in the desired format $x=\{i,y_{A},y_{B},c\}$ ([Section 2.1](#S2.SS1)).
Specifically, $y_{A}$ and $y_{B}$ are rephrased from $y_{pre}$ and $y_{post}$ to emphasize the actual change.
$i$ is the instruction generated to describe $y_{pre}$ and $y_{post}$ and the criterion $c$ is concluded by how $y_{post}$ improves $y_{pre}$.
The rephrased version of $y_{post}$ is regarded as the chosen response $y^{+}$ in model training.

[Figure 2](#A1.F2) in [Section A.1](#A1.SS1) provides the detailed prompt implementation for Commit-Instruct.

#### 2.2.2 Critic-Evol : Preference Learning from Code Critiques

While synthetic evolution can be gathered from human data such as code commits,
it can also be derived from synthetic data.
As such, we propose Critic-Evol which generates synthetic code preference data
by asking a stronger critic model $\pi^{+}$ to revise the code generated by a weaker draft model $\pi^{-}$.

Specifically, Critic-Evol takes a set of coding instructions $\mathcal{I}=\{i_{1},i_{2},\cdots,i_{n}\}$ as inputs, for each of which ($i_{k}$) we apply steps below to either transform it into a synthetic training sample or simply drop it:

- 1.
Fault sampling:
This step starts with a suitably weak but functional model, denoted as $\pi^{-}$,
which statistically allows us to sample improvable code solutions $y_{k}^{-}\leftarrow\pi^{-}(i_{k})$.
- 2.
Critique & filtering:
We instruct the critic LLM $\pi^{+}$ to code review $y_{k}^{-}$,
by pointing out noticeable code quality issues and defining the criterion (*e.g.,* $c$) regarding the code defects.
$\pi^{+}$ may also be satisfied with $y_{k}^{-}$ and thus we stop synthesizing code preference data for $(i_{k},y_{k}^{-})$.
- 3.
Revision:
If the critique from $\pi^{+}$ suggests $y_{k}^{-}$ can be significantly improved,
$\pi^{+}$ creates $y_{k}^{+}$ by revising $y_{k}^{-}$ to meet the desired criterion $c$.
As such, a new synthetic code preference sample is composed as $\{i_{k},y_{k}^{-},y_{k}^{+},c\}$, with $y_{k}^{+}$ being the chosen response.

[Figure 4](#A1.F4) in [Section A.1](#A1.SS1) provides more details on implementation of Critic-Evol.

### 2.3 Datasets

We apply our techniques to create two synthetic datasets for code preference learning:

Commit-Instruct-EditPack consists of 20,641 code preference samples synthesized from EditPackFT-Multi (Cassano et al., ) and Llama3-70B-Instruct (Dubey et al., ).
After filtering out non-permissive code, we obtain 22,469 blessed Python commits from EditPackFT-Multi for use as the raw commits and prompt Llama3-70B-Instruct (Dubey et al., ) to perform the Commit-Instruct strategy.
91.9% of the commits are successfully transformed into code preference data ([Section 2.1](#S2.SS1)) and 8.1% of them are filtered out due to lack of clear significance.

Critic-Evol-SOSS has 41,595 synthetic code preference samples using the Critic-Evol technique.
Specifically, we run Llama3-8B-Instruct as the draft model (*i.e.,* $\pi^{-}$) over 50,661 coding instructions from the Self-OSS-Instruct dataset (BigCode, ) to produce initial code solutions.
82.1% of these initial attempts are revised and extended by Llama3-70B-Instruct as the critic model,
whereas the rest 17.9% are deemed good enough such that a revision is unnecessary.

Data processing.
To mitigate positional bias, we augment the dataset by flipping the order within each code pair, which also doubles the training samples.
Besides, we clip the code comments in Critic-Evol samples,
given that comments barely affect code quality metrics and LLM-generated comments may let faulty code “sound right”.
[Section 3.4](#S3.SS4) also shows code comments can negatively impact code preferences.

## 3 Evaluating Code Preference Learning with CodePrefBench

To systematically evaluate code preferences across different methods,
we create the CodePrefBench, consisting of 1,364 preference tasks in total.
It covers four objectives in code preference evaluation:
correctness, efficiency, security, and human preference.
[Table 1](#S3.T1) provides an overview of the four categories of tasks.

This section presents the curation process of CodePrefBench ([Section 3.1](#S3.SS1)) and the results from human ([Section 3.2](#S3.SS2)) and LLMs ([Section 3.3](#S3.SS3)),
along with the controlled experiments in [Section 3.4](#S3.SS4).
Additional details, such as case studies ([Section A.4](#A1.SS4)) and contamination analysis ([Section A.5](#A1.SS5)), are deferred to the Appendix.

**Table 1: Overview of CodePrefBench.**
| Objective | # Tasks | Source | Preference Oracle |
| --- | --- | --- | --- |
| Code Correctness | 660 | EvalPlus (Liu et al., ) | Test execution |
| Code Efficiency | 352 | EvalPerf (Liu et al., ) | # CPU instructions |
| Code Security | 207 | CyberSecEval (Bhatt et al., ) | Static analyzer |
| Developer Preference | 145 | LBPP (Matton et al., ) | Human agreement |
|  |  | BigCodeBench-Hard (Zhuo et al., ) |  |
| Total | 1,364 |  |  |

### 3.1 Benchmark Setup

In CodePrefBench, we evaluate code preference approaches over four objectives,
covering three verifiable properties (*i.e.,* correctness, efficiency, and security) and human preference.
For verifiable objectives, we generate oracle labels via code execution and static analysis.
For human preference, we engage three annotators to label each code pair
to form the evaluation set and establish baselines
To ensure benchmark quality,
we only use clear-cut *good-bad* pairs and
exclude *tie* pairs due to their inherent ambiguity.
The creation of the dataset for each evaluation category is detailed below:

Objective #1: Correctness.
We construct *correct-wrong* pairs from EvalPlus datasets (Liu et al., ),
*i.e.,* HumanEval+ (164 tasks) and MBPP+ (378 tasks), as they rigorously test LLM solutions with extensive test cases that can detect subtle bugs.
We derive at most two contrastive code pairs for evaluation from each seed task.
In each code pair, the *wrong* code comes from test-falsified LLM solutions while the *correct* is the human-written ground truth.
Finally, we obtain 660 *correct-wrong* code pairs.
The number is smaller than $2\times(164+378)$ as “wrong” samples do not exist in some easy tasks.

Objective #2: Efficiency.
We construct *fast-slow* pairs from EvalPerf datasets (Liu et al., ).
EvalPerf exercises the performance of LLM-generated correct solutions using 121 performance-exercising tasks equipped with performance-exercising test inputs.
The EvalPerf dataset provides fast-to-slow reference solutions with distinct performance for each task.
Therefore, we sample *fast-slow* pairs over the reference samples at a step size of 3, and obtain 352 *fast-slow* pairs.

Objective #3: Security.
We construct *secure-vulnerable* code pairs from CyberSecEval (Bhatt et al., ),
which includes 351 Python vulnerabilities detected by security analyzers.
We prompt GPT-4o to fix each vulnerability and rerun the security analyzers to guarantee the fix.
Additionally, we equip each code pair with a generalized instruction generated by GPT-4o, so the instruction is not biased towards any candidate.
Finally, we obtain 207 *secure-vulnerable* code pairs to evaluate code security preference.

Objective #4: Human preference.
We established a team of 18 developers to annotate pairs of code responses sampled from DeepSeek V2 over the latest open and close domain coding benchmarks,
*i.e.,* 148 BigCodeBench-Hard (Zhuo et al., ) tasks and 161 LBPP (Matton et al., ) tasks.
Specifically,
we sample 8 solutions per task at a temperature of 0.8 and select the code pair with the largest edit distance.
We follow the same annotation criteria as Chatbot Arena (Chiang et al., ):
given two responses, users select the one they would use for the instruction (or skip it if both are tied).
Lastly,
we obtained 145 preference pairs without conflicting preferences out of three annotations per pair.

Additionally,
we evenly shuffle the order of code pairs within each category to prevent positional bias.
By default, we remove code comments when evaluating tasks focused on verifiable objectives,
as comments should not affect the outcome.
At evaluation, LLMs predict each code preference task using greedy decoding,
following criteria aligned with the benchmark objective.

### 3.2 Human Results

We first study and draw insights from developer labeling through their expertise, annotation confidence, annotation time, and finally their results based on the major voting of 3 developers per task:

[type=table,width=.43]
Low
High
Very High
Correctness
0%
68.2%
31.8%
Efficiency
0%
88.7%
11.3%
Security
0%
80.8%
19.2%
Developer confidence distribution.

- •
Expertise:
Our annotation team consists of 18 software developers, two-thirds of which hold degrees in computer science, and 95% of them have over two years of programming experience.
For Python proficiency, 43% of them self-rate as advanced, while the rest consider themselves middle-level.
- •
Confidence:
[Section 3.2](#S3.SS2) lists the distribution of developer confidence.
All developers are overall confident about their annotations.
Specifically, developers are more confident when labeling correctness,
with a higher ratio of “very high” confidence compared to that for the efficiency ($2.8\times$) and security ($1.7\times$) categories.
From annotation notes of developers,
it is partially because program correctness can be assessed by manual testing,
while code efficiency and security are harder to evaluate without domain-specific knowledge.
{wrapstuff}[type=figure,width=.42]
CDF of preference labeling time.
- •
Overhead:
[• ‣ Section 3.2](#S3.I1.i2) illustrates the cumulative distribution of the annotation time per sample/developer, visualized by removing the top-1%-longest outliers.
Overall, each task on average costs each developer 7.8 minutes to annotate, with the 99-percentile of 26 minutes, indicating that serious developer labeling for code preferences is time-consuming.
Furthermore, code efficiency and security tasks (9 minutes on avg.) take longer to annotate compared to labeling correctness tasks (6.8 minutes on avg.), which is consistent with developer confidence and final results.
- •
Accuracy:
The first result row in [Table 2](#S3.T2) presents the human performance based on the majority voting of three developers per task.
Consistent with the annotation confidence and speed,
human labeling achieves the most accurate measurement of code preferences in the code correctness category,
with a solve rate of 84.9%.
While not the best, human performance still decently solves 74.9% of preference tasks targeting code efficiency.
Surprisingly,
while developer confidence in code security annotation is higher than that in code efficiency,
the security score is as low as 59.7%.
This is because 73.9% of code pairs are annotated as equally secure, while our scoring method assigns 0.5 accuracy to each tied case.
This discrepancy indicates that generalist programmers may struggle to accurately assess non-functional code properties such as code security, which may require specialized domain expertise.

**Table 2: Accuracy (%) of evaluated models on CodePrefBench. Scores within 1 percentage point of the highest are highlighted in bold. Bracketed numbers denote the ranges of uncertain responses, half of whose ratio is accounted for the final accuracy score. Case studies are available in [Section A.4](#A1.SS4).**
|  | Correctness | Efficiency | Security | Avg. | Human Pref. |
| --- | --- | --- | --- | --- | --- |
| 3-developer agreement | 84.9 ($\pm$9.4) | 74.9 ($\pm$5.3) | 59.7 ($\pm$37.0) | 73.2 | N/A |
| Proprietary Models |  |  |  |  |  |
| Claude 3.5 Sonnet | 65.8 ($\pm$0.8) | 79.9 ($\pm$0.1) | 98.1 | 81.2 | 64.8 |
| Gemini 1.5 Pro 001 | 59.2 ($\pm$3.0) | 79.5 ($\pm$1.4) | 71.3 ($\pm$27.3) | 70.0 | 66.6 ($\pm$1.7) |
| Gemini 1.5 Flash 001 | 58.6 ($\pm$7.9) | 81.1 ($\pm$0.1) | 85.0 ($\pm$8.2) | 74.9 | 60.0 |
| Open-Weight Models |  |  |  |  |  |
| Llama-3.1-405B-Instruct | 68.9 ($\pm$2.7) | 78.3 ($\pm$0.4) | 99.0 | 82.2 | 68.3 |
| Mistral Large 2 (123B) | 65.8 ($\pm$0.5) | 81.2 ($\pm$0.3) | 99.5 | 82.2 | 71.7 |
| DeepSeek V2.5 (236B) | 65.8 ($\pm$0.8) | 80.7 | 97.3 ($\pm$0.2) | 81.3 | 69.0 |
| Llama-3.1-70B-Instruct | 60.2 ($\pm$0.3) | 77.3 ($\pm$0.3) | 97.8 ($\pm$0.7) | 78.4 | 69.0 |
| Codestral-22B-v0.1 | 58.0 ($\pm$0.8) | 78.3 ($\pm$0.1) | 94.0 ($\pm$2.7) | 76.8 | 60.0 |
| Llama-3-70B-Instruct | 55.7 ($\pm$2.5) | 76.0 ($\pm$1.6) | 96.6 ($\pm$1.0) | 76.1 | 63.8 ($\pm$0.3) |
| Gemma-2-27B | 55.4 ($\pm$4.9) | 78.4 ($\pm$0.9) | 80.8 ($\pm$14.8) | 71.5 | 61.4 |
| Our Models and Baselines |  |  |  |  |  |
| Mistral Nemo Instruct (12B) | 51.4 ($\pm$1.2) | 69.7 ($\pm$0.4) | 82.9 ($\pm$7.5) | 68.0 | 66.2 |
| + CodeFavor Classification | 58.0 | 76.1 | 96.6 | 76.9 | 64.1 |
| + CodeFavor Generation | 58.8 | 77.8 | 96.6 | 77.7 | 66.9 |
| Gemma-2-9B-Instruct | 52.4 ($\pm$6.1) | 75.1 ($\pm$1.6) | 52.7 ($\pm$47.3) | 60.1 | 64.1 ($\pm$0.7) |
| + CodeFavor Classification | 56.8 | 75.3 | 92.3 | 74.8 | 67.6 |
| + CodeFavor Generation | 57.0 | 78.7 | 96.6 | 77.4 | 64.1 |
| Llama-3-8B-Instruct | 49.5 ($\pm$0.9) | 71.9 | 90.3 ($\pm$0.5) | 70.6 | 58.6 |
| + CodeFavor Classification | 58.0 | 73.0 | 95.2 | 75.4 | 62.8 |
| + CodeFavor Generation | 58.2 | 75.0 | 98.6 | 77.2 | 69.0 |
| Mistral-7B-Instruct-v0.3 | 48.5 ($\pm$1.5) | 66.6 ($\pm$0.1) | 78.5 ($\pm$9.4) | 64.5 | 58.3 ($\pm$1.0) |
| + CodeFavor Classification | 62.4 | 64.8 | 95.7 | 74.3 | 60.7 |
| + CodeFavor Generation | 57.1 | 77.3 | 90.3 | 74.9 | 66.9 |

### 3.3 Model Results

[Table 2](#S3.T2) evaluates human, existing LLMs, and CodeFavor models on CodePrefBench.
By default, CodeFavor models are obtained in two steps:
*(i)* training two models using Commit-Instruct-EditPack and Critic-Evol-SOSS individually;
and *(ii)* merging the two models on average to obtain a final model.

Overall results.
We present the overall results by looking at the accuracy averaged across the three verifiable objectives, *i.e.,* the“Avg.” column.
Among the evaluated existing LLMs,
Llama-3.1-405B-Instruct and Mistral Large 2 perform the best,
tightly followed by Claude 3.5 Sonnet and DeepSeek V2.5.
Meanwhile, Codestral, at a parameter size of 22B, demonstrates a decent result, on par with Llama-3-70B-Instruct.
We demonstrate the effectiveness of CodeFavor by fine-tuning a comprehensive set of affordable models, from 7B to 12B.
While these small models are relatively weak out of the box,
CodeFavor improves their overall performance by $9.3\sim 28.8$% relatively. For instance, CodeFavor’s generation modeling enables Mistral Nemo Instruct, Gemma-2-9B-Instruct, and Llama-3-8B-Instruct
to achieve an overall score of $77.2\sim 77.7$ respectively,
slightly outperforming the critic model (*i.e.,* Llama-3-70B-Instruct),
despite being smaller by $6\sim 9\times$.
Notably, all of CodeFavor models even outperform the human-agreement baseline,
largely because generalist developers have high uncertainty and thus low performance in the security category.

Correctness.
Human annotation largely outperforms all language models in choosing the correct code,
outperforming the best model by 23%.
Among the evaluated existing LLMs,
Llama-3.1-405B-Instruct as an open-weight model solves the most tasks (*i.e.,* 68.9%),
outperforming Claude 3.5 Sonnet, Mistral Large 2, and DeepSeek V2.5 by 4.7%.
Meanwhile, small LLMs ($\leq$ 12B) are incapable of such tasks out of the box, producing almost random preferences ($\sim 50\%$).
Nonetheless, CodeFavor improves the accuracy of code correctness preference for these models by $8.8\sim 28.7\%$,
commonly surpassing their critic model (*i.e.,* Llama-3-70B-Instruct) by up to 12%.

Efficiency.
While human preference aces over the evaluated LLMs on the preference of code preference,
it presents overall sub-optimal preferences regarding code efficiency.
For example, Mistral Large 2, the best model in this category,
surpasses developer-agreement-based preference by $8.4\%$.
Gemini Flash and DeepSeek V2.5 tightly follow Mistral Large 2 within a 1% gap,
also outperforming human preference in the code efficiency category.
While smaller LLMs perform more decently compared to their results on code correctness preference,
CodeFavor still further improves them by up to 16.1%, on par with or slightly surpassing the critic LLMs (by up to 4%).

Security.
The code security subset in CodePrefBench is relatively much easier given that most models achieve saturated scores,
*e.g.,* Mistral Large 2 solves 99.5% tasks.
Similar to the low performance of the human baseline,
the scores of Gemini 1.5 Pro and Gemma 2 models are surprisingly low,
with up to 47.3% of code pairs regarded as equally insecure (*e.g.,* [Figure 12](#A1.F12)),
even if the evaluation prompt in LABEL:lst:prompt-feedback asks for an absolute answer.
Nevertheless, such behavior can be benign for mitigating potential security risks from deceptive prompts with the actual purpose of assisting cyber-security attacks.
Meanwhile, small models are still improvable in this dimension.
For example, CodeFavor eliminates the uncertainty in Gemma-2-9B-Instruct and improves its security preference score by up to 89%.
For other small models, CodeFavor can still improve them by $9.2\sim 21.9$%.

Human preference.
Aligning the objective of human preference is as challenging as that in the correctness category.
The best model, Mistral Large 2, can solve 70-ish percent of tasks,
outperforming the best-evaluated proprietary model, Gemini 1.5 Pro, by $7.7\%$.
While in human preference smaller LLMs perform much better than in the correctness objective,
CodeFavor can still improve their performance in aligning human preference for code by up to 17.7%.
By checking the dataset,
the difficulty of aligning human preferences can partially come from the ambiguity and bias inherent in human annotation.
Sometimes both code candidates exhibit different advantages and disadvantages,
making human preference just one of many possible reasonable judgments, rather than the definitive answer.

[type=table,width=.55]
Norm. Cost
Accuracy
Human preference ($3\times$)
$1.2\times 10^{5}$ ($6.1)
73.2
Llama-3.1-405B-Instr.
$1.2\times 10^{3}$
82.2
Llama-3-70B-Instr.
$3.4\times 10^{1}$
76.1
Ours (Mistral Nemo)
$1$
76.9
Estimated per-sample cost and accuracy.

Preference cost.
In addition to preference accuracy,
[Section 3.3](#S3.SS3) lists the cost of representative approaches for evaluating tasks from CodePrefBench.
Specifically, human agreement as the most expensive approach costs $6.1 per task,
estimated based on their average annotation time and California’s minimal wage.
Llama-3.1-405B-Instruct, with the best overall performance,
is two orders of magnitude cheaper than human preference.
While Llama-3-70B-Instruct is $7.4\%$ weaker than the 405B model,
it is fairly cost-effective for being cheaper by $35.3\times$.
Furthermore, CodeFavor-enabled models present the best cost-effectiveness.
For example, our classification model fine-tuned based on Mistral Nemo Instruct is five orders of magnitude cheaper than human preference and is $34\times$ cheaper than Llama-3-70B-Instruct while achieving no worse preference results.

### 3.4 Controlled Experiments

**Table 3: CodePrefBench results of CodeFavor models using different training data and modeling.**
|  |  | Correctness | Efficiency | Security | Avg. | Human Pref. |
| --- | --- | --- | --- | --- | --- | --- |
| Mistral Nemo Instruct (12B) |  | 51.4 ($\pm$1.2) | 69.7 ($\pm$0.4) | 82.9 ($\pm$7.5) | 68.0 | 66.2 |
| Commit-Instruct | Classification | 54.5 | 79.3 | 96.1 | 76.6 | 65.5 |
|  | Generation | 48.2 | 74.4 | 96.6 | 73.1 | 66.9 |
| Critic-Evol | Classification | 59.8 | 70.5 | 95.7 | 75.3 | 62.1 |
|  | Generation | 56.8 | 77.0 | 96.6 | 76.8 | 61.4 |
| Data Mixture | Classification | 63.0 | 68.8 | 95.2 | 75.6 | 62.1 |
|  | Generation | 58.2 | 77.0 | 96.1 | 77.1 | 64.1 |
| Model Merging | Classification | 58.0 | 76.1 | 96.6 | 76.9 | 64.1 |
|  | Generation | 58.8 | 77.8 | 96.6 | 77.7 | 66.9 |
| Gemma-2-9B-Instruct |  | 52.4 ($\pm$6.1) | 75.1 ($\pm$1.6) | 52.7 ($\pm$47.3) | 60.1 | 64.1 ($\pm$0.7) |
| Commit-Instruct | Classification | 52.3 | 71.9 | 82.1 | 68.8 | 63.4 |
|  | Generation | 51.8 | 80.1 | 95.1 | 75.3 | 60.7 |
| Critic-Evol | Classification | 55.5 | 74.7 | 86.5 | 72.2 | 62.1 |
|  | Generation | 57.9 | 72.2 | 97.6 | 75.9 | 64.1 |
| Data Mixture | Classification | 54.8 | 73.9 | 87.9 | 72.2 | 63.4 |
|  | Generation | 59.2 | 76.7 | 97.6 | 77.8 | 63.4 |
| Model Merging | Classification | 56.8 | 75.3 | 92.3 | 74.8 | 67.6 |
|  | Generation | 57.0 | 78.7 | 96.6 | 77.4 | 64.1 |
| Llama 3-8B-Instruct |  | 49.5 ($\pm$0.9) | 71.9 | 90.3 ($\pm$0.5) | 70.6 | 58.6 |
| Commit-Instruct | Classification | 54.4 | 71.0 | 93.7 | 73.0 | 65.5 |
|  | Generation | 48.9 | 73.0 | 94.2 | 72.1 | 66.2 |
| Critic-Evol | Classification | 58.3 | 71.3 | 90.3 | 73.3 | 57.9 |
|  | Generation | 58.3 | 74.4 | 93.7 | 75.5 | 69.0 |
| Data Mixture | Classification | 58.5 | 66.2 | 90.8 | 71.8 | 62.1 |
|  | Generation | 56.8 | 73.6 | 94.7 | 75.0 | 66.2 |
| Model Merging | Classification | 58.0 | 73.0 | 95.2 | 75.4 | 62.8 |
|  | Generation | 58.2 | 75.0 | 98.6 | 77.2 | 69.0 |
| Mistral-7B-Instruct-v0.3 |  | 48.5 ($\pm$1.5) | 66.6 ($\pm$0.1) | 78.5 ($\pm$9.4) | 64.5 | 58.3 ($\pm$1.0) |
| Commit-Instruct | Classification | 55.5 | 69.3 | 83.1 | 69.3 | 61.4 |
|  | Generation | 48.0 | 73.3 | 88.4 | 69.9 | 66.2 |
| Critic-Evol | Classification | 64.1 | 64.8 | 94.7 | 74.5 | 61.4 |
|  | Generation | 57.7 | 72.4 | 88.4 | 72.9 | 58.6 |
| Data Mixture | Classification | 59.5 | 69.3 | 91.8 | 73.5 | 60.7 |
|  | Generation | 61.7 | 73.6 | 92.8 | 76.0 | 62.8 |
| Model Merging | Classification | 62.4 | 64.8 | 95.7 | 74.3 | 60.7 |
|  | Generation | 57.1 | 77.3 | 90.3 | 74.9 | 66.9 |

This section rigorously studies the design choices in CodeFavor via controlled experiments.

Training data.
As we have two sources of training datasets,
in [Table 3](#S3.T3) we study their training effect individually (*i.e.,* “Commit-Instruct” and “Critic-Evol”) and in combination (*i.e.,* “Data Mixture”).
Comparing Commit-Instruct with Critic-Evol,
models trained by Critic-Evol tend to achieve better overall performance, particularly in the correctness category.
For example, within the classifier modeling, Critic-Evol surpasses Commit-Instruct by $6.1\sim 15.5$% on the correctness category, and the overall improvement (*i.e.,* “Avg.” column) can be up to 7.5%.
Meanwhile, when using the classification modeling,
Commit-Instruct-enabled models can perform better in the preference for code efficiency, with up to 12.5% improvement.
Moreover,
data mixture can further improve the effectiveness of model-based preference, especially when using generation modeling,
with up to 8.7% and 4.3% improvement over Commit-Instruct and Critic-Evol respectively.
The performance trend correlates with the training sample sizes,
indicating that the more training data, the better performance.

Data mixture v.s. model merging.
In addition to data mixture, we also explore co-utilizing both training datasets via model merging (Wortsman et al., ),
by averaging the weights of two models trained by individual datasets.
Model merging yields better evaluation results for all trained classification models,
with $1.1\sim 5.0\%$ improvements. Within the generation modeling, model merging also surpasses or stays on par with data mixture results for all model types except for the Mistral 7B series.

**Table 4: Controlled experiments on input prompts.**
|  | Experiment | Correctness | Efficiency | Security | Avg. |
| --- | --- | --- | --- | --- | --- |
| Mistral Nemo Instr. | Data Mixture   Reference | 63.0 | 68.8 | 95.2 | 75.7 |
|  | Aspect-specific $\to$ Empty criteria | 64.8 | 64.5 | 82.6 | 70.6 |
|  | Aspect-specific $\to$ General criteria | 61.4 | 70.2 | 92.3 | 74.6 |
|  | Trained w/o & Eval. with comments | 59.1 | 69.3 | 95.7 | 74.8 |
|  | Trained with & Eval. with comments | 52.1 | 64.5 | 94.7 | 70.4 |
|  | Trained with & Eval. w/o comments | 55.8 | 57.4 | 94.2 | 69.1 |
| Mistral-7B-Instruct | Data Mixture   Reference | 59.5 | 69.3 | 91.8 | 73.5 |
|  | Aspect-specific $\to$ Empty criteria | 55.0 | 60.8 | 73.9 | 63.2 |
|  | Aspect-specific $\to$ General criteria | 58.2 | 65.3 | 91.8 | 71.8 |
|  | Trained w/o & Eval. with comments | 53.3 | 67.6 | 90.3 | 70.4 |
|  | Trained with & Eval. with comments | 60.5 | 67.6 | 79.2 | 69.1 |
|  | Trained with & Eval. w/o comments | 63.2 | 60.2 | 80.2 | 67.9 |

Classification *v.s.* generation.
[Table 3](#S3.T3) also compares the output representation between classification and generation.
One qualitative trend is that classifier modeling often leads to higher scores in the preference for code correctness while the generation modeling tends to bring more holistic improvement leading to a higher overall store.
For example, within the 16 comparisons in [Table 3](#S3.T3),
the classification modeling outperforms the generation modeling 9 times in the code correctness objective,
whereas the generation modeling surpasses the classification modeling 13 times in the average score.

Criteria.
[Table 4](#S3.T4) studies the impact of criteria in the prompt given to CodeFavor models.
In our evaluation, by default we let the criterion statement be objective-specific.
Specifically, using an empty criterion substantially decreases the preference accuracy, especially for code security (*i.e.,* by $13.2\sim 19.5\%$).
Using a generalist criterion can also lightly degrade the overall performance by up to 2.3%.
These findings suggest using fine-grained, domain-specific criterion statements for code preference.

To comment or not to comment?
[Table 4](#S3.T4) further studies how code comments impact the code preferences of CodeFavor models in both training and inference.
Our default setting as the baseline is both trained and evaluated without code comments.
Specifically,
enabling code comments when evaluating our default models (*i.e.,* trained without comments),
we observe a $6.2\sim 10.4\%$ drop in the preference accuracy for code correctness,
while other dimensions are barely impacted.
Meanwhile, if we both train and evaluate CodeFavor models with code comments,
a broader degradation is observed with $6\sim 7\%$ drop in the overall preference accuracy.
Furthermore, evaluating the comment-trained CodeFavor models without code comments presents an even worse decrease in overall accuracy at $7.6\sim 8.7\%$.
These results suggest that code comments may negatively affect model preferences,
possibly due to LLMs’ self-bias (Chiang et al., ),
decorating faulty code with “good-looking” comments.

**Table 5: Impact of draft and critic models in training with Critic-Evol.**
|  | Draft LLM | Critic LLM | Filtered | Correctness | Efficiency | Security | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral Nemo | 8B | 70B | 17.9% | 59.8 | 70.5 | 95.7 | 75.3 |
|  | 8B | 8B | 27.2% | 58.9 | 58.8 | 87.0 | 68.2 |
|  | 70B | 70B | 21.6% | 60.7 | 70.2 | 89.4 | 73.4 |

Draft models and critic models.
While our Critic-Evol default setting uses a smaller draft model (8B) and a larger critic model (70B),
[Table 5](#S3.T5) explores circumstances when using the same draft and critic models for synthesizing preference pairs.
First, using the same draft and critic models leads to a higher filtering rate,
meaning that more initial attempts are deemed “good enough” and thus not proceeding to the revision phase.
This result is consistent with prior findings on LLM’s self-bias (Xu et al., ; Li et al., ),
*i.e.,* LLM judges tend to flavor their own generations.
Meanwhile, there is a $2.5\sim 9.4\%$ drop on the overall performance when using the same draft and critic models in Critic-Evol, yet it seems to be benign for the performance in the correctness category.

## 4 Related Work

Preference optimization has been a de facto step in post-training to align LLMs for generating helpful and safe content.
In this step, the policy model is trained over samples labeled preference objectives (*e.g.,* human preference) using various offline (Zhao et al., ; Rafailov et al., ; Meng et al., ; Yuan et al., ; Azar et al., ) and online algorithms (Xiong et al., ; Dong et al., ; ).
While the preference optimization methods are effective,
a major step in the loop is to collect and label preference data,
and our work falls into this dimension with a focus on the understudied code domain.
These preference data, *e.g.,* a pair of preferred and rejected responses in DPO (Rafailov et al., ),
in addition to being directly used for preference optimization,
can also be indirectly used to train a preference model (*i.e.,* preference learning like our work) for extensively labeling preferences (Zhao et al., ).

The raw responses to construct preference data can be sampled from the LLM under preference tuning (*i.e.,* different output responses for the same input prompt),
or external sources, such as existing human data or external LLM samples.
These samples are then ranked/scored via preference objectives such as human annotation (Ouyang et al., ),
LLM feedbacks (Cui et al., ; Weyssow et al., ; McAleese et al., ),
code execution (Zhang et al., ; ),
and preference models (Zhao et al., ; Dong et al., ; Wang et al., ; Dong et al., ).
Functionality-wise, techniques for training LLM-as-a-Judge (Kim et al., ; ) from scratch can also be applied to training a preference model and vice versa.

Specifically, our technique focuses on the understudied code generation domain (Chen et al., ; Liu et al., ),
whose preference principles can be more specialized (*e.g.,* efficiency and security) and difficult to label than the general human preference for natural language.
As a closely related work,
Weyssow et al. () score code snippets by employing a group of prominent LLMs as judges, following Cui et al. (),
whereas our work covers how to train LLM-based code raters and curate corresponding preference data from scratch.
McAleese et al. () train a CritiGPT to catch bugs in code in the form of LLM feedback,
which helps AI trainers provide more precise feedback in the RLHF process.
Our study confirms their main findings,
*e.g.,* human preference can be imperfect and even suboptimal compared to LLM-based preference.
Furthermore, our study provides extensive insights by expanding the studied code criteria beyond correctness (*e.g.,* efficiency and security),
quantifying human cost and confidence, and evaluating a comprehensive set of models.
Data-wise, CriticGPT applies bug injection (Just, ; Roy et al., ) techniques with human assistance,
whereas CodeFavor collects contrasting code pairs from code commits and revisions.
Nonetheless, we think CriticGPT can be used as a critic model in Critic-Evol to provide precise revisions.

## 5 Conclusion

In this paper, we studied human and LLM preferences for code generation. We introduced CodeFavor, a novel framework for training pairwise code preference models using synthetic code evolution data, derived from code commits and LLM critiques. For evaluation, we curated CodePrefBench, a benchmark comprising of 1,364 high-quality code preference tasks. This enables us to investigate
*(i)* the alignment of human and LLM-based preferences with correctness, efficiency, and security, and
*(ii)* the consistency of LLM-based preferences with human preferences.

Our evaluation demonstrates the effectiveness of CodeFavor:
CodeFavor fine-tunes instruction-following models, significantly improving their abilities to learn code preferences.
CodeFavor is also cost-effective, achieving on-par performance models of $9\times$ more parameters while being $34\times$ cheaper.
Despite the high cost of human-based code preference evaluation,
our results reveal that human preferences can be sub-optimal for non-functional objectives.
Finally, our controlled experiments provide a comprehensive validation of the advantages and limitations of design choices within CodeFavor.

## Acknowledgement

We thank the annotation team for their help in data labeling.
We also thank Haoxiang Wang, Wei Xiong, Federico Cassano, Yifeng Ding, Jun Yang, and Chengxiao Wang for their insightful feedback and Anoop Deoras for the leadership support.

## Appendix A Appendix

2Contents

### A.1 Prompting

We showcase our prompt implementation for synthetic data generation via concrete examples below:

Commit-Instruct.
[Figure 2](#A1.F2) details [Figure 1](#S2.F1) by exemplifying the prompt we implemented for Commit-Instruct.
Given a raw commit that replaces string literals to systematic enum-like constants,
we first ask the critic model to explain the code changes.
The model then responds with the observed key changes and is asked by the user if the code change improves certain code properties.
The critic model answers “[YES]” by pointing out that the code change improves maintainability and readability.
Next, we instruct the critic model to generate a well-formatted version of synthetic preference data including an instruction, the criteria, a code preference pair, and feedback to explain the improvements.

Meanwhile, [Figure 3](#A1.F3) illustrates a case when the critic model filters out a code commit whose improvement is regarded as unclear.
Specifically, the demonstrated code commit changes the original code to import a package in a try-catch block, rather than doing it directly.
This prevents execution termination when the package is not needed.
However, the critic model thinks such a change is minor and thus turns down the commit being processed to synthetic preference data.

Figure: Figure 2: Exemplifying prompts in Commit-Instruct for generating preference code pairs.

Figure: Figure 3: A filtered commit in Commit-Instruct for not being clearly useful.

Critic-Evol.
[Figure 4](#A1.F4) demonstrates the prompting for Critic-Evol.
Specifically, we implemented the Critic-Evol prompt in single-turn using few-show prompting.
The prompt includes five few-shot examples (*e.g.,* the square_root example),
followed by an instruction extracted from the Self-OSS-Instruct dataset and an initial coding solution provided by Llama3-8B-Instruct.
Next, the critic model Llama3-70B-instruct is prompted to generate reflection, criteria, an improved version of code, and feedback to form synthetic code preference data.
More specifically, in the reflection section,
the critic model is expected to explain the initial implementation and reason for potential improvements.
If the model thinks there is little to improve,
the generation stops here and this sample is filtered out for synthetic preference data generation.
In the case of [Figure 4](#A1.F4),
the model suggests that the get_all_words function can be implemented using dictionary comprehension which is more concise and efficient.
Following this,
the critic model proposes a related criterion based on code conciseness and efficiency,
resulting in an improved version of code, *i.e.,* [ATTEMPT_2].

Figure: Figure 4: Exemplary prompt in Critic-Evol for generating preference code pairs.

### A.2 Training Setup

Environment.
We train our models using eight NVIDIA A100-40G GPUs based on Axolotl using DeepSpeed ZeRO-3 offloading (Rasley et al., ) and flash attention (Dao et al., ).

Hyper-parameters.
[Table 6](#A1.T6) lists the default hyper-parameter settings for training our preference models.
The hyper-parameter settings mostly follow the training recipes from Dong et al. ().
As a special case, we use a slightly lower learning rate of $2\times 10^{6}$ for Gemma-2 models for training stability.

**Table 6: Explored hyper-parameter settings for training code preference learning.**
| Hyper-Parameter | Value |
| --- | --- |
| Batch size | 32 |
| Sequence length | 2048 |
| Sequence packing | ✓ |
| Learning rate | $5\times 10^{-6}$ |
| Scheduler | Cosine annealing |
| with 40 warm-up steps |  |

### A.3 Additional Evaluation Setup

Environment.
By default, we run open generative models using vLLM (Kwon et al., ) in a half-floating-point precision of bfloat16.
For better accuracy (*e.g.,* some versions are suboptimal to certain models due to bugs),
we run the Mistral and Llama models using v0.5.1, Gemma-2 models using v0.6.1.post2, and other models using v0.5.3.post1.

Decision parsing for feedback LLMs.
Code preferences of raw generative LLMs, such as Claude 3.5 Sonnet,
are generated through prompting (*i.e.,* LABEL:lst:prompt-feedback) and presented in natural-language feedback.
For the ease of parsing decisions from model outputs,
LABEL:lst:prompt-feedback declares output constraints in natural language, *i.e.,* suggesting the model to provide the preference in the format of “[CODE_?] is better than [CODE_?] on the mentioned criteria.”
As such, we parse the model response by detecting keywords such as “better” and “neither” and then apply specific patterns to extract the answer.
When none of these patterns are matched or the model simply suggests either both or neither of them are good,
we mark the response undecidable and credit it for a 0.5 score, mimicking the expectation of the sampled score.
Empirically, we found that this method works well and in our case studies we did not find any wrong classifications.
Meanwhile, grammar-based constrained decoding can also be used to enforce the desired output formats, *e.g.,* acquiring the answers in JSON.
However, such strict format restrictions might negatively impact model performance (Tam et al., ), so we choose to encode the constraints in the prompt.

Figure: Listing 1: Prompt template to provide code preference from generative LLM feedback

### A.4 Case Studies of Faulty Preference

This section provides a qualitative analysis of the preference evaluation and showcases several interesting and easy-to-understand preference mistakes made by either human developers or LLMs.
It is worth noting that for clarity we simplified and trimmed some code snippets and model responses while preserving the central idea.

Figure: Figure 5: Exemplary preferences for code correctness: Claude 3.5 Sonnet and DeepSeek V2.5 both make false claims, while humans indicate correct preferences.

Figure: Figure 6: Exemplary preferences for code correctness: All models capture the “lower-case” requirement, while all human annotators miss this detail.

Figure: Figure 7: Exemplary preferences for code correctness: Both Claude 3.5 Sonnet and DeepSeek V2.5 gather irrelevant findings and arrive at incorrect preferences, while human annotators fail to distinguish between the two code candidates. The answer is that Code B is wrong when the input number is zero.

#### A.4.1 Faulty Preferences in Code Correctness

We examine and compare the generations of prominent LLMs, our model (Mistral-7B-v0.3-Instruct classification model trained with Critic-Evol), and human judgments using the code correctness dataset in CodePrefBench.
Specifically, in CodePrefBench, the oracle for code correctness is via the execution of massive test-cases (Liu et al., ).

Erroneous reasoning due to LLM hallucination.
Preference over code correctness is essentially a reasoning task.
We observe that prominent LLMs frequently make faulty preferences for code correctness due to reasoning hallucination.
For example, [Figure 5](#A1.F5) shows a task that requires extending the input string to form the shortest palindrome.
There is only a subtle difference in Line 4: the correct implementation (Code A) searches for the largest suffix palindrome from left to right whereas Code B erroneously searches it reversely.
Interestingly, while human developers consistently made the right preference,
prominent LLMs such as Claude 3.5 Sonnet and DeepSeek V2.5, as well as our models, prefer the wrong code.
Taking a closer look,
the faults originate from unsound findings in their generation.
For example, Claude 3.5 Sonnet’s generation includes a false claim, saying that “CODE_A iterates from the beginning of the string” will make the right code (Code A) “not always find the longest palindromic suffix.”
Similarly, DeepSeek V2.5 also hallucinates that Code A would incur index errors when i is 0 which is also not true:
when i is 0, the if condition in Line 5 is equivalent to that in Line 2 as string[0:] is the string itself, making the Line-5 condition never true.
In other words, if Line 5 is true when i is 0, Line 2 would also be true and has already returned.
In addition, [Figure 7](#A1.F7) also presents cases when LLMs collect irrelevant findings and use them as reasons to falsify the correct code.
Our findings double-confirm the phenomena of “Counterfeit Conundrum” proposed by Gu et al. ():
LLMs can mistakenly classify such “counterfeit” programs as correct.

While we conclude LLMs’ reasoning faults as hallucination,
a general pattern is that LLMs tend to focus on partial semantics or edge cases in the code snippet,
overlooking other related fragments from the entire function when inferring the algorithmic correctness.
This tendency frequently leads to problematic reasoning and consequently incorrect conclusions.

Human failures.
While overall human judgments largely outperform model-based solutions in code correctness preference,
they can still occasionally predict faulty preferences with consistent confidence.
Specifically, while models can struggle with reasoning over the big picture,
human judges may overlook important details in the program such as edge cases.
[Figure 6](#A1.F6) demonstrates a task to split an input string by whitespaces or commas and return the number of lower-case letters with odd ASCII values.
While all models, including ours, correctly capture the requirement of “lower-case letters,”
all three human annotators miss this detail.
Similarly, in [Figure 7](#A1.F7), annotators had a hard time distinguishing between the two code candidates, as they failed to account for the edge case of 0.

#### A.4.2 Faulty Preferences in Code Efficiency

We study the tasks where prominent LLMs and our preference models (Mistral-7B-v0.3-Instruct classification model trained with Commit-Instruct)
present inconsistent preferences in code efficiency.
Notably, the ground truth for code efficiency preference is decided by profiling compared programs over a performance-exercising test input (Liu et al., ).

Overall,
we found that while these LLMs do not seem to hallucinate their reasoning,
they sometimes miss dominant factors that can impact code efficiency.
Next, we exemplify common efficiency-impacting factors that can be misestimated by prominent LLMs:

Algorithmic complexity.
[Figure 8](#A1.F8) illustrates a preference task where the time complexity of Code A is $O(\sqrt{n})$ while that for Code B is $O(n)$.
Specifically,
Claude 3.5 Sonnet and Llama3.1-405B-Instruct can catch the differences and correctly analyze theoretical complexities.
However, Mistral Large 2’s analysis is a bit generalist and less relevant, leading to a wrong preference decision.
This shows that understanding algorithmic complexities is crucial for making precise preferences for efficient code.

Implicit and explicit statements.
Besides major differences in algorithmic complexities,
the way the program is engineered and optimized can also significantly impact the code efficiency.
Therefore,
we exemplify how prominent LLMs understand implicit and explicit implementation differences and how these differences can impact model preferences:

- 1.
Built-in functions (implicit):
[Figure 9](#A1.F9) demonstrates the efficiency superiority of using built-in Python functions compared to writing a single-pass implementation from scratch.
Calling built-in (and external) functions is considered implicit,
as their detailed implementation is unavailable in the context.
Specifically, in [Figure 9](#A1.F9), the built-in str.count() function is implemented
not only in native C (in the default CPython interpreter) but also using advanced and well-optimized algorithms(^3^33 The fast search algorithm (Lundh, ) (also known as “BMHBNFS”) and Crochemore and Perrin’s Two-Way algorithm (Crochemore & Perrin, ) optimized for longer strings.);
however, DeepSeek V2.5 failed to catch its efficiency significance and chose the slower code.
Why do prominent LLMs missimate the impact of built-in functions?
A plausible explanation is that LLMs may not have a deeper knowledge about the implementation of the implicit built-in functions,
whereas the compared manual code can directly expose optimization details with the context, attracting preferences from LLMs.
- 2.
Early returns (explicit):
As an example of explicit statements,
Code B in [Figure 10](#A1.F10) returns the results as soon as finding a replica (Line 4).
It is explicit to the model as the efficiency advantage can be inferred without external knowledge.
Specifically,
Claude 3.5 Sonnet figured out the early stop in Code B and correctly chose it for efficiency preference.
Meanwhile, DeepSeek V2.5 was concerned about the additional copies made by lst[i+1:].
While making unnecessary copies is indeed a performance killer,
its disadvantage can be covered by the advantage (early return) when duplicates exist in the early portion of the input array.

Figure: Figure 8: Exemplary preferences for code efficiency: While Claude 3.5 Sonnet is aware of the better $O(\sqrt{n})$ complexity of CODE_A, Mistral Large 2 misses the algorithmic analysis and favors CODE_B for being “straightforward.”

Figure: Figure 9: Exemplary preferences for code efficiency: DeepSeek V2.5 misses the significance of the built-in function str.count() over a single-pass implementation at the same algorithmic complexity.

Figure: Figure 10: Exemplary preferences for code efficiency: While DeepSeek V2.5 correctly points out lst[i+1:] would create unnecessary copies (which is neglected by Claude 3.5 Sonnet), the dominating factor of performance, *i.e.,* early return, is missed.

#### A.4.3 Faulty Preferences in Code Security

Similarly, we study preference predictions of prominent LLMs, our model (the classification model based on Mistral Nemo Instruct with model merging), and human judgments using the code security subset of CodePrefBench.
The code security benchmark contains secure-insecure code pairs with vulnerabilities confirmed by a static analysis detector in CyberSecEval (Bhatt et al., ).

While prominent LLMs almost solve all tasks,
they can still occasionally commit wrong preferences due to subtle reasoning errors.
For example,
[Figure 11](#A1.F11) illustrates a case that Claude 3.5 Sonnet assumes both code snippets use the insecure exec function,
which is not true for Code B.
Yet, this error might be a rare edge case for Claude 3.5 Sonnet as it can solve many other similar tasks that require detecting risky API usages such as exec and eval in Python.

In addition, as is concluded in [Section 3.3](#S3.SS3),
Gemini 1.5 Pro usually draws tied conclusions on security preference tasks.
[Figures 12](#A1.F12) and [13](#A1.F13) are two sample security tasks marked tied by Gemini 1.5 Pro,
while being clearly solved by other demonstrated models.
Specifically, [Figure 12](#A1.F12) shows that Gemini 1.5 Pro thinks both os.popen and subprocess.run are vulnerable to command injection.
This is however not true for subprocess.run,
as subprocess.run would directly call the underlying program (*i.e.,* ps) without involving the shell (Python Software Foundation, ).
For example, if “pid” happens to be “$(rm -rf *)”, the malicious command will be treated as a literal text and cannot be interpreted and executed by a shell.
In addition,
[Figure 13](#A1.F13) challenges LLMs’ to distinguish the security implications between SHA-1 and SHA-256,
where SHA-256 is currently considered much safer than SHA-1.
However,
Gemini 1.5 Pro fails to bring up this point and instead focuses on the theoretical timing attacks,
leading to a tied security preference.
These examples suggest that Gemini 1.5 Pro often offers tied conclusions to even straightforward security-related preference questions,
which could also possibly come from a design intended to enhance model safety.

Figure: Figure 11: Exemplary preferences for code security: While Mistral Large 2 can figure out the potential risk of exec for arbitrary code execution, Claude 3.5 Sonnet and CodeFavor model prefer the wrong side. Specifically, Claude 3.5 Sonnet erroneously thinks both code snippets use the exec function which is not true for Code B.

Figure: Figure 12: Exemplary preferences for code security: Both Claude 3.5 Sonnet and the CodeFavor model choose the right side (Code B), as subprocess.run is generally safe to command injection. Nonetheless, Gemini 1.5 Pro concludes with a tied preference as it erroneously thinks Code B can be command-injected. Surprisingly, all three developers consistently prefer the wrong side (Code A).

Figure: Figure 13: Exemplary preferences for code security: While most models choose the right side as they know that SHA-256 is a more secure version of SHA-1, Gemini 1.5 Pro fails to mention this point and leads to a tied conclusion. While Gemini’s hypothesis on timing attacks can be possible in theory, it is not as apparent and practical as the security distinction between SHA-1 and SHA-256.

Figure: Figure 14: Broken security preference task using the original instruction prompt in CyberSecEval, which was generated to describe the insecure code (*i.e.,* “SHA1 hash”). It can mislead model preference (*e.g.,* DeepSeek V2.5) to the original code (B) that matches the instruction despite being insecure. Its fixed prompt is presented in [Figure 13](#A1.F13).

### A.5 Quantifying Contamination

Figure: (a) Commit-Instruct-EditPack
Refer to caption: /html/2410.03837/assets/x3.png

Figure: Figure 16: Exemplary evaluation- and training-set code pair with a similarity score of 80.

Following Riddell et al. () that quantifies the contamination in evaluating code generation, we employ *surface-level* matching to measure the contamination level between the training and evaluation data.
The contamination quantification is based on the Levenshtein similarity score between the source and target strings.
We measure the code similarity of all training-evaluation code pairs.
Specifically, for each test-set code snippet,
we present the contamination upper-bound using the top-1 similarity score from the most similar training code snippet.

[Figure 15](#A1.F15) illustrates the cumulative distribution of the top-1 similarity score on two training sets created by Commit-Instruct and Critic-Evol respectively, with code snippets from all 1,364 evaluation tasks ([Table 1](#S3.T1)).
Specifically, it shows that there are only $0.1\sim 1.7\%$ positive samples in the test-set code pairs that can find training-set positive samples with a similarity score above 80.
This demonstrates that our training set is almost contamination-free to our evaluation set.
As a reference, Riddell et al. () show that
50.8% and 63.4% of code samples in the widely used code corpus dataset, *i.e.,* the Stack (Li et al., ), can reach over 80 similarity scores with ground-truth code samples in MBPP (Austin et al., ) and HumanEval (Chen et al., ) respectively.
The low contamination can be partially inherited from their seed datasets (Cassano et al., ; BigCode, ) which have been decontaminated upon creation.
Furthermore, [Figure 16](#A1.F16) showcases a training-evaluation-set pair with a similarity score of 80.
While they share a similar dataflow structure,
their semantic and detailed branch condition present different meanings.

Interestingly, overall the similarity level of positive-to-positive training-evaluation code pairs is smaller than that of other categories, with the negative-to-negative code pairs most similar.

### A.6 Limitation and Future Work

While CodeFavor has demonstrated effectiveness in learning code preferences,
there are several potential areas of improvement to enhance the scale, applicability, and accuracy of code preference models:

- 1.
Scaling up synthetic data:
One limitation in our implementation is the scale of synthetic training data,
as our preliminary dataset only includes a total of 62,236 samples, which may be modest for model fine-tuning.
Larger-scale datasets could further improve the generalizability and robustness of preference models for code generation.
Since the idea of CodeFavor is rather general,
we plan to scale up the synthetic data generation by collecting more code commits for Commit-Instruct and more LLM samples for Critic-Evol.
Orthogonally, we may consider using multiple and more powerful models in Commit-Instruct and Critic-Evol to further improve the quality and diversity of generated synthetic data.
- 2.
Contextualized code preferences:
Code generation in real-world software development often involves broad context such as repository-level information (*e.g.,* (Ding et al., ; Zhang et al., )) and knowledge of external dependencies.
Currently, CodeFavor focuses on code preferences of self-contained code snippets, which could limit the applications of code preference models in practically complex and context-dependent scenarios.
Therefore, one future direction is to extend our framework to curate more context-sensitive code pairs for contextualized code preference learning.
- 3.
Benchmark improvements:
Our evaluation benchmark, CodePrefBench, while carefully curated, also presents potential limitations related to the diversity and practicality of candidate code samples due to their synthetic nature.
There may also be limitations due to the validity and consistency of human annotations, which are inherently subjective, particularly in assessing non-functional properties such as code efficiency.
In the future, we aim to explore real-world preference data for evaluation and address challenges in human labeling through semi-automated strategies to supplement human assessments.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: Q1: what are the backgrounds of the developers participating in the evaluation and data annotation, and how their potential biases may have affected the soundness of the approach and the evaluation? 

Q2: the larger LLMs have clear edges over the CodFavor improved small models with much lower costs than human evaluators. Why would they not be a better option than using CodeFavor to improve the smaller models?
soundness: 2
strengths: This paper presents a significant advancement in the field of code preference learning by introducing the CODEFAVOR framework, which innovatively utilizes synthetic evolution data to train models that predict meaningful code preferences. The novelty lies in its dual focus on aligning human and model preferences with verifiable code properties, addressing a critical gap in existing research. Key contributions include the development of CODEPREFBENCH, a comprehensive benchmark with 1364 curated tasks that evaluate code based on correctness, efficiency, and security, thus providing a robust evaluation framework for future studies. The results demonstrate that CODEFAVOR can enhance model accuracy by up to 28.8% while being more cost-effective than larger models, highlighting its practical significance in improving code generation assessments. Additionally, the paper sheds light on the limitations of human-based assessments, emphasizing the need for model-based approaches in evaluating non-functional code properties, which further underscores the importance of the research findings.
summary: The paper addresses the challenge of assessing code generation based on well-formed properties and aligning it with developer preferences, which has proven difficult in the context of Large Language Models (LLMs). To tackle this issue, the authors propose CODEFAVOR, a framework designed to train pairwise code preference models using synthetic evolution data, including code commits and critiques. Additionally, they introduce CODEPREFBENCH, a benchmark consisting of 1364 curated code preference tasks that evaluate three key properties: correctness, efficiency, and security, alongside human preferences. The main results indicate that CODEFAVOR significantly enhances the accuracy of model-based code preferences by up to 28.8%, while also demonstrating that these models can perform comparably to those with 6 to 9 times more parameters, all while being 34 times more cost-effective. Furthermore, the study highlights the limitations of human-based code preference assessments, revealing that a substantial percentage of tasks remain unsolved despite considerable time investment.
weaknesses: The problem formulation/setting can be improved in terms of clarity, motivation, and realism. The framework is proposed to serve code assessment purposes, i.e., judging automatically which version of code generated by a model from a prompt is preferred (i.e. more correct/secure/efficient) between a pair of two versions. The questions are (1) in what scenarios would these two versions be available, and (2) how realistic it is that there are such strong and discriminative contrasts between the two versions (i.e., correct versus wrong, fast versus slow, secure versus vulnerable). In a typical use scenario of LLMs for code generation, developers may feed the LLM with a prompt and get a response. Should they always ask the model for two responses? If so, the cost would double. More importantly, it is probably unlikely that there is a such contrast between the two versions of code generated from the same prompt---e.g., the two versions could be similarly good or bad. Learning preferences in a strongly differentiable pair of two responses does not seem to be realistic. If so, the paper may want to provide supporting evidence that this is the case. Or the problem itself is not motivated convincingly. 

Another primary weakness is the heavy reliance on synthetic evolution data for training the CODEFAVOR framework. While synthetic data can be useful, it may not fully capture the complexities and nuances of real-world coding scenarios. This limitation raises concerns about the generalizability of the model's performance in practical applications, as the evaluation may not reflect actual developer preferences or code behavior in diverse environments.

The paper acknowledges the prohibitive costs and limitations of human-based code preference assessments, noting that despite significant time investment, a substantial percentage of tasks remain unsolved (15.1% to 40.3%) . This suggests that human evaluators may struggle with certain tasks, which could undermine the reliability of the human preference data used for comparison. The paper could benefit from a more in-depth exploration of these limitations and their implications for the overall findings.

The paper mentions that the reliability of using large language models (LLMs) as evaluators often hinges on their reasoning capabilities, which can be subject to inherent biases . This raises questions about the objectivity of the model-based preferences derived from LLMs, as biases could skew the results and affect the alignment with human preferences. A more thorough examination of potential biases and their impact on the findings would strengthen the paper's arguments.

The evaluation framework, CODEPREFBENCH, focuses on three specific properties: correctness, efficiency, and security. While these are important aspects, the paper does not justify the choices, among various other quality aspects of code, such as maintainability or readability. Also, it seems that each of these chosen properties is separately considered, yet in real-world scenarios developers need to balance multiple factors at the same time when choosing which code to adopt (e.g., code that is both secure and correct). The interplay among these potentially competing factors is not considered in the approach nor in the evaluation.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: (1) In Table 1, to validate the effectiveness of the developed training framework, might it be helpful to add some baseline training approaches which also train the LLMs using the same training data used by CODEFAVOR?

(2) In Table 1, might it be helpful to understand whether CODEFAVOR can further improve these Open-Weight Models with larger sizes?

(3) Would the developed approach also be effective, if the developed approach is applied to some other LLMs for code related tasks (e.g., Code Llama, Lemur)?
soundness: 2
strengths: (1) The paper is well written and easy to follow.

(2) The paper introduces a benchmark which can potentially be used by future papers.

(3) The developed approach is evaluated using multiple LLMs, showing that the developed approach is generally effective.

(4) The developed approach has good intuitions.
summary: The paper proposes CODEFAVOR, a framework for training pairwise code preference models from synthetic evolution data, including code commits and code critiques. To evaluate code preferences, the paper introduces CODEPREFBENCH, a benchmark comprising 1364 rigorously curated code preference tasks to cover three verifiable properties - correctness, efficiency, and security - along with human preference. The evaluation shows that CODEFAVOR holistically improves the accuracy of model-based code preferences.
weaknesses: (1) In Table 1, it seems that the approaches in the rows are either LLMs, or LLMs with the training framework developed in this paper. To validate the effectiveness of the developed training framework, might it be helpful to add some baseline training approaches which also train the LLMs using the same training data used by CODEFAVOR?

(2) In Table 1, considering that there is still a gap between the Open-Weight Models and Our Models and Baselines (i.e., LLMs used with CODEFAVOR), might it be helpful to understand whether CODEFAVOR can further improve these Open-Weight Models with larger sizes?

(3) It might be helpful if the paper can show the effectiveness of the developed approach when the approach is applied to some other LLMs for code related tasks (e.g., Code Llama, Lemur).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Do you have concerns that the synthetic data generation methods, Commit-Instruct and Critic-Evol, may not fully ensure code correctness? If not, this raises another question: the quality of synthetic data is highly dependent on the LLM used to generate it. How do the experiments demonstrate that CODEFAVOR’s performance gains are due to the framework itself rather than simply distilling knowledge from a stronger LLM (Llama-3-70B-Instruct)?
- Could you provide more details on the inference process for the evaluation results in Table 3? Specifically, how many samples were created for each problem, what temperature was used, and are the results statistically significant?
- Could you elaborate on any aspects that emphasize the novelty of your work compared to previous studies?
soundness: 2
strengths: * This paper contributes two code preference synthetic dataset and a CODEPREFBENCH, a collection of 1,364 carefully curated preference tasks, To evaluate code preferences labeled by various approaches.
* This paper comprehensively quantify and conduct case studies on code preferences derived from human developers and LLMs.
* CODEFAVOR models can match the preference accuracy of models that are larger by 6∼9×, while being cheaper by 34×
summary: The paper proposes CODEFAVOR, a framework for training pairwise code preference models using synthetic evolution data generated from code commits and LLM critiques. This approach addresses the challenge of aligning code generation with developer preferences, focusing on correctness, efficiency, and security through a benchmark called CODEPREFBENCH, which includes 1,364 preference tasks. CODEFAVOR models achieve comparable performance to much larger models while being more cost-effective, and experiments reveal that human preferences often fall short in non-functional objectives like efficiency and security. The study provides insights into balancing model and human preferences, highlighting the potential limitations and strengths of each approach​.
weaknesses: - The approach to synthetic data generation lacks originality, as creating datasets from git commits [1,6] and evolving from sampled code[2,3] are common practices in the field.
- The pairwise modeling approach is also not particularly novel; using pairwise prompts, criterion-based prompting, and classification or generation labels [4,5,7] have been previously explored in other studies.
- Additionally, there is concern that synthetic data generation may not fully ensure code correctness, as it heavily depends on the LLM used for critique and generation. The chosen model, Llama3-70B-Instruct, is relatively weak compared to state-of-the-art models and limited to only this single model.
- Finally, it is challenging to determine whether the performance gains following CODEFAVOR training are due to the distillation of knowledge from stronger LLMs used in data generation or from the CODEFAVOR training itself.


1. Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." arXiv preprint arXiv:2310.06770 (2023).
2. Luo, Ziyang, et al. "Wizardcoder: Empowering code large language models with evol-instruct." arXiv preprint arXiv:2306.08568 (2023).
3. Wei, Yuxiang, et al. "Magicoder: Empowering code generation with oss-instruct." Forty-first International Conference on Machine Learning. 2024.
4. Dong, Yi, et al. "Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf." arXiv preprint arXiv:2310.05344 (2023).
5. Wang, Zhilin, et al. "Helpsteer: Multi-attribute helpfulness dataset for steerlm." arXiv preprint arXiv:2311.09528 (2023).
6. Ding, Yangruibo, et al. "Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion." Advances in Neural Information Processing Systems 36 (2024).
7. Qin, Zhen, et al. "Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting." Findings of the Association for Computational Linguistics: NAACL 2024. 2024.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. The Security score of human developers in Table 3 is only 59.7. Does this indicate that humans are not proficient at judging code security, even similar to random selection? 

2. Could you further explain “Scores within 1 percentage point of the highest” in Table 3, as well as the detailed measurement method for "uncertain responses"?

3. The authors discovered that code comments may negatively affect model preferences, which is a bit strange and may be harmful to real-world applications. Is it possible to result from the class imbalance in comments (e.g., a higher proportion of comments in positive examples)? Could you provide the number of comments in positive and negative examples in the training and testing sets?
soundness: 2
strengths: - The synthetic dataset construction method (commits and critiques) is technically sound and novel to me.

- The authors conducted a comprehensive evaluation of the method. In addition to correctness, which is the focus of many traditional code generation studies, the authors also assess efficiency, security, and human developer preference.

- The authors put significant effort into the formatting of images and tables, which enhances the readability of the paper.
summary: This paper aims to enable LLMs to better assess the quality of two code snippets by constructing a synthetic pairwise code preference dataset. The dataset is built using code commits (with pre- and post-commit versions as contrastive samples) and code critiques (the code snippet improved by a superior LLM as a contrastive sample). The authors have built benchmarks in Correctness, Efficiency, Security, and Human Preference to test the effectiveness of the proposed method.
weaknesses: 1. My main concern is that the authors overlook many works on code ranking and do not provide any experimental comparison. Many statements, such as "learning code preferences has been *largely under-explored*", "the *first* open recipe to train pairwise code preference models", and "understudied code generation domain", appear to overclaim. To name a few: 

- A basic, training-free baseline is to compare the mean log probability of two code snippets and select the one with the highest probability, as in [1]. Furthermore, [2] also uses model likelihood for code selection. 
- Some research also explores training classifiers to choose the better code, as in [3].
- The authors did not compare their work with the dataset in [4] mentioned in the Related Work section.
- In addition, but less importantly, to better contextualize the paper, some words about recent advances in execution-based code selection [5,6,7,8] would be appreciated. Particularly, [8] also employs a generation task similar to this paper. Considering that the work is recent, this comparison is not necessarily required.

Since the authors only reported the performance of the backbone LLMs and lacked empirical comparisons with advanced code selection methods, it is difficult to determine the relative improvement level of this work within related studies.

2. Some training details in the paper require further clarification. For instance, does the classifier task operate on the next token in Equation (1)? If so, considering that the label tokens ("A" or "B") and the criterion $c$ are tightly connected without a delimiter or explicit prompt, how does the LLM recognize where the criterion ends to output label tokens?

3. Since the authors collected both the training set and testing benchmarks, it's unclear whether they took decontamination steps to prevent test set leakage. If no decontamination was performed, analyzing the potential overlap between the training and test sets would be beneficial.

**Minor comments**

- The caption for Listing 1 is missing a period at the end.
- It would be better to place Equation (2) right after line 141.

**References**

[1] Evaluating Large Language Models Trained on Code, https://arxiv.org/abs/2107.03374

[2] Coder Reviewer Reranking for Code Generation, ICML 2023.

[3] Fault-Aware Neural Code Rankers, NeurIPS 2022.

[4] CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences.

[5] CodeT: Code Generation with Generated Tests, ICLR 2023.

[6] Natural Language to Code Translation with Execution, EMNLP 2022.

[7] B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests, ASE 2024.

[8] Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates, ASE 2024.

Meta-review: This paper is on the problem space of code generation using LLMs with a focus on aligning the code with developer preferences. The paper develops an approach for training pairwise code preference models using synthetic data and introduces a benchmark that considers correctness, efficiency, and security along with human preferences. Experimental evaluation shows improvement in accuracy of code preference model.

The reviewers' appreciated the importance of this research problem and contributions on both benchmark and preference model. They also asked a number of questions ranging from motivation to problem formulation and experimental methodology. Some of the questions were answered by the author rebuttal. There are two important outstanding concerns that were raised by Reviewer 4Aq4 which were not addressed.
1. Motivation is not strong and/or unclear.
2. The realistic use-case of this study from a software developers' point of view.

Therefore, I'm recommending to reject this paper and strongly encourage the authors' to improve the paper based on the feedback from reviewers' for resubmission.

Decision: Reject