Learning Code Preference via Synthetic Evolution

Authors: Jiawei Liu, Thanh V Nguyen, Mingyue Shang, Hantian Ding, Xiaopeng Li, Yu Yu, Varun Kumar, Zijian Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Code Generation, Large Language Model, Preference Learning, Evaluation

Abstract:
Large Language Models (LLMs) have recently demonstrated remarkable coding capabilities.
However, assessing code generation based on well-formed properties and aligning it with developer preferences remains challenging.
In this paper, we explore two key questions under the new challenge of code preference learning:
(i) How do we train models to predict meaningful preferences for code? and
(ii) How do human and LLM preferences align with verifiable code properties and developer code tastes?
To this end, we propose CodeFavor,
a framework for training pairwise code preference models from synthetic evolution data,
including code commits and code critiques.
To evaluate code preferences,
we introduce CodePrefBench, a benchmark comprising 1364 rigorously curated code preference tasks to cover three verifiable properties—correctness, efficiency, and security—along with human preference.
Our evaluation shows that CodeFavor holistically improves the accuracy of model-based code preferences by up to $28.8$%.
Meanwhile, CodeFavor models can match the performance of models with $6\sim 9\times$ more parameters
while being $34\times$ more cost-effective.
We also rigorously validate the design choices in CodeFavor via a comprehensive set of controlled experiments.
Furthermore, we discover the prohibitive costs and limitations of human-based code preference:
despite spending 23.4 person-minutes on each task, $15.1\sim 40.3$% of tasks remain unsolved.
Compared to model-based preference,
human preference tends to be more accurate under the objective of code correctness,
while being sub-optimal for non-functional objectives.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730125361587

thread_type:
official_review

<thread_object>
creator: reviewer_cqfq

<content>
summary:
This paper aims to enable LLMs to better assess the quality of two code snippets by constructing a synthetic pairwise code preference dataset. The dataset is built using code commits (with pre- and post-commit versions as contrastive samples) and code critiques (the code snippet improved by a superior LLM as a contrastive sample). The authors have built benchmarks in Correctness, Efficiency, Security, and Human Preference to test the effectiveness of the proposed method.

strengths:
- The synthetic dataset construction method (commits and critiques) is technically sound and novel to me.

- The authors conducted a comprehensive evaluation of the method. In addition to correctness, which is the focus of many traditional code generation studies, the authors also assess efficiency, security, and human developer preference.

- The authors put significant effort into the formatting of images and tables, which enhances the readability of the paper.

weaknesses:
1. My main concern is that the authors overlook many works on code ranking and do not provide any experimental comparison. Many statements, such as "learning code preferences has been *largely under-explored*", "the *first* open recipe to train pairwise code preference models", and "understudied code generation domain", appear to overclaim. To name a few: 

- A basic, training-free baseline is to compare the mean log probability of two code snippets and select the one with the highest probability, as in [1]. Furthermore, [2] also uses model likelihood for code selection. 
- Some research also explores training classifiers to choose the better code, as in [3].
- The authors did not compare their work with the dataset in [4] mentioned in the Related Work section.
- In addition, but less importantly, to better contextualize the paper, some words about recent advances in execution-based code selection [5,6,7,8] would be appreciated. Particularly, [8] also employs a generation task similar to this paper. Considering that the work is recent, this comparison is not necessarily required.

Since the authors only reported the performance of the backbone LLMs and lacked empirical comparisons with advanced code selection methods, it is difficult to determine the relative improvement level of this work within related studies.

2. Some training details in the paper require further clarification. For instance, does the classifier task operate on the next token in Equation (1)? If so, considering that the label tokens ("A" or "B") and the criterion $c$ are tightly connected without a delimiter or explicit prompt, how does the LLM recognize where the criterion ends to output label tokens?

3. Since the authors collected both the training set and testing benchmarks, it's unclear whether they took decontamination steps to prevent test set leakage. If no decontamination was performed, analyzing the potential overlap between the training and test sets would be beneficial.

**Minor comments**

- The caption for Listing 1 is missing a period at the end.
- It would be better to place Equation (2) right after line 141.

**References**

[1] Evaluating Large Language Models Trained on Code, https://arxiv.org/abs/2107.03374

[2] Coder Reviewer Reranking for Code Generation, ICML 2023.

[3] Fault-Aware Neural Code Rankers, NeurIPS 2022.

[4] CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences.

[5] CodeT: Code Generation with Generated Tests, ICLR 2023.

[6] Natural Language to Code Translation with Execution, EMNLP 2022.

[7] B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests, ASE 2024.

[8] Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates, ASE 2024.

questions:
1. The Security score of human developers in Table 3 is only 59.7. Does this indicate that humans are not proficient at judging code security, even similar to random selection? 

2. Could you further explain “Scores within 1 percentage point of the highest” in Table 3, as well as the detailed measurement method for "uncertain responses"?

3. The authors discovered that code comments may negatively affect model preferences, which is a bit strange and may be harmful to real-world applications. Is it possible to result from the class imbalance in comments (e.g., a higher proportion of comments in positive examples)? Could you provide the number of comments in positive and negative examples in the training and testing sets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710055250

thread_type:
official_review

<thread_object>
creator: reviewer_zrjn

<content>
summary:
The paper proposes CODEFAVOR, a framework for training pairwise code preference models using synthetic evolution data generated from code commits and LLM critiques. This approach addresses the challenge of aligning code generation with developer preferences, focusing on correctness, efficiency, and security through a benchmark called CODEPREFBENCH, which includes 1,364 preference tasks. CODEFAVOR models achieve comparable performance to much larger models while being more cost-effective, and experiments reveal that human preferences often fall short in non-functional objectives like efficiency and security. The study provides insights into balancing model and human preferences, highlighting the potential limitations and strengths of each approach​.

strengths:
* This paper contributes two code preference synthetic dataset and a CODEPREFBENCH, a collection of 1,364 carefully curated preference tasks, To evaluate code preferences labeled by various approaches.
* This paper comprehensively quantify and conduct case studies on code preferences derived from human developers and LLMs.
* CODEFAVOR models can match the preference accuracy of models that are larger by 6∼9×, while being cheaper by 34×

weaknesses:
- The approach to synthetic data generation lacks originality, as creating datasets from git commits [1,6] and evolving from sampled code[2,3] are common practices in the field.
- The pairwise modeling approach is also not particularly novel; using pairwise prompts, criterion-based prompting, and classification or generation labels [4,5,7] have been previously explored in other studies.
- Additionally, there is concern that synthetic data generation may not fully ensure code correctness, as it heavily depends on the LLM used for critique and generation. The chosen model, Llama3-70B-Instruct, is relatively weak compared to state-of-the-art models and limited to only this single model.
- Finally, it is challenging to determine whether the performance gains following CODEFAVOR training are due to the distillation of knowledge from stronger LLMs used in data generation or from the CODEFAVOR training itself.


1. Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." arXiv preprint arXiv:2310.06770 (2023).
2. Luo, Ziyang, et al. "Wizardcoder: Empowering code large language models with evol-instruct." arXiv preprint arXiv:2306.08568 (2023).
3. Wei, Yuxiang, et al. "Magicoder: Empowering code generation with oss-instruct." Forty-first International Conference on Machine Learning. 2024.
4. Dong, Yi, et al. "Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf." arXiv preprint arXiv:2310.05344 (2023).
5. Wang, Zhilin, et al. "Helpsteer: Multi-attribute helpfulness dataset for steerlm." arXiv preprint arXiv:2311.09528 (2023).
6. Ding, Yangruibo, et al. "Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion." Advances in Neural Information Processing Systems 36 (2024).
7. Qin, Zhen, et al. "Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting." Findings of the Association for Computational Linguistics: NAACL 2024. 2024.

questions:
- Do you have concerns that the synthetic data generation methods, Commit-Instruct and Critic-Evol, may not fully ensure code correctness? If not, this raises another question: the quality of synthetic data is highly dependent on the LLM used to generate it. How do the experiments demonstrate that CODEFAVOR’s performance gains are due to the framework itself rather than simply distilling knowledge from a stronger LLM (Llama-3-70B-Instruct)?
- Could you provide more details on the inference process for the evaluation results in Table 3? Specifically, how many samples were created for each problem, what temperature was used, and are the results statistically significant?
- Could you elaborate on any aspects that emphasize the novelty of your work compared to previous studies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730874703580

thread_type:
official_review

<thread_object>
creator: reviewer_cc8r

<content>
summary:
The paper proposes CODEFAVOR, a framework for training pairwise code preference models from synthetic evolution data, including code commits and code critiques. To evaluate code preferences, the paper introduces CODEPREFBENCH, a benchmark comprising 1364 rigorously curated code preference tasks to cover three verifiable properties - correctness, efficiency, and security - along with human preference. The evaluation shows that CODEFAVOR holistically improves the accuracy of model-based code preferences.

strengths:
(1) The paper is well written and easy to follow.

(2) The paper introduces a benchmark which can potentially be used by future papers.

(3) The developed approach is evaluated using multiple LLMs, showing that the developed approach is generally effective.

(4) The developed approach has good intuitions.

weaknesses:
(1) In Table 1, it seems that the approaches in the rows are either LLMs, or LLMs with the training framework developed in this paper. To validate the effectiveness of the developed training framework, might it be helpful to add some baseline training approaches which also train the LLMs using the same training data used by CODEFAVOR?

(2) In Table 1, considering that there is still a gap between the Open-Weight Models and Our Models and Baselines (i.e., LLMs used with CODEFAVOR), might it be helpful to understand whether CODEFAVOR can further improve these Open-Weight Models with larger sizes?

(3) It might be helpful if the paper can show the effectiveness of the developed approach when the approach is applied to some other LLMs for code related tasks (e.g., Code Llama, Lemur).

questions:
(1) In Table 1, to validate the effectiveness of the developed training framework, might it be helpful to add some baseline training approaches which also train the LLMs using the same training data used by CODEFAVOR?

(2) In Table 1, might it be helpful to understand whether CODEFAVOR can further improve these Open-Weight Models with larger sizes?

(3) Would the developed approach also be effective, if the developed approach is applied to some other LLMs for code related tasks (e.g., Code Llama, Lemur)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731111061638

thread_type:
official_review

<thread_object>
creator: reviewer_4aq4

<content>
summary:
The paper addresses the challenge of assessing code generation based on well-formed properties and aligning it with developer preferences, which has proven difficult in the context of Large Language Models (LLMs). To tackle this issue, the authors propose CODEFAVOR, a framework designed to train pairwise code preference models using synthetic evolution data, including code commits and critiques. Additionally, they introduce CODEPREFBENCH, a benchmark consisting of 1364 curated code preference tasks that evaluate three key properties: correctness, efficiency, and security, alongside human preferences. The main results indicate that CODEFAVOR significantly enhances the accuracy of model-based code preferences by up to 28.8%, while also demonstrating that these models can perform comparably to those with 6 to 9 times more parameters, all while being 34 times more cost-effective. Furthermore, the study highlights the limitations of human-based code preference assessments, revealing that a substantial percentage of tasks remain unsolved despite considerable time investment.

strengths:
This paper presents a significant advancement in the field of code preference learning by introducing the CODEFAVOR framework, which innovatively utilizes synthetic evolution data to train models that predict meaningful code preferences. The novelty lies in its dual focus on aligning human and model preferences with verifiable code properties, addressing a critical gap in existing research. Key contributions include the development of CODEPREFBENCH, a comprehensive benchmark with 1364 curated tasks that evaluate code based on correctness, efficiency, and security, thus providing a robust evaluation framework for future studies. The results demonstrate that CODEFAVOR can enhance model accuracy by up to 28.8% while being more cost-effective than larger models, highlighting its practical significance in improving code generation assessments. Additionally, the paper sheds light on the limitations of human-based assessments, emphasizing the need for model-based approaches in evaluating non-functional code properties, which further underscores the importance of the research findings.

weaknesses:
The problem formulation/setting can be improved in terms of clarity, motivation, and realism. The framework is proposed to serve code assessment purposes, i.e., judging automatically which version of code generated by a model from a prompt is preferred (i.e. more correct/secure/efficient) between a pair of two versions. The questions are (1) in what scenarios would these two versions be available, and (2) how realistic it is that there are such strong and discriminative contrasts between the two versions (i.e., correct versus wrong, fast versus slow, secure versus vulnerable). In a typical use scenario of LLMs for code generation, developers may feed the LLM with a prompt and get a response. Should they always ask the model for two responses? If so, the cost would double. More importantly, it is probably unlikely that there is a such contrast between the two versions of code generated from the same prompt---e.g., the two versions could be similarly good or bad. Learning preferences in a strongly differentiable pair of two responses does not seem to be realistic. If so, the paper may want to provide supporting evidence that this is the case. Or the problem itself is not motivated convincingly. 

Another primary weakness is the heavy reliance on synthetic evolution data for training the CODEFAVOR framework. While synthetic data can be useful, it may not fully capture the complexities and nuances of real-world coding scenarios. This limitation raises concerns about the generalizability of the model's performance in practical applications, as the evaluation may not reflect actual developer preferences or code behavior in diverse environments.

The paper acknowledges the prohibitive costs and limitations of human-based code preference assessments, noting that despite significant time investment, a substantial percentage of tasks remain unsolved (15.1% to 40.3%) . This suggests that human evaluators may struggle with certain tasks, which could undermine the reliability of the human preference data used for comparison. The paper could benefit from a more in-depth exploration of these limitations and their implications for the overall findings.

The paper mentions that the reliability of using large language models (LLMs) as evaluators often hinges on their reasoning capabilities, which can be subject to inherent biases . This raises questions about the objectivity of the model-based preferences derived from LLMs, as biases could skew the results and affect the alignment with human preferences. A more thorough examination of potential biases and their impact on the findings would strengthen the paper's arguments.

The evaluation framework, CODEPREFBENCH, focuses on three specific properties: correctness, efficiency, and security. While these are important aspects, the paper does not justify the choices, among various other quality aspects of code, such as maintainability or readability. Also, it seems that each of these chosen properties is separately considered, yet in real-world scenarios developers need to balance multiple factors at the same time when choosing which code to adopt (e.g., code that is both secure and correct). The interplay among these potentially competing factors is not considered in the approach nor in the evaluation.

questions:
Q1: what are the backgrounds of the developers participating in the evaluation and data annotation, and how their potential biases may have affected the soundness of the approach and the evaluation? 

Q2: the larger LLMs have clear edges over the CodFavor improved small models with much lower costs than human evaluators. Why would they not be a better option than using CodeFavor to improve the smaller models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732201854660

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta Response & Submission Revision

comment:
We thank the reviewers for the inspiring comments and we have updated our submission by making the following notable changes, highlighted using purple in the updated submission PDF:

* **(Experiment) CodeFavor for larger models (Appendix A.6 / Table 9):** We applied CodeFavor to a much larger model, namely Gemma2-27B-IT, showing an overall improvement of 9%. (4Aq4, cc8R)  
* **(Experiment) Adding Bradley-Terry modeling as a baseline (Appendix A.6 / Table 10):** Using BT modeling, it achieves a suboptimal overall result compared to classification/generation, but it can achieve a substantially better correctness score specifically (cc8R)  
* **(Experiment) Applying CodeFavor to CodeLlama (Appendix A.6 / Table 11):** CodeFavor improves coding models also quite well; the untuned coding model achieves a better correctness score compared to general models of similar sizes (cc8R)  
* **(Experiment) Comment distribution in positive & negative samples (Appendix A.6 / Table 12):** The results show that the negative impact of comments does not simply come from distribution imbalance in training/evaluation sets (cqfq)  
* **(Experiment) Comparing against Logprob Mean & code scoring models (Table 3):** logprob mean leads to rather random results and CodeFavor models largely outperform the leading 8B general reward model (cqfq)  
* **(Discussion) Acknowledge missed prior work** (zRJn, cqfq)  
* **(Writing) Clarifying training details (Section 2.1):** prompt and output token are separated by special tokens defined by the chat template (cqfq)

We look forward to hearing further feedbacks from the reviewers!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732494500081

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle reminder of reviewer responses

comment:
Dear Reviewers,

As the discussion period deadline approaches, we would appreciate your feedback on our response. Your feedback would greatly contribute to our work and the ICLR community.

Thank you for your time and consideration!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733175339112

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers 4Aq4, cc8R, and zRJn:

We'd like to gently remind you that December 2nd (end of day in AoE) is the last day for reviewer feedback and we can reply to additional questions until tomorrow. Please kindly take a look at our replies and updated manuscript with extensive additional experiments and reference updates, and let us know if your concerns have been addressed.

Thank you for your time!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper is on the problem space of code generation using LLMs with a focus on aligning the code with developer preferences. The paper develops an approach for training pairwise code preference models using synthetic data and introduces a benchmark that considers correctness, efficiency, and security along with human preferences. Experimental evaluation shows improvement in accuracy of code preference model.

The reviewers' appreciated the importance of this research problem and contributions on both benchmark and preference model. They also asked a number of questions ranging from motivation to problem formulation and experimental methodology. Some of the questions were answered by the author rebuttal. There are two important outstanding concerns that were raised by Reviewer 4Aq4 which were not addressed.
1. Motivation is not strong and/or unclear.
2. The realistic use-case of this study from a software developers' point of view.

Therefore, I'm recommending to reject this paper and strongly encourage the authors' to improve the paper based on the feedback from reviewers' for resubmission.