Preference Optimization for Reasoning with Pseudo Feedback

Authors: Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, Furu Wei
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, code generation, natural language reasoning, reinforcement learning

Abstract:
Preference optimization techniques, such as Direct Preference Optimization (DPO), are frequently employed to enhance the reasoning capabilities of large language models (LLMs) in domains like mathematical reasoning and coding, typically following supervised fine-tuning. These methods rely on high-quality labels for reasoning tasks to generate preference pairs; however, the availability of reasoning datasets with human-verified labels is limited.
In this study, we introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions to reason problems as an evaluation against associated \emph{test cases}. 
We explore two forms of pseudo feedback based on test cases: one generated by frontier LLMs and the other by extending self-consistency to multi-test-case.
We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks. Specifically, using Mathstral-7B as our base model, we improve MATH results from 58.3 to 68.6, surpassing both NuminaMath-72B and GPT-4-Turbo-1106-preview. In GSM8K and College Math, our scores increase from 85.6 to 90.3 and from 34.3 to 42.3, respectively. Building on Deepseek-coder-7B-v1.5, we achieve a score of 24.3 on LiveCodeBench (from 21.1), surpassing Claude-3-Haiku.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730432965351

thread_type:
official_review

<thread_object>
creator: reviewer_djul

<content>
summary:
This paper proposes Pseudo-Feedback Preference Optimization (PFPO), a novel approach for preference optimization in large language models (LLMs) tailored to reasoning tasks in mathematical and coding domains. PFPO sidesteps the dependency on human-verified labels by generating pseudo feedback through test cases, either created by frontier LLMs or using self-consistency from the policy LLM itself. Using Llama-3.1-8B/Mathstral-7B for mathematical reasoning and Deepseek-coder-7B-v1.5 for code generation, PFPO achieved notable improvements on the MATH, GSM8K, College Math, and LiveCodeBench benchmarks.

strengths:
1. Comprehensive experiments across a range of reasoning tasks and LLM models.
2. Clear presentation and good formalization.
3. Empirical results show the method’s effectiveness.

weaknesses:
1. Limited Novelty at ICLR Level:  
   The technique of leveraging unit tests as feedback for direct preference optimization (DPO/PPO) is already established in existing research. Previous works, such as CodeRL [1], and more recent studies [2][3], have incorporated unit tests and compiler feedback within reinforcement learning (RL)/PPO frameworks for code generation. Similarly, fine-grained feedback with DPO has been applied in mathematical reasoning [4], and LLM-based feedback with DPO has been explored for code generation [5]. Although PFPO Self-iter demonstrates strong performance, its approach of utilizing LLM-generated feedback signals for DPO closely aligns with these prior works, which limits its methodological originality.

   - [1] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. C. H. (2022). CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. *Advances in Neural Information Processing Systems*, 35, 21314-21328.
   - [2] Liu, J., Zhu, Y., Xiao, K., Fu, Q., Han, X., Yang, W., & Ye, D. (2023). Rltf: Reinforcement learning from unit test feedback. arXiv preprint arXiv:2307.04349.
   - [3] Dou, S., et al. (2024). Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391.
   - [4] Lai, X., et al. (2024). Step-dpo: Step-wise preference optimization for long-chain reasoning of LLMs. arXiv preprint arXiv:2406.18629.
   - [5] Weyssow, M., Kamanda, A., & Sahraoui, H. (2024). CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences. arXiv preprint arXiv:2403.09032.

2. Insufficient Discussion of Limitations in Self-Generated Pseudo Labels:  
   The paper could benefit from a more thorough examination of the potential biases and risks associated with self-generated pseudo labels, such as model bias and the tendency to overfit synthetic feedback. A clearer analysis would provide a more robust perspective on PFPO's limitations. Additionally, connecting this limitation to the plateau effect noted in Question 1 would be insightful, as similar issues with DPO setups have been documented, as seen in [4]. Empirical evaluation or discussion on the effects of varying types and quantities (quesiton 3) of synthetic feedback could strengthen this analysis further.

questions:
1. PFPO with self-iteration clearly plateaus after several iterations. Can the authors provide more insights on what stage causes this, such as whether the quality of generated test cases plateaus or the policy model stops learning?
2. In Table 1, the baseline score for Llama-3.1-8B-base seems missing. Could this be clarified?
3. In section 4.2.2, synthetic test cases outperform ground truth, likely due to greater quantity. Since generated inputs and ground truth programs are available, why not run the programs on generated inputs to match the number of ground truth test cases, enabling a fairer comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698534028

thread_type:
official_review

<thread_object>
creator: reviewer_nhwb

<content>
summary:
This paper introduces PFPO (Pseudo-Feedback Preference Optimization) that creates pseudo feedback for reasoning tasks by framing the labeling of solutions to reason problems as an evaluation against associated test cases. They explore two forms of pseudo feedback based on test cases: one generated by frontier LLMs and the other by extending self-consistency to multi-test-case. Experiments are conducted on math reasoning and code generation and observe performance improvements.

strengths:
* The paper presents a novel approach to preference optimization by leveraging pseudo feedback.
* The experiments are well-designed and comprehensive, covering both mathematical reasoning and coding tasks. The results show substantial improvements over baseline models and even surpass some state-of-the-art models, indicating the high quality of the proposed methods.
* The paper is well-structured and clearly written. The methodology is explained in detail, and the experimental setup and results are presented in a way that is easy to follow.

weaknesses:
Please see the questions below.

questions:
* The method relies on frontier LLMs to generate pseudo feedback. How does the performance of the proposed method degrade if the frontier LLMs are not available or their performance is suboptimal?
* How does the quality of pseudo feedback (from both frontier LLMs and self-consistency) impact the final performance of the LLM? Is there a way to measure or improve this quality?
* Could the authors provide more details on the computational requirements and potential optimizations for the iterative training process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701556348

thread_type:
official_review

<thread_object>
creator: reviewer_epjk

<content>
summary:
The authors propose a technique to generate pseudo-feedback using test cases. For math problem-solving, the test cases are unique (since each math problem has a singular, numerical answer, and a model either produces or doesn’t produce that answer). For code generation, multiple test cases are produced: the paper explores both test case generation using a frontier LLM and using the model being trained.

This approach provides a new method that does not rely on the high-quality human-verified data required for DPO. Implementing this approach with Llama-3.1-8B, Mathstral-7B, and Deepseek-coder-7B-v1.5-Instruct for math and code reasoning respectively, shows improvements from 5 to over 10% across various benchmarks.

strengths:
* Creates the pseudo feedback framework in a way that combines both math and code reasoning tasks.
* Because the pipeline for generating pseudo feedback is entirely automated, the process is scalable and less expensive than recreating such a process with humans would be.
* The experiments are robust, conducted with three open LLMs, and with multiple combinations of techniques across two domains.
* The results of the experiments are not only significant, but also well-explained. The authors do a good job of properly explaining any particularly interesting/surprising results.
* The approach using frontier LLMs removes the need for ground-truth labels, enabling better use of unlabeled reasoning data for model improvement.
* The authors provide an appendix with more interesting insights and all the necessary details needed to recreate their work/expand on it, making for a meaningful contribution to the field.

weaknesses:
* The primary weaknesses I see with this approach are the limitations caused by self-consistency over self-generated pseudo feedback. I could see improvement over iterations being marginal for more challenging math/coding problems, as the self-consistent "answer" for them is likely incorrect.

questions:
* Same as the weakness — I’d love to see whether performance on a subset of more challenging problems (which could be measured by looking at problems where the base models provided an incorrect answer even after applying self-consistency) improved over time with self-generated pseudo-feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to alleviate the problem of scarcity of preference data, and propose to methods to generate pseudo-feedback in the context of reasoning problems such as math and coding by using the evaluation on test cases as the feedback.  The paper explores two forms of pseudo feedback, and convincingly demonstrate the effectiveness of the proposed methods on benchmark reasoning datasets. 

While there were slight concern on the technical novelty of the proposed methods, all reviewers agree the empirical evidence is convincing on the effectiveness of the proposed methods in practice. We thus recommend acceptance.