Preference Optimization for Reasoning with Pseudo Feedback
Authors: Fangkai Jiao
Geyang Guo
Xingxing Zhang
Nancy F. Chen
Shafiq Joty
Furu Wei
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Model
Code Generation
Natural Language Reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We develop a framework to continuously improve LLMs for reasoning and code generation through self-consistency-based pseudo feedback.

Abstract: Preference optimization techniques, such as Direct Preference Optimization (DPO), are frequently employed to enhance the reasoning capabilities of large language models (LLMs) in domains like mathematical reasoning and coding, typically following supervised fine-tuning. These methods rely on high-quality labels for reasoning tasks to generate preference pairs; however, the availability of reasoning datasets with human-verified labels is limited.
In this study, we introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions to reason problems as an evaluation against associated \emph{test cases}. 
We explore two forms of pseudo feedback based on test cases: one generated by frontier LLMs and the other by extending self-consistency to multi-test-case.
We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks. Specifically, using Mathstral-7B as our base model, we improve MATH results from 58.3 to 68.6, surpassing both NuminaMath-72B and GPT-4-Turbo-1106-preview. In GSM8K and College Math, our scores increase from 85.6 to 90.3 and from 34.3 to 42.3, respectively. Building on Deepseek-coder-7B-v1.5, we achieve a score of 24.3 on LiveCodeBench (from 21.1), surpassing Claude-3-Haiku.

Published as a conference paper at ICLR 2025
PREFERENCE OPTIMIZATION FOR REASONING WITH PSEUDO FEEDBACK
Fangkai Jiao' 1.t
Geyang Guo
Xingxing Zhang?
Nancy F. Chen
Shafiq Joty
Furu Wei?
'Nanyang Technological University
Microsoft Research
31²R, A*STAR
1 Georgia Institute of Technology
5 Salesforce Research
ABSTRACT
Preference optimization techniques, such as Direct Preference Optimization (DPO), are frequently employed to enhance the reasoning capabilities of large language models (LLMs) in domains like mathematical reasoning and coding, typically following supervised fine-tuning. These methods rely on high-quality labels for reasoning tasks generate preference pairs; however, the availability of reasoning datasets with numan-verified labels limited. In this study, we intro- duce novel approach generate pseudo feedback for reasoning tasks by framing the labeling of solutions to reason problems as an evaluation against associated test cases. We explore generated by frontier LLMs and the other by extending self-consistency to multi- test-case. We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improve- ments across both tasks. Specifically, using Mathstral-7B as our base model, we improve MATH results from 58.3 to 68.6, surpassing both NuminaMath-7225 and GPT-4-Turbo- 110 -preview. In GSM8K and College Math, our scores increase from 85.6 to 90.3 and from 34.3 42.3, respectively. Building on Deepseek.cooer-7BBv11 we achieve a score of 24.3 on LiveCodeBench (from 21.1), surpassing Claude-3-Haiki
forms of pseudo feedback based
cases: one
INTRODUCTION
Large language models (LLMs) have demonstrated exceptional capabilities in reasoning tasks such math reasoning and coding (Roziere et al. 2023 Dubey et al., 2024] Guo et al. 2024). A de facto pipeline for enhancing the reasoning capabilities of LLMs involves further exposing them to rea- soning specific data through continued pre-training or supervised fine-tuning (Roziere et al. 2023
Dubey et al. 2024, Yu al. 2023 Tang al. 2024 Dong al.) 2023). followed by preference learning techniques such as direct preference optimization (DPO: Rafailov et al. (2023) or proximal policy optimization (PPO: /Schulman al.) (2017). Both DPO and PPO depend on reliable labels for reasoning problems to generate preference pairs and train reward models (Lightman et al.) [024]
(esato al., 2022). Unfortunately, reasoning datasets with large-scale, numan-verified labels re- main limited, and scaling them through domain experts is becoming increasingly time-consuming and expensive, particularly as LLMs continue to evolve in capabilities (Burns et al., 22024) [owmmnn
al. 2022), which greatly limits the potential of preference learning methods such DPO and PPO.
Scalable oversight (Bowman et al. 2022) demonstrates that the annotation effort of human experts be significantly reduced with the assistance of non-expert LLMs. However, complete elimina- tion human annotation remains unattainable. Building on this, Khan et al. (024a) further reduced labeling costs by incorporating dehating mechanism, though this approach constrained to reason-
Work done during intermship at Microsoft Research.
'The code is released at: http://giihllooommcccccciiaiiinr

---

Published as a conference paper at ICLR 2025
ing tasks with finite answer space (e.g., multiple-choice questions). Other works have employed self-consistency-brree -basee answers or their variants as pseudo-labels to filter self- geffraeeeettdd solutions (Huang et al. 2022 Yang al. 2024c), but these methods struggle to generalize to reasoning tasks that lack explicit answer labels (e.g., coding).
To address these challenges, we frame the labeling of solutions to reasoning problems the evalu- ation of these solutions against the test cases of the problems. For tasks with explicit answer labels (e.g., mathematical reasoning and multiple-choice questions), we treat them cases with a single test pair, where the input is empty, and the output is the answer label. In contrast, for tasks without explicit answer labels (e.g.. coding), we consider them as problems with multiple test case pairs. A solution to a reasoning problem is deemed correct if and only if it passes all associated test cases. Sample solutions generated by an L.M for the same problem can be validated using the test case suite, with correct and incorrect solutions used to construct preference pairs for DPO training or to train a reward model for PPO. this paper, we propose two types of pseudo feedback (i.e., pseudo test cases) for reasoning problems, both of which eliminate the need for human experts and can be applied at scale, First, we explore pseudo feedback from frontier LLMs, where we decompose the process of creating pseudo test cases into multiple steps to ensure that each step is manageable for frontier LLMs. Intuitively, if an LLM can pass test cases carefully curated by a stronger LLM, is likely provide correct solution. Previous work [Wang et al. (2022); [nnll et al. (2024) has demonstrated that self-consistency improves the reasoning performance of LLMs.. Based on this insight, we introduce second form of pscudo feedback, utilizing self-consistency from our policy LLM, which is of vital importance when frontier LLMs are no longer available. Unlike the method Wang al. (2022), which is limited to single-test-case problems, our self-consistency feedback is designed to generalize to problems with multiple test cases. We also find that these two types of pseudo feedback complement each other and can be applied iteratively
pipeline. We conducted experiments on both mathematical reasoning and coding using pseudo feedback for preference op- timization and we observe improvements across both tasks. Specifically, using Mathstral-77 as our base model, we improved our MATH results from 58.3 to 68.6, surpassing both NuminaMath-72B and GP TorTTTuroooooo 1100--preview eww. In GSM8K and College Math, our results increased from 85.6 to 90.3 and from 34.3 to 42.3, respectively. Building on -eepseek-coder-7BB-ll.. a score of 24.3 on LiveCodeBench (from 21. 1), surpassing Claude 3 -Ha Haiku.
achieved
In nutshell, our contribution in this paper be summarized as follows:
We formulate the labeling of solutions to reasoning problems as the process of evaluating them against the associated test cases, which facilitates preference optimization.
We explore two types of pseudo feedback based on test cases: one created from frontier LLMs and the other derived from generalized self-consistency w.r.t. multiple test cases.
Experiments on mathematical reasoning and coding demonstrate the superiority of these two types of feedback. We also find they can be applied in a pipeline and iteratively to further improve the reasoning performance.
2 RELATED WORK
LLMs exhibit remarkable capabilities by tuning on high-quality data annotated by experts or more advanced models (Achiam et al. 2023). However, these external annotations can be costly, posing a challenge to further enhance model's performance. Inspired by the natural evolution process of human intelligence, researchers explore self-evolution methods (Tao et al. (024)) that enable models to autonomously acquire, refine. and learn from their own knowledge. Some works (Wang et al.) 20236 Ding et al., 2024) reformulate the training objective to directly model performance improve- ment. Others tune the model with its own responses. They first filter the model's outputs relying on ground truth labels (Zelikman et al., 2022. Wang et al. 2024b), expert annotations (Dubey et al.) 2024), or more advanced models ( (Yang et al. 2024a| Kirchner et al. 2024). and then use the result- ing refined examples for supervised or contrastive learning (Chen et al. 2024b Yuan et al. 2024). However, they still depend on external supervision and cannot extend to larger unlabeled datasets.
Recent work (Huang et al., (222)) constructs pseudo labels via consisteeey, but the improvement
is limited, possibly due model collapse (Shumailov al. 2023 Alemohammad et al. 2023
2

---

Published as a conference paper at ICLR 2025
For related work about mathematical reasoning (Wei et al. 2022;
He- Yueya et al. 2023
Chen al.
2021b
2022:
Lightman et al. 2024 Wang et al. 2024a; Jiao et al., 2024 Lai et al. 20224] Cobbe
et al.
2021b;
Li et al.] 2022a Weng al.
2022] Yu al.
2023
Luo al. [023] Mitra et al.
2024
Yue et al. 2023) and code generation (Guo et al. 2024 DeepSeek-A et al., 2024 Nijkamp
al.
2023h a Zelikman et al., 2022 Li et al.
2023a
2022b Wei al.
2024 Le et al. 2022: Liu
et al.
2023] Dou et al. 2024; Weyssow et al. 2024). we will discuss them in Appendix F duc to the
limitation of space.
3
METHOD
In reasoning tasks such mathematical reasoning and coding, the solution to problem be ver- ified using a standard answer or a set test cases. This property makes possible to automatically create preference pairs for LLM solving reasoning tasks and further improve reasoning capa- bilities of the LLM with preference optimization. However, annotating reasoning problems with answers or test cases manually expensive and time consuming. As result, this process difficult Preference Optimiza-
to executed in large scale. Therefore, we propose PFPO tion), a method to automatically create pseudo answers or test cases to facilitate preference learning. In this section, we first introduce preference optimization for reasoning in Section [.1 (assuming gold answers test cases are available). Then we will go details of PFPO, which creates pseudo answers or test cases.
3.1 PREFERENCE OPTIMIZATION FOR REASONING
Suppose we have set of reasoning problems with their test cases T: (T); T:) where T = {(1,, 01 ), (1z, 02 )..... . (ipT) 9T)) }} and (ig, 0% is the input-output pair of test case. Note that T is generalized representation for either a collection of test cases or the gold answer for prob- lem x. If I is coding problem, 7 is set of test cases to verify the correctness of the corresponding solution of x. While if x one of the other reasoning problems such as mathematical reasoning or multi-choice science questions, there is only one test case in T = {{ i, o)) and the input is empty.
For example, "compute 1+ 1" is a math question with i = as its test case input and = as its test output.
Given a reasoning problem x and its test cases T, we are ready to evaluate the correctness a solution produced by LLM my follows:
21
(o(,, oz))
where g(---) -) is a function to either execute the solution or extract the answer from y. In the most strict form, y correct solution problem x when r Otherwise (i.e., 1), y an incorrect solution. Note that in mathematical reasoning, there is only one test case and r E {0, 1).
Note that given problem and its corresponding test cases T, the process of verifying an arbitrary solution does not need any human labeling effort. We can construct preference pairs for an LLM automatically as follows. First, we use an L.LM ma to sample N solutions Y = {y1 J2......................... yN for problem and obtain their verification results R = (ri, r2, ,r,...,,,), J. To further improve To. we can use PPO (Schulman al,, 2017) to optimize these feedback online or use DPO (Rafailov al.) 2023) to do preference optimization offfine. In this work, we employ DPO due to its simplicity. Then, we create preference pairs from R and valid pairs (Ywr 31) requires Tw = 1 and 71 1.
p = [V rr 1,71 < 1, E R,Ti E R}
(2)
Given these valid preference pairs, We optimize our LLM we using the following objective:
Bog To (x) Ter (x)
(u (x)
(DPO(TTT Neel D)
log
3 log
(3)
गe The [x)
where Tee is the reference model before the DPO stage (usually it is the model of the supervised fine-tuning stage). | is hyper-parameter control the distance between my and mel-
3

---

Published as a conference paper at ICLR 2025
General LLM
Quslien Please the fulleving Pythoe function implersent audiikos atgarilaas for veclors.
Solator vy: CST Fregue
R S
Suilie GIT Prggus
Oubeut (.40
vector addia Lugttorr Laa(horr Bour
Saudic by GIT Progue
Enecate
Fest Case hepers
Oupus
Fragucct
aDD-2 31
Poliey Mad
Soltier ug: Col Proguer
87.41
Oupaat (.4)
The process of employing coffisensstteny (i.e., majority voting) to construct pseudo test cases for code generation problem. The outputs owing the highest frequency will be treated as pseudo outputs for verifying generated programs.
Figure 1:
3.2 PSEUDO FEEDBACK PREFERENCE OPTIMIZATION FOR REASONING
In this section, we introduce how to obtain pseudo feedback for reasoning problems and the method of leverage them for preference learning.
3.2. PSEUDO FEEDBACK FROM FRONTIER LLM
Single-Test-Cii Feedback For single-test-case reasoning tasks such as mathematical reasoning, the input is explicitly given in the problem itself. Therefore, do not need to create new test cases. Given a problem æ, we can use a frontier LLM to generate a solution ÿ and extract its pseudo answer g(ū,-) as our pseudo feedback (see Equation Db. The solution (T(-(>>) from our model is likely
to be correct if g(y, 0) g(7. 0):
r= 19 g(ū, 0)))
(4)
Since solutions of many reasoning datasets used for LLM supervised fine-tuning are created by frontier LLM (Taori al. 22023] Tang et al.) 2024), we can re-use the SFT datasets and extract the pseudo feedback as a free lunch.
Multi- Teiti-Test--asee Feedback For multi-test-case reasoning tasks such as coding, test cases for cod- ing problems are usually not available and manually label them are expensive. We choose to generate pseudo test cases by prompting frontier LLMs. There three steps to generate test cases as shown in Figure
Given problem x, generate input test cases I = (0 ......,... (x) by prompting genera
Step 1:
L.L..
Step 2: Generate pscudo (code) solutions y = {36 ve..... 1By3 for problem x using a frontier LLM.
Generate pseudo output test cases O = for, .g...., o'k using majority voting by executing all solutions in y for each input test case in I.
Step 3:
The output test case of corresponds to the input é is obtained as follows: after executing all pseudo solutions, we obtain set of candidate pseudo output O₂ {o(vá The output test case og is the most frequent element in O²:-
d' arg max f(o) oEO
where f(o) = [x E O x ====o]
of frequency function that gives the number of times an element
o appears in O² The resulting set pseudo test cases T" = {( of ). (īg dz),..., ( (ix. ox At this point, we can verify arbitrary solution y problem
in Equation
'Here we differentiate general L.M with the frontier one generating only the inputs is much casier compared with solving the problem itself. Thus this process does necessarily rely SOTA L.Mss.
4

---

Published as a conference paper at ICLR 2025
Pseudo Feetbbukk from Frontier LIM
Pseudo Feedback from Self-Cannistency
Miri ce The answer 16.
The answer is 7.
y
Mar answer
Yar.. The answer is
Prompt.x; Ky: What's the result of ..?
Pscudo Label: Wboxed[7]
Model #l43
Preforence Optimization
Modeli,
Solutions to Ya.-Ma
Yariom 16.
X
Preference Pair 2>
Yaston The arnswer
Figure 2: The overall training workflow of our method. For simplicity, we only show single step with outcome feedback for mathematical reasoning. Given an arbitrary prompt, we will sample multiple solutions from the current policy model and construct preference pairs according to pseudo feedback from frontier LLM or celf-consistency. Finally, the constructed preference pair will be used improve the policy model through DPO.
Note that we do choose to generate both imput and output test cases single step by prompting I.Mss, Gu et al. (204)) have pointed out that, generating the test case output based given input challenging task, which requires strong reasoning capabilities of LLMs. Also note that the single-test-case pseudo feedback described earlier is essentially equivalent to multi-test-case method feedback with number test cases equals to one and the input test case is empty.
3.2.2 PSEUDO FEEDBACK FROM SELF- SELF-CONSISTENCY
Methods above leverage frontier LLMs to create pseudo feedback. We can alternatively create feed- back from our policy model Ty to facilitate self- celf-improvementt without external guidance. We start from the method for the multi-test-case reasoning tasks, since the single-test-case counterpart is special case of it. Specifically, we re-use the input test cases generated in Step 1 (Section 3.2.1 The main difference starts from Step 2. the second step, we use our policy model Ty to sample pseudo solutions. The pseudo output the third step is also based executing all pseudo solutions from our policy model TT. We can apply the same process to single-test-case reasoning tasks such as mathematical reasoning, which is equivalent using majority voted answer from Ty samples as pseudo feedback. We can again use Equation D to verify the correctness of solutions from
3.2.3 PREFERENCE LEARNING UNDER PSEUDO FEEDBACK
Given the problem and the pseudo feedback (i.e., test cases) T we have created, the preference optimization process is follows. We first sample N solutions Y = {y1, 1/2 yN to problem from our policy π.. We then obtain our verification results R {ri, (2../2...... (i.e., executing all solutions on all test cases). We then move to create preference pairs using a different method as in Equation]] 2
using Equation
P. = (Vr, Ir T. ri a. R, E R)
where e and are two hyper- parameersss controlling the quality lower-bound of positive samples and the margin, respectively. Because our pseudo test cases may contains errors and if solution yn is required to pass all test cases, we may end up with no positive solutions for problem æ. As result, a solution passes enough tests (rw. 2 <) and significantly more tests than another solution (Tw Ti σ), we treat them ((y 3i)) as eligible preference pair.
The above preference pairs in P. are based on outcome feedback of test cases. Recent studies (Wang al. 2024a| Jiao et al. 2024) demonstrate that the outcome feedback can be used to estimate the expected returns intermediate reasoning steps, which can help the model produce better reasoning trajectory. Motivated from this, also construct the step-level process preference data. Following PDPO (Jiao et al.) (024), given the solution prefix g, we employ the same policy model to sample M completions following ÿ. and treat the averaged outcome feedback & of the completions as the
5

---

Published as a conference paper at ICLR 2025
expected returns of ÿ.
f == (y r(y o y)
(7)
where o is the concatenation operator. After that, the process preference data can be defined as: P. - (thw úi) In
e, ft aw R, ft R}
The final preference data we use is a combination of the outcome and process preference datasets P = P.s. We use the DPO objective (Equation 3 to optimize the policy model
Iterative Training PFPO can applied after the supervised fine-tuning (SFT) stage and we can train the policy model iteratively with both the feedback from frontier LLMs and from self- consistency. Imperially, we find applying LLM feedback first and then followed by self-consistency step.
feedback rather than the opposite achieves better results. Figure 2 illustrates the process of single
EXPERIMENT
4.1
EXPERIMENTAL SETUP
Prompt Collection For mathematical reasoning, we followed [ang et al. (2024) to create 800K prompts under the help of GPT-4o. 500K of them are paired with one solution written by GPT- 4o to construct pseudo feedback from frontier LLM. The 500K data is named MathScale-50Ok, and the other prompts are called MathScale-300K. We also filtered out around 790K prompts from NuminaMath by removing those that cannot extract the predicted answers from or having appeared in the test set of MATH. For validation, we randomly sampled 2,000 question-solutio pairs from the training of MWPBench (Tang et al. 2024) after removing the questions from GSM8K (Cobbe et al. 2021a).
For code generation, have collected the problems from the training of APPs (Hendrycks al. 2021a). Magicoder (Wei et al.) (2024) and xCodeEval (Khan et al. 2024b), which contains 5,000, 9,000 and 6,400 questions, respectively. We remove all prompts where the case inputs failed to be synthesized. We randomly sampled 500 questions from the training set of APPs for validation. For APPs and xCodeEval, we use GPT-4o to generate the test case inputs. For Magicoder, we employ Mistral-Large-Instree 1nstutttttr
1 for test case inputs generation, because of large size of the original magicoder dataset. The detailed prompt can be found in Appendix C.I
Evaluation For mathematical reasoning, the performance is evaluated on the test set of MATH (Hendrycks et al. 2021b). GSMSK (Cobbe et al.) 2021a), and College Math (Tang et al. evaluate the models HumanEval (Chen et al. 2024 by Accuracy. For code generation,
2021a), MBPP (sanitized version) (Austin et al., 2021), APPs (Hendrycks et al.) (001a), and Live- CodeBench (Jain et al. 2024) by Pass@1 I. Without specific clarification, all evaluations are con- ducted using zero-shot prompting and greedy decoding.
For simplicity, we only highlight the main results method. For more details include the detailed source of prompts, from which checkpoints are the models initialized, please refer to Appendix A All hyper-parameters for different experiments can be found in Appendix B
4.2
EXPERIMENTAL RESULTS
4.2.1 MATHEMATICAL REASONING
We took two models for experiments: 1lama-3.1-8B-bart (Dubey et al. 2024 and Mathstral-7B. v0.1. We first conducted SFT on 500K MathScale data with GPT-4o annotation, followed by our method, with GPT-4o generated labels pseudo feedback. As shown in Table the pseudo feed- back from GPT-4o can achieve consistent improvements on Llama-3.1-88 1-8b-base and Mathstral-7B. On MATH and College MATH, PFPO-LLM have made 1.2 and 4.1 averaged aboslute improve- ments compared with Llama-3. 1-b w/ SFT and Mathstral-7B w/ SFT, respectively.

---

Published as a conference paper at ICLR 2025
Table I: Overall results on mathematical reasoning benchmarks. PFPO-LLM refers the training
phase employing the pseudo feedback from frontier model (GPT-40), while PFPO-Self indicates the phase using pseudo feedback constructed from self-generated solutions. NuminaMath-7228 C-
is built on Qwen2-72B by fine-tuning NuminaMath. 1 Results are from Chan al. (0224). We
employ an evaluation strategy similar to
Yang et al. (2024b).
MATH
GSM8K
College Math
GPT-40-2024-0512
78.7
95.8
46.7
GPT-4-Turro-222400000
72.8
94.8
44.2
LPP14--Tubb--100-peeviee GPT-4-0613
64.3
55.0
93.5
39.0
NuminaMaath77BBCoff (Beeching (Becchn
2024
67.1
91.7
39.8
-lama-3.1-BB-nsttee (Dubey al. 2024
47.5
27.5
(Dubey el 2024
68.1
95.5
41.8
Llama-3.1-8B
(Dubey ct al.
2024
20.3 (4-shot)
56.7 (8-shot)
20.1 (4-shot)
w/ SFT
53.8
85.1
34.6
w/ PFPO-LLM Iter.
55.0
86.6
35.8
w/ PFPO-Self Iter.
55.9
87.6
36.6
w/ PFPO-Self Iter. 2
56.6
88.9
37.0
w/ PFPO-Self Iter. 3
57.0
88.8
36.7
w/ PFPO-Self Tter.
57.4
89.1
37.6
w/ PFPO-Self Tter.
57.8
89.6
38.0
Mathstral-7B-v0. (Mistral Team 2024h
58.3
85.6
34.3
w/ SFT
61.4
87.3
38.4
w/ PFPO-LLM Iter.
66.7
90.0
41.3
w/ PFPO-Self Iter.
67.8
90.8
42.0
w/ PFPO-Self
68.6
90.3
42.2
w/ PFPO-Self Iter. 3
68.2
90.4
42.3
At the second phase, we started iterative pDPO training on unseen prompts with seff-consistency- based pseudo feedback. For Llama-3.1-88 we used the prompts from NuminaMath-790k to syn- thesize solutions and construct pseudo feedback via selffconnsitency. The prompts are divided into non-overlapped splits for iterative training. As shown in the table, by employing pseudo feed- back, the models achieve continuous improvements across different iterations. Specifically, our method achieves the best results Iteration 5. Compared with Llama-3,1-8h w/ PFPO-LLM Iter. 0, it achieves consistent improvements with 2.8 on MATH, 3.0 on GSM8K, as well 2.2 on Col- lege Math, revealing the potential of iterative preference optimization via pseudo feedback from seff-consistency.
For Mathstral-7B. back, since we did not observe improvements on NuminaMath. The prompts across different iterations are the same. As shown in the table, Mathstral-7B w/ PFPO-Self Iter. 2 achieves 1.9 absolute improvements on MATH, compared with the SFT model. And it can outperform the stronger counterparts like MaminaMatth72B--Ca 1 106-prev with only 7B parameters, which have demonstrated the effectiveness of pseudo feed- back. Besides, we find the performance will saturate after several iterations. We discuss the possible reasons in Appendix
the prompts in MathScale-300K for iterative training with pseudo feed-
Llama-3.1-70B Instruutt and GPT-4-Turbo-
A.2 A.2
4.2.2 CODE GENERATION
For code generation, we selected Deepseek-coder--BBy --r.odder-7B-11..--ntreee (Guo et al., 2024) for experi- ments. We first use GPT-4o to generate 11 program solutions for each question the APPs training set, and use the ground-truth test cases to remove those having failed tests. The left solutions are kept to first fine-tune the base model. The resulted model is referred as w/ SFT (APPs).
Direct Preference Optimization via Test Case Execution Feedback As described in Sec- tion 3.2.2 we have constructed preference dataset via executing the generated programs over real or synthetic test cases. The evaluation results are shown in Table and

---

Published as a conference paper at ICLR 2025
Overall results (Pass@11) on program generation benchmarks. PFPO-Self refers our training from pseudo feedback method, and the content in the brackets afterwards indicates the source of prompts. Specifically, M.C. M. refers the prompt of Magicoder (Wei al. (022). and xCode. is the short for xCodeEval (Khan al. 2024b). Introductory, Interview, and Competition indicate the three difficulty levels APPs. w/ (p) (APPs) refers to that the execution feedback is synthesized based on the groundtruth test cases annotated in APPs training set.
Table 2:
APPs
HumanFyal
MBPP
Overall
Introductory
Interview
Competition
GPT-4-0613
35.1
61.8
34.4
10.6
87.8
82.1
112-44-202240513
32.2
93.3
87.2
1lama-3.1-8B-Inae Dubey et al.
2024
11.5
29.4
8.5
2.7
72.6
71.2
Llama-3.1170BB 1--00B-Insttrrct (Dubey a..
2024
24.9
51.8
21.3
9.1
80.5
83.3
Codestral-22B-VD0 (Mistral Al leam
2124a
20.3
45.2
16.9
5.8
81.1
78.2
Codeeween1.577B..c2 @wenn Team
2024
80.5
Owen2.5-Coder-7B-nnnttie (Hui et
21124
15.7
37.3
12.3
4.1
85.4
86.0
Deepseck-cxderr3388--me Instrrtt (Guo al
2024
18.4
44.2
14.5
4.4
77.4
79.0
Deepsesk-coderrvll..-nntt.
14.3
35.7
10.8
75.6
73.9
SFT (APPs)
72.0
72.8
w/ DPO (APPs)
16.3
36.2
13.3
5.3
74.4
74.3
w/ pDPO (APPS)
16.9
37.3
13.8
6.1
73.8
73.2
PFPO-LLM (APPS)
17.9
38.3
14.7
73.8
75.9
w/ PFPO-Self Iten. (APPs)
17.4
37.5
14.8
5.4
73.2
75.1
w/ PFPO Iter. (APPs & M.C.)
18.0
39.2
14.9
6.2
79.3
75.5
w/ PFPO-Self Iter. (APPs & M.C. xCode.)
19.1
40.9
15.9
6.9
73.8
75.1
Table 3: Overall results on LiveCodeBench. We follow the recommended setting by sampling 10
solutions for each problem with temperature as 0.2, and estimating the Pass@1 results. The cutoff date of the test questions is from 2023-09-01 2024-09-01. All results except those of our models are referenced from the official leaderboard https://iiecoodeeenngghhbb
Overall
Easy
Medium
Hard
-laude-3.5-Sonme 3.5-Soneet
51.3
87.2
45.3
11.0
Claude-3 3-onnet
26.9
67.2
7.3
1.4
Claude-3-Haiku
24.0
61.3
5.5
0.9
GPT:3.5-Turbo-0125
24.0
0.3
Jlama-3.1-70B-Instuee (Dbeey et al.
2024
31.8
67.9
17.3
4.1
Llama-3-70B-Instrur (Dubey
2024
27.4
59.4
15.6
1.3
CodeQwen1.5 1B- 5B-Chat (wen Team
2024
16.8
35.9
10.9
0.3
DeepSeekCoder-V22 V2-23BB (DeepSeek Al
2024
41.9
79.9
32.0
4.9
Deepecck-Cooter333B--nttt (Guo al.
2024
23.4
56.1
8.6
0.9
Deepsekk-coder:-Bv11....
21.1
51.3
7.4
0.2
w/ SFT (APPs)
22.9
53.0
10.6
0.2
DPO (APPs)
22.9
53.7
9,4
1.0
w/ pDPO (APPs)
22.9
55.0
8.1
1.3
w/ AFPO-LLM Iter. O (APPs)
24.0
56.8
9.3
1.4
PFPO-Self Iter. (APPs)
23.4
54.2
0.7
w/ PFPO-Self Iter. (APPs & M.C.)
23.7
55.8
9.5
1.1
w/ PFPO-Self Iter. (APPs M.C. xCode)
24.3
56.8
9.8
1.6
First, we aim to discuss the effectiveness of fully synthetic test cases, topic that has not yet been extensively explored. We use w/ DPO and w/ pDPO to denote methods utilizing ground truth test cases gather execution feedhack, while PFPO-Self Iter. O (APPs) employs the same prompt set but simulates execution feedback using pseudo test cases. From the results presented, observe that pseudo test cases outperform ground truth ones in almost all benchmarks, with the exception of HumanEval.
Table 4: The averaged number of test cases of each problem in the training
APPs.
Avg. No. Original Training
Synthetic
5.16
9.95
particular, PFPO-Self Iter. leads both 0.5 absolute improvements APPs and LiveCodeBench compared with the groundtruth pDPO. This improvement attributed to the potential of increasing the number of synthetic test cases, thereby reducing the false positive rate associated with missing
8

---

Published as a conference paper at ICLR 2025
22
21.06
21.34
Gxel
300
Gael
No. Date Peints
No. Date Points
No. Date Points
20:
200
18-
200
100
50
0-
01
G.D.
S.C.
S.C.
20
40
20
40
20 40 60
(b)
(0)
(d)
Figure 3: (a) The @ass@1 performance of the test set of APPs of our model variant, w/ PFPO- Self Iter. (APPs & M.C. & xCode, DPO) under different strategies. G.D. indicates greedy de- coding, S.C. is the short for self-consistency. We took the groundtruth test case inputs to execute the programs sampled from our policy model. For each group of test case inputs, we employ self- consistency across different programs to determine the pseudo outputs. For S.C. -P, we select one of the programs matching the pseudo outputs for evaluation. For S.C. we simply check whether the pseudo outputs are consistent with the ground-trus truth outputs, which indeed highlights bound of test case self-consistency. since sometimes we fail select at least one solution matching all the pseudo outputs. (b) (d) We measure the top-1 I frequency among the outputs and average them by the number of test cases. The x-axis indicates the averaged top-1 frequency, and the y-axis demonstrates the corresponding amount of data samples. The three figures show the relationships on (b) all questions, (c) introductory-lever questions, and (d) interview-leve questions, respectively.
upper
corner cases. As shown in Table 4 the questions in the APPs training set contain only 5.16 test cases on average, whereas we can ensure that each problem has approximately 10 test cases.
Besides, by comparing w/ PFPO-LLM Iter. 0 (APPs) and w/ PFPO-Self Iter. 0 (APPs), we find that the pseudo feedback generated by frontier LLM also demonstrates better results on code generation. This also reflected through the quality of synthetic test cases, and we have a deep analysis in Appendix A.4
Iterative Training Afterwards, we attempted to introduce more training data an iterative man- to see if the pseudo feedback can continuously enhance the performance. As shown in Table and after three rounds of training. PFPO-Self Iter. has made 3.7 and 1.4 absolute improvements on APPs and LiveCodeBench, respectively. We also observed more significant improvements on the Easy and Hard level of LiveCodeBench, i.e., 3.8 and 1.4 absolute points, respectively. Besides, on HumanEval and MBPP, we did not observe extra improvements after introducing xCodeEval. This is probably because xCodeEval contains only standard input/output problems, which could lead to distribution shift. The small scale of test cases and problems in HumanEval and MBPP also make the evaluation not really stable.
4.3 RELATIONSHIP BETWERN SELF--CONSISTRNCY AND RELIABLE PSEUDO FEEDBACK
In this section, we will explore the impact of test-case-basee based celf-consistency during inference for coding. We sample 64 solutions for each problem in the APPs test set and apply self-consistency over the provided test cases to select the final programs. Two strategies are developed within this process: Self-Consistence for Programs (S.C. P) and Sell-Consistency for Test Cases (S.C.
T). Following Section 3.2.2 we execute the sampled programs on the test case inputs annotated by the dataset and determine the pseudo outputs for each group of inputs through majority voting.
First of all, we hope to evaluate the correctness of the self-consistency based pseudo outputs by checking the consistency with the ground truth outputs. If this is true, we treat this question is passed. This strategy is called S.C. T. After that, we can check if there exists at lease one program among the sampled solutions such that outputs can match all the pseudo outputs, which is called S.C. P. In other words, for S.C. - T, we assume that there is such a program among the candidates that can obtain the expected pseudo outputs, so we only care about the pseudo outputs are as expected. For S.C. P. we also consider if such program is included in the sampled solutions. To this end, S.C. - T can be considered the upper bound of celf-consistency. as there are cases where no solution matches the pseudo outputs. As illustrated in Figure 3 (a). the sat-case-basee self-consistency also makes significant improvements during test time. Besides, by continuously sampling solutions against the pseudo outputs, the pass rate would be further improved.

---

Published as a conference paper at ICLR 2025
70
72
Accuraca
66
Accuracy
Accuray
Accuracy
66
Eant nothhie
70
Bol veshee
20
neighta greed cacldng
64-
goad cacading
69
gred) dR22
68
62
67
68
00
Sample@K
Samplegk
SamplegK
Sample@k
(0) w/ SFT.40
(b) PFPO-40
(0) PFPO-Self Iter.
(d) PFPO-Self Iter.
Figure 4: Performance comparison of four Mathstral-7B variants by scaling inference
Besides, we also want to explore the reliance of self-consistency based pseudo feedback. From Fig- (b) to (d), we display the distribution of top-1 output frequency averaged by the number of test cases, i.e., the confidence of the policy model, problems. We find that, self-consistency over test cases provide reliable pseudo feedback when dif- ferent programs achieve high consistency rate. Although the margin between positíve and negative solutions is reduced with the increase of difficulty, by continuously improving the sampling budget, the uncertainty lower down the possible inaccurate feedback. On mathematical reasoning, similar conclusion still holds. Figure [ demonstrates the accumulated ratio of corrected predictions with the development of self-consistency-balee prediction ratio among all candidate answers.
overall, introductory-level. and interview-level
also be reduced. This may help to both control the difficulty of training set, and
4.4 SCALING INFERENCE AND REWARD MODELING FROM PSEUDO FEEDBACK
In this section, we increase the inference budget (Snell al.) 2024) assess how further improves mathe- matical reasoning. Specifically, we explore two strate- gies: self-consistency and best-of-N weighted (Li et al. 2023b). The reward model used for best-of-N weighted is optimized on the same training set Mathstral-7B w/ PFPO-Self Iter. 1, thus also benefiting from the pseudo feedback derived from self-consistency.
Rolie of Correct Predictions
0.9
0.7
The results in Figure indicate that: (i) Scaling infer- ence makes significant improvements, and employing an extra reward model to score responses benefits, especially when using smaller sampling bud- get. (ii) Pseudo feedback can also enhance reward mod- eling. (iii) As highlighted by [Snell et al.] (2024), com- bining pseudo reward models with other inference tech- niques (e.g., weighted best-of-N (Li et al.) (2023)), look- ahead search, and step-level beam search) may improve performance. We leave the relevant exploration as future work.
0.25
0.50
0.75
1.00
Figure 5: The accumulated ratio of cor- rect predictions over all questions. The x-axis denotes the ratio of the top- fre- quency among all predicted candidates, ie., the confidence for seff-consistency. Each point in the figure indicating how many predictions are correct when the confidence is lower than the x-value.
bring more
5 CONCLUSION
In this paper, we demonstrated the potential of synthesizing pseudo-feedback from both frontier LLMs and the model itself. By incorporating feedback from GPT-4o, we improved Llama-3.1-88 base-SFT's performance on the MATH from 53.8 to 55.0 and enhanced Mathstral-7BB performance from 61.4 to 66.7. Furthermore, by leveraging self-generated feedback based on self-consistency. we increased Llama-3.11885 performance to 57.8 and Mathstral-7B"s to 68.6. For code gener- ation, we introduced a novel test case-level self-consistency strategy. By employing fully self- constructed pseudo test cases, we boosted the SFT model's performance from 15.4 to 19.1 on APPs and from 22.9 to 24.3 on LiveCodeBench. Additionally, we analyzed the relationship between self- consistency and reliance synthetic pseudo labels, offering insights for future research. For future work, we hope to combine the pscudo reward models with scaling inference techniques to seek more improvements more challenge scenarios.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 2023.
Sina Alemohammad, Josue Casco-Rodriguee Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk. Self-consuming generative models go mad. arXiv preprint (3..xxi2337.018500 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv: 07733, 2021.
Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. Numinamath 72b cot. https:// 2024.
huggingface.com co
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosiute, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran- Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds Ben Mann, and Jared Kaplan. Mcasuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03544 2022.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner. Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to- Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference Machine Learning- OpenReview.net, 2024.
Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. CoRR, abs/2406.200944 2024.
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with DPO implicit rewards. CoRR, abs/2406.09760, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Kerbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino. Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. arXiv preprint arXiv:2107.033774 2021a
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021b.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprins arXiv:2211.12588 2022.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024b.
11

---

Published as a conference paper at ICLR 2025
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021a.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144 2021b
DeepSeck-AL, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang. Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. CoRR, abs/2406. 2024.
Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Am- rit Bedi, and Furong Huang. Sail: Self-improving efficient online alignment of large language models. XrXiv preprint arXiv:2406. /5567, 2024.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment. TMLR, 2023, 2023.
Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang. Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang. Xuan- jing Huang, and Tao Gui. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. CoRR, abs/2402.00391, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CORR, abs/2407.21788. 2024.
Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty- first International Conference on Machine Learning, ICML. OpenReview.net, 2024.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. CoRR, ab4/2401.14196, 2024.
Joy He- Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman. Solving math word problems by combining language models with symbolic solvers. arXiv preprint 2xXiv:2304.09000 2023.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021b.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self- self-improve. arXiv preprint arXiv:2210. 16100 2022.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.0 120099.22/0 186, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. CoRR, abs/2403.07974, 2024.
Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen, and Shafiq Joty. Learning planning- based reasoning by trajectories collection and process reward synthesizing. arXiv preprint arXiv:2402.00658, 2024.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Ed- ward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive LLMs leads to more truthful answers. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pp. 23662-23733, PMLR, 21-27 Jul 2024a. URL otttps://proceedinng mlr press v235/ 4 .ttll
Mohammad Abdullah Matin Khan, M. Saiful Bari, Xuan Do Long, Weishi Wang, Md. Rizwan Parvez, and Shańq Joty. Xcodeeval: An execution- based large scale multilingual multitask bench- mark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 6766-6805. Association for Computational Linguistics, 2024b.
Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of Ilm outputs. arXiv preprint arXiv:2407. 13692 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. CoRR, abs/2406. 18629, 2024.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022, New Orleans. LA, USA, November 28- December 9, 2022, 2022.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poiries João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Mong-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason T. Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey. Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni. Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dovan-Gavitt, Danish
13

---

Published as a conference paper at ICLR 2025
Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferran- dis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions Machine Learning Research, 2023, 2023a.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian- Tian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprini arXiv:2206.023356 2022a.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computattonal Linguistics (Volume 1: Long Papers), pp. 5315- 5333. Association for Computational Linguistics, July 2023b.
Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago. Thomas Hubert, Peter Choy, Cy- prien de Masson d' d'Autmme, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. CoRR, abs/2203.0788 14, 2022b.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations. OpenReview.net, 2024.
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. RLTF: rein- forcement learning from unit test feedback. Trans. Mach. Learn. Res., 2023, 2023.
Haipeng Luo. Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint 27X25:2308.09583, 2023.
Mistral AI Team.
Codestral. Codestrall ral- 22B-v0. 2024a.
https::/haggiigfaae..c//tttttttt
Mistral
AI Team.
Mathstral.
https://haggiingfee..c///tstttl
Mathstral- 1aasstrrl-BB-o.111 v0.1. 2024b.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830 2024.
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Codegen2: Lessons for training 11ms on programming and natural languages. ICLR, 2023a.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong, Codegen: An open large language model for code with multi-turn program synthesis. ICLR, 2023b.
Qwen Team. Code with codeqwenl.5, April 2024. URL ht:ps:://wen.m..qiii. o Gllg/codeqwen!
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances in Neural Information Processing Systems, 2023.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint 2rXiv:2308.12955 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707 003477,00777 2017.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305. 17493, 2023.
14

---

Published as a conference paper at ICLR 2025
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute opti- mally can be more effective than scaling model parameters. CoRR, abs/2408.00311, 2024.
Zhengyang Tang, Xingxing Zhang. Benyou Wang, and Furu Wei. Mathscale: Sealing instruction tuning for mathematical reasoning. arXiv preprint 17Xi::240300284 2024.
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei
Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models.
arXiv preprint arXiv:2404.14336 2024.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instraction-folloiinng llama model.
https: //tt:///gttub...// www.coo////...oo. laa- tabb/stante stanssrafoorrroooooo alpaca 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211. 14277, 2022.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhi- fang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp- 9426 9429. Association for Computational Linguistics, 2024a.
Tianduo Wang, Shichen Li, and Wei Lu. Self-training with direct preference optimization improves chain-of-thought reasoning, 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery. and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1117,, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pp. 13484-13508. Association for Computational Linguistics, 2023a.
Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, and Heng Ji. Enable lan- guage models to implicitly learn self-improvement from data. arXiv preprint arXiv:2310.00899, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. In International Conference on Machine Learning. PMLR, 2024.
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. Large language models are rea- soners with velffverification. arXiv preprint arXiv:2212.09500 2, 2022.
Martin Weyssow, Aton Kamanda, and Houari A. Sahraoui. Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models coding preferences. CoRR, abs/2403.09032, 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under kl-constraint. In Forty-first International Conference on Machine Learning. OpenReview.net, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024a.
15

---

Published as a conference paper at ICLR 2025
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Swen2.5-math technical report: Toward mathematical ex- pert model via self-improvemmet. CoRR, abs/2409. 12122, 2024b.
Yuqing Yang, Yan Ma, and Pengfei Liu. arXiv:2407. 13647, 2024c.
Weak-to-strong reasoning.
arXiv preprint
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint urXiv:2309. 12284, 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint urXiv:240.110020 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 15:15476- 2022.
A
MORE RESULTS AND ANALYSIS
We list detailed results, the source of training prompts, as well some variants presented in the main content, in Table and 12
A.1 COMPARISON BETWEEN DPO AND PDPO
From the results across different base models and tasks, we find that pDPO usually have signif- icantly better results than DPO. The only exception appears HumanEval and MBPP, which is possibly due to the two benchmark does not require detailed reasoning process, and the mismatch in distribution causes performance degradation.
A.2 VARIANTS OF ITERATIVE DPO TRAINING
In our experiments, we have employed several variants of Iterative DPO training for different con- siderations. For NuminaMath-79000 790K, collecting all prompts for single iteration of DPO training can make it more challenging avoid policy shifting, as pointed out by Xiong et al. (2024). To this end, we split the whole dataset into several parts for itcrative training. During each iteration, we around 160K prompts collect solutions, construct pseudo feedback, and optimize the policy model.
For MathScale-30000 300K, since the dataset is much smaller, tions, and the process ends when we cannot observe significant improvements.
all prompts across different itera-
For code generation, prompts to be mixed with the original prompts, and collect new solutions. The pseudo test cases are also constructed based on the new solution programs. The reason is that do have too much data but we also hope to approach on-policy training.
a hybrid strategy. During each iteration, introduce new dataset of
During each iteration, the reference model is initialized from the policy model in last iteration, instead of the original base model.
A.3 EFFECT FROM THE AMOUNT OF SYNTHETIC TEST CASES
As shown Table and 3 PFPO-Sed Self achieves even higher results than the model optimized on ground-truth test cases. One possible reason behind this is have synthesized more test cases for self-consistency to reduce the false positive rate. To further verify this point, we down-sampled the synthetic test cases to align with the number of ground truth test cases. Specifically, for each coding
16

---

Published as a conference paper at ICLR 2025
problem, if the synthesized test cases exceeded the number of ground truth test cases, randomly selected a subset of synthesized cases to match the number of ground truth test cases. Otherwise, all synthetic test cases were retained.
As shown in Table 7 the results demonstrate that reducing the number of synthetic test cases sig- nificantly decreases performance. This highlights the importance of scaling test cases for verifying program correctness effectively.
A.4 QUALITY ANALYSIS OF SYNTHETIC TEST CASES
In this section, we will analyze the quality of the synthetic test cases. Specifically, for each question in the APPs training set, we selected ground-true truth program to evaluate the synthetic test cases. A synthetic test case is valid, if and only if given the test case imput and its output matches the output of the ground-truth program. We calculate the pass rate as metric, which represents the percentage of valid test cases among the total generated. The results are shown in Table g where the Model column denotes the model optimized by the pseudo feedback constructed from the corresponding test cases.
From the table, we can conclude that: (1) The synthetic test cases constructed by the weaker policy models has already demonstrate good quality. For example, 62.68% of the test cases for first round training. i... Self--ellfSelf Iter. 0 are correct. (2) Iterative training on fixed prompt set will lead to overfitting. As the saturation of the quality of synthetic test cases, the policy model will fail to collect more correct instances from the training set, and the model will thus stop learning. (3) Pseudo feedback from frontier LLMs usually demonstrates better quality. We find that the test cases synthesized from frontier LLM's solutions achieve 82.41 % pass rate. Yet, we find that PFPO- LLM cannot outperform PFPO-Self Iter. 2, which is possibly due less training data and the off-policy optimization of DPO, and can be alleviated by more rounds of traininig.
A.5 ANALYSIS ABOUT PLATEAUED PERFORMANCE OVER ITERATIONS
As also observed by Xiong et al. (2024) and Chen et al. (024a)), iterative style DPO tends to saturate after several iterations, regardless of whether the preference pairs are human labeled synthetic. We believe there are at least two reasons for the saturation. The first one (as mentioned in Ap- pendix AA44) is the quality of synthetic test cases may plateau at some point. The second reason could be that as the model improves with each iteration, more problems in the training set become "too easy" for the model its current stage. For extremely challenging problems where no correct solution can be generated despite many attempts (sampling). we are unable to create valid prefer- ence pairs and are thus unlikely to solve them in the next iteration. And the number of moderately challenging problems, which are most effective for improving the model, gradually decreases after each iteration.
One possible solution to this is introducing unseen problems in each iteration to increase the likeli- hood of hitting new moderately challenging problems. In the third block of Table D we simulated the above setting by introducing new problems across iterations. As detailed in Appendix A, we split the NuminaMath dataset into five subsets of equal size and use a different subset for each iteration. We observed consistent improvements across iterations (see w/ PFPO-Self Iter. 1 to 5 rows in Table 1). However, please note that applying the above method requires a significant amount of unseen high-quality questions in that domain. Therefore,
cannot apply this strategy with "Mathstral-77B. vO.1" based models (Mathstral-7B:s 7B-00. 1 has been trained on Numina already) or in coding domain (high quality prompts in coding domain is very limited). Finally, is worth noting that our method's advantage lies in its scalability, as no additional human annotation for answers is required when incorporating new problems.
A.6 PERFORMANCE ON MORE CHALLENGING DATASET
We also evaluated our performance subset of challenging math problems to explore whether the iterative training procedure also makes continuous improvements.
For the dataset construction, began by sampling 16 solutions generated by Mathstral-7B w/ SFT on the MATH test set. We then identified questions for which the predicted answers, even after applying majority voting, remained incorrect. This yielded a MATH-Challenging test set of 1 1,580
17

---

Published as a conference paper at ICLR 2025
questions. As shown in Table I0 the results indicate that our method can achieve improvements even on challenging questions over iterations.
B HYPEER-ARAMETTE
All hyper-parameters are listed in Table
C PROMPTS
C.I TEST CASE INPUTS GENERATION
The 2-shot prompt for test case inputs generation shown in Figure
7 and
C.2 PROMPT FOR COMPETITION-LEVES CODE GENERATION WITH RATIONALE
The 1-shot prompt for competition-leve code generation is shown Figure 9 10 and SFT, we remove the 1-shot example.
During
D MORE IMPLEMENTATION DETAILS ABOUT PDPO
Our own implementation pDPO include the following steps: (1) Sample multiple solutions for each problem from the policy model. (2) Supposing non-empty line is reasoning step, sample multiple prefixes for each solution following a fixed ratio, where each prefix is composed of several continuous reasoning steps from the begining. If the prefix dataset is too large, we will control that each problem will have fixed amount prefixes. In most experiments, we will set the ratio as 30% and choose the fixed amount in (8, 10, 20), as shown in Table 5 (3) Take the prompt, problem, as well as each solution prefix to compose new input, and sample 3 completions for it. (4) Estimate the expected returns of each prefix by checking the results approached by the completions. (5) Here is the main difference with the original inplementation of pDPO (Jiao et al.) 2024): The authors choose first sample small amount of prefixes well as the estimated expected returns for training another reward model, and use the reward model to annotate the complete solutions. This is to reduce the computation resource usage. Instead, in this paper, we simply estimate expected returns for all sampled solutions within given budget, and train the policy model on the prefixes (incomplete solutions) via DPO directly.
E RESULTS OF DIFFERENT TRAINING ORDER
Table Table/occompare. the results of different training order. Assuming MathScale-3000 and MathScale- 500K share similar difficulty, the results indicate that the better quality the training data has, the earlier iteration it should be employed in.
F RELATED WORK
F.1 MATHEMATICAL REASONING
To enhance the model's critical capability in mathematical reasoning, researchers have explored various techniques to guide the reasoning process. These efforts include prompts engineering (Wei
et al. (022), tool usage (He- Yueya et al. [2023 Chen et al.) 2021b 2022), process rewarding (Light-
man et al. 2024 Wang et al. 2024a Jiao et al. 2024 Lai et al. 2024). and verifiers (Cobbe et al.]
2021b Li et al.) 2022a Weng et al. 2022). In addition to improve frozen LLMs. researchers also
work on synthesizing math related data to fine-tune them YYu et al. 2023
Luo et al. 2023 Mitra
al. 2024
Yue al. 2023)
18

---

Published as a conference paper at ICLR 2025
F.2
CODE GENERATION
Code generation has long been recognized as challenging due to data sparsity. The emergence of LLMs (Guo et al. 2024 DeepSeek-Al al. 2024; Nijkamp et al. 2023b Zelikman al. 2022
pretrained on corpus including rich code has partially reduced this problem. Yet, it is still far from enough due the significant gap between code completion and programming solve complex problems. Li et al. (0226) make earlier efforts towards competition-leve code generation, which is achieved by filtering solutions from large sampling space via example test cases. Wei al. (2024) propose to synthesize instruction tuning dataset for code generation via self-instruct (Wang ct al., 2023a). Besides, [e et al. (2022); Liu et al. (2023) and Dou et al. (2024) propose to use the feedback from compiler and unit test to improve code generation through reinforcement learning. However, these approaches still rely on human annotation to obtain test cases, and thus cannot be scaled to large scale training. Dou al. (2024) further employ the unit test based feedback for process supervision. Weyssow et al. (2024)) have constructed a preference dataset for code generation annotated by GPT-3.5. Yet, the absence of test cases hinder it to be generally employed in on-policy learning.
Table 5: The syper-parametert used in our experiments. K S.C. refers to the amount of sampled solutions for each problem that used to determine the pseudo label via self-consistency. Instead, K DPO indicates the amount of sampled solutions per question used for constructing preference pairs. No. Prefix is the amount of sampled prefixes per question used for process DPO training. There are two kinds of values for No. Prefix. The integers are the exact amount, while the percentage indicates that we sampled the specific ratio prefixes among
solutions for each problem. This used to address the problem of limited training data. I is the coefficient used in DPO training. is the weight for NLL loss. is the reward (feedback) lower bound of positive samples to control the use fraction to simply indicate the number of sampled
quality. For pDPO training, sometimes completions as well as the successful attempts. For example, 3 means that we sample 3 completions for each prefix, and if there is at least 1 completion is successful, it kept. a is the margin control the gap between positive and negative samples.
Model
K S.C.
K DPO
No. Prefix
B
Mathstral-7B w/ SFT
w/ DPO (M.S.-500k, Tter. 0)
10
10
0.5
0.2
1.0
0.0
w/ pDPO (50k.-500k, Iter. 0)
10
10
0.1
0.2
0.0
w/ PDPO (M.S.-300K-S.C.. Iter. I)
10
10
10
0.5
1.0
0.0
w/ pDPO (M.S.-300K-S.C.. Iter. 2)
10
10
10
0.5
1.0
0.0
Llama-3.1 w/ SFT
w/ DPO (5.S.- Tter. 0)
10
10
0.5
0.2
1.0
0.0
w/ pDPO (M.k.-500k, Iter. 0)
10
10
10
0.5
0.2
0.0
w/ pDPO (Numina-S.C. 160k, Iter. 1)
8
0.5
0.2
0.0
w/ pDPO (Numina-S.C. 320k, Iter. 2)
16
8
0.5
1.0
0.0
w/ pDPO (Numina-S.C. 480k, Iter. 3)
8
0.5
1.0
0.0
pDPO (Numina-S.C. 640k, Iter. 3)
8
0.6
0.2
w/ pDPO (Numina-S.C. 790k, Iter. 3)
32
0.6
0.2
Deepseck-coder-v/ 1-5hhha w/ SFT
w/ DPO (APPs)
0.1
0.0
1.0
w/ pDPO (APPs)
30%
0.1
0.2
w/ DPO (APPs S.C.)
0.1
0.0
0.5
w/ PDPO (APPs- S.C.)
30%
0.1
0.0
0.5
DPO (APPs M.C. S.C.)
0.4
0.2
0.5
w/ DPO (APPs & M.C. xCode. S.C.)
0.4
0.2
0.5
w/ pDPO (APPs & XCude. S.C.)
0.5
0.2
w/ pDOO (APPs M.C. S.C.)
20
0.4
0.2
0.5
19

---

Published as a conference paper at ICLR 2025
Table The experimental results of models trained via different orders. The prompt set for DPO training keeps the same with the original setting, where PFPO-Self employs MathScale-300K and PFPO-LLM takes MatoScale-500K.
MATH
GSM8K
Colledge Math
Mathstral-7B-vO.1
58.3
85.6
34.3
w/ SFT-40
61.4
87.3
38.4
w/ PFPO-LLM Iter. 0
66.7
90.0
41.3
w/ PFPO-Self Iter. 1
67.8
90.8
42.0
w/ PFPO-Self Iter.
64.6
89.8
39.4
w/ PFPO-LLM Iter.
65.2
89.1
39.3
Table 7: The experimental results exploring the influence by the amount of synthetic test cases. w/ down-sumpled synthetic test cases refers to setting that we reduce the amount synthetic test cases to which is equal and less than the that of the ground-truth test cases annotated for each problem in the training set of APPs.
APPs
LiveCodeBench
DeepSeek-coder-y1.55cr
14.3
21.1
w/ SFT
15.4
22.9
w/ golden test cases
16.9
22.9
w/ synthetic test cases
17.4
23,4
w/ downesampled synthetic cases
14.8
21.1
Table 8: The pass rate evaluated by executing the annotated ground-truth solution program on our synthetic cases. The Model column denotes the using the corresponding test cases for preference optimization.
Model
Pass Rate
PFPO-LLME LLM Iter. 0 (APPs)
82.41
PFPO-Self Iter. (APPs)
62.68
PFPO-Self Iter. (APPs M.C.)
66.80
PFPO-Self Iter. (APPs & M.C. & xCode.)
66.75
20

---

Published as a conference paper at ICLR 2025
Table 9: Overall results on mathematical reasoning benchmarks. The indentation across different levels represents the corresponding model initialized from the parent-level. M.S. refers the short of MathScale, and S.C. means CeffsConnis.
MATH
GSM8K
College Math
1aama-3.1-8B--aa
w/ SFT (M.S.-5000)
53.8
85.1
34,6
w/ DPO (M.S.-500k)
53.8
86.0
35.1
w/ pDPO (M.S.-500k)
55.0
86.6
35.8
pDPO (Numina-S.e, 160k)
55.9
87.6
36.6
w/ pDPO (Numina-S.C. 320k)
56.6
88.9
37.0
w/ pDPO (Numina-S.C, 480k)
57.0
88.8
36.7
w/ pDPO (Numina-S.ck 640k)
57.4
89.1
37.6
w/ pDPO (Numina- S.C. 790K)
57.8
89.6
38.0
Mathstral-7B-v0.1
58.3
85.6
34.3
SFT (M.S.-500k)
61.4
38.4
w/ DPO (M.S.-500k)
63.0
88.6
39.1
w/ pDPO (M.S.-500k)
66.7
90.0
41.3
w/ pDPO (M.S.-300K-S.C.. Iter. 0)
67.8
90.8
42.0
w/ pDPO (M.S.-300K-S.CC Iter. 1)
68.6
90.3
42,2
w/ pDPO (M.S.-300K-S.C. Iter. 2)
68.2
90.4
42.3
Table 10: The performance on the more challenging sub-test set of MATH, which is constructed from the failed questions Mathstral-7B w/ SFT via Maj @ 16.
Mathstral-7B
Greedy Decoding
w/ SFT
19.6
PFPO-LLM Iter. 0
23.6
PFPO-Self Iter. 24.6
PFPO-Self Iter.
26.7
Table 11: Overall results (Pass @ 1) program generation benchmarks. M.C. refers to the prompt set Magicoder (Wei et al.) 2024), and xCode. is the short for xCodeEval (Khan et al. 2024b). Introductory, Interview, and Competition indicate the three difficulty levels of APPs. S.C. indicates that the test cases used for constructing preference dataset are synthesized through self-consistency. On the contrary, the without S.C. refers that the test cases the golden ones from the original dataset.
APPs
HumanFval
MBPP
Overall Introductory Interview Competition
Deepseck-coder--l.SStt
14.3
35.7
10.8
3.2
75.6
73.9
w/ SFT (APPs GPT-40)
15.4
37.8
11.6
4.1
72.0
72.8
w/ DPO (APPs)
16.3
36.2
13.3
5.3
74.4
74.3
w/ pDPO (APPs)
16.9
37.3
13.8
6.1
73.8
73.2
w/ DPO (APPs- S.C.)
16.8
39.2
13.2
5.3
76.2
75.9
w/ pDPO (APPs S.C.)
17.4
37.5
14.8
5.4
73.2
75.1
w/ DPO (APPs & M.C. S.C.)
18.0
39.2
14.9
6.2
79.3
75.5
w/ DPO (APPs & M.C. & xCode. S.C.)
19.2
42.2
15.8
6.5
75.0
74.2
pDPO (APPs & M.C. & xCode. S.C.)
19.1
40.9
15.9
6.9
73.8
75.1
w/ pDPO (APPs & M.C. S.C.)
18.5
40.0
15.3
6.5
75.6
76.3
21

---

Published as a conference paper at ICLR 2025
Table 12: Overall results on LiveCodeBench. We follow the recommended setting by sampling 10 solutions for each problem with temperature as 0.2, and estimating the Pass@ results. The cutoff date of the test questions is from 2023-09-01 to 2024-09-01. All results except those of our models are referenced from the official leaderboard.
Overall
Easy
Medium
Hard
-laude335--Sonne
$1.3
87.2
45.3
11.0
DeepSeekCoder-VV
41.9
79.9
32.0
4.9
Claude-3-Sonne
26.9
67.2
7.3
1.4
Claude-3-Haiku
24.0
61.3
5.5
0.9
GPT-3.5-Turbo-0125
24.0
55.0
11.6
0.3
LLama-3-70b-mmstuuct
27.4
59.4
15.6
1.3
Deepsack-Cooder-333--Ch
56.1
8.6
Deepseck-coder-v1.5-hhh
21.1
51.3
7.4
0.2
SFT (APPs GPT-4o)
22.9
10.6
w/ DPO (APPs)
22.9
53.7
9.4
1.0
w/ pDPO (APPs)
22.9
55.0
w/ DPO (APPs S.C.)
24.2
55.7
11.0
0.8
w/ pDPO (APPs S.C.)
23.4
54.2
10.3
0.7
w/ DPO (APPs & M.C. S.C.)
23.7
55.8
9.5
1.1
w/ DPO (APPs& M.C. & xCode. S.C.)
23.7
55.8
9.4
w/ pDPO (APPs & M.C. & xCode. S.C.)
24.3
56.8
9.8
1.6
w/ (APPs M.C. S.C.)
24.6
56.9
11.4
22

---

Published as a conference paper at ICLR 2025
You are an expert programmer. Your task to write some test cases to the programming problems to help verify the expected program solutions. You only need to give me the inputs in the required format, Now, let me introduce the details to you:
##f Program Format
There will be two kinds of programming problems, One type of problem accepts standard input-oulput stream, As result, the test case inputs should contain only the inpuls text stream.
Another kind of problem is based on function calling, which shows segment of starter code to illustrate the function head, defining the name of the arguments to be accepted. In this case, should return me the inputs the format of function callinàg, like function name( "arguments).
IP Response Format
You should return the inputs coon__ format. You need generute ** **10** groups of test case inputs, und each key field is named 'test, case where 'i' the index of the The value of each key test inputs the required format, which should be string.
Examples Standard Input-Output and Function Calling.
### Standard Input-Output Stream
#**# Programming Problem
Polycarp $n$ different binary words. A word called binary contains only characters and For example, these words are binary: "0001", "11", "0" and "001 100".
Polycarp wants to offer his set of SnS binary words to play game "words". In this game. players name words and cach next word (starting from the second) must start with the last character of the previous word. The first word can be any. For examplc, these soquence of words can be named during the game: "0101", "1", "10", "00", "00001".
Word reversal is the operation of reversing the order of the characters, For example, the word "011 1" after the reversal becomes "1110", the word "11010" after the reversal becomes "01011".
Probably, Polycarp has such set of words that there is game rules. this situation, he wants reverse some words from his set that: the final set of Sn$ words still contains different words (i.e. all words are unique); there is way put all words of the final set of words in the order that the final sequence of Sn' words is consistent with the game rules.
them in the order correspondent to the
Polycurp wants
minimal number of words. Please, help him.
-np-Input--...
first of the input contains one integer St$ (51 Me 10"4S) number of cases input. Then cases follow.
The first of a test case contains one integer We Me 2icdot10*5S) number words in the Polycarp's set. Next lines contain these words. of SnS words aren't empty and contains only charucters '0' and 'I', The sum word lengths docsn't exceed $4"cdot10066
words are different.
Guarunteed, that the sum of Snš for test cases in the input doesn't exceed S2'cdot10'5S, Also, guaranteed that the sum of word lengths for all test cases in the input doesn't exceed $4%cdot10"6S.
-..--utut----
Print answer for all of StS test cases in the order they appear.
Figure 6: 2-shot prompt for test case inputs generation. Page 1.
23

---

Published as a conference paper at ICLR 2025
there is no answer for the test case, print -1. Otherwise, first line the output should contain SkS (80 n$) - the minimal number of words the set which should be reversed. The second line of the output should contain $k$ distinct integers - the indexes of the words in the set which should be reversed. Words are numerated from SIS to $o$ in the order they appear. If Sk-0$ you can skip this line (or you can print an empty line). If there are many answers you can print any of them.
....- Example.........................
Input
4
4
0001
1000
0011
0111
3
010
101
0
2
001
0001
00001
Output
1
3
-1
0
2
1
##t#l! Response
"test "1
1111
"test
Function Calling
НННИ Programming Problem
Given single positive integer x, we will write an expression the form x (opl) x (op2) (op3) where each operator opl, op2, etc. is either addition, subtraction, multiplication, division (t. or (). For example, with x - 3, we might write 3 3/3 + 3 which is a value of
When writing such un expression, we udhere to the following conventions:
The division operator (/) returns rational numbers. There are no parentheses placed anywhere. We use the usual order of operations: multiplication and division happens before addition and subtraction. It's not allowed to use the unary negation operator (-). For example, "x x" valid expression as it only uses subtraction, but "-x + x" is not because it uses negation.
Figure 7: 2-shot prompt for test case inputs generation. Page 2.
24

---

Published as a conference paper at ICLR 2025
We would like expression with least number of operators that the expression equals the given target. Return the least number operators used.
Example 1:
Input:
langet
Output:
Explanation: 3/33 The expression contains operations.
Example 2:
Input: x 5, langet - 501
Output:
Explanation: 5 5 5 5 5 5 The expression contains 8 operations.
Example 3:
Input: 100, target 100000000
Output: 3
Explanation: 100 100 100 - 100. The expression contains 3 operations.
Note:
2 <= 100
target
class Solution:
laresthhpsiippesssiieeeeellll int, target: int)
#*## Response
"test case 11:
19)".
"test case "esssstppsEppeessss Tarpeess///sree(.... 32)",
"test case 2"; "leastOpslixpress Targett6, 100)",
INM Get Started
Note that in the ubove examples, I omit some test case inpuls. You should return **10a* groups of inputs to in jssonsojjer object' format.
#### Programming Problem
[[Question]]
НRиЕ Response
Figure 8: 2-shot prompt for test case inputs generation. Page 3.
25

---

Published as a conference paper at ICLR 2025
expert programmer. will show you programming problem. carefully comprehend the down solution program to pass under the given time and memory
requirements in the problem, constraints.
""REMEMBER* to strictly follow steps below to help reduce the potential flaws: (1) According to the input scale and the time/memory constraints, think about the time complexity and space
complexity of your solution.
(2) Think "steppby--tep** to design the ulgorithm.
(3) Translate your thoughts into Python progrum to solve it.
Besides, your Python solution progrum should be located between <BEGIN> and <END> tags:
<BEGIN>
t= int(input())
print(ans)
<END>
Here is example:
## Problem
You perform SnS actions with this array: during SiS-th action, the following sequence of operations appears: Choose maximum length subarray
given an aray Sa$ of length SnS consisting
(continuous subscgment) consisting only zeros, among all such segments choose the leftmost one: Let this segment S[1; rJS Sr-I+1S odd (not divisible by S2S) assign (sct) frae [2]] (where SiS the number current action), otherwise $r-1+ 1S even) assign (sct) Sa[tfrac [2+]-
length $5$ (initially (a- choose the segment S[I; 5JS and assign Sa[3] ;= 15, so $aS becomes S(0, 0, O, OJ$; then we choose the sogment $[1; 2J$ assign Sa[1] :- 25, so Sa$ becomes S[2, 0, 0. OJS; then we choose the segment S[4; 5]S and assign Sa[4] ;- 38, so SaS becomes $12, 0, 3, 0J$; then we choose segment $[2: 2JS and assign Sa[2] 4S, so Sa$ becomes S[2, 4, 1, OJS; and last we choose segment S[5; 5JS and assign $a[5] :- 55, so Sa$ becomes S12, 4, 1, 3, 5JS.
Consider array
0JS). Then changes follows: Firstly,
Your task to find the array $a$ of length SnS after performing all šnš actions, Note that the answer exists and unique.
You have to answer $1S independent test cases.
....IInpt--....
The first line of the input contains integer StS ($1 Me Mlle 10*4S) the number of test cases. Then StS test cases follow.
The only line of the test contains integer Snš (S1 Vie Me 2 lodot 10"5S) the length of $aS.
guaranteed that the of SnS all test does exceed $2 lodot 10"55 (Sisum Me ledot 10*5$).
Outputtt.-
For euch test case, print the unswer arry SaB length after performing Snß actions described in problem statement. Note that answer exists and unique.
Figure 9: 1-shot competition-lewe -evell code generation prompt. When being applied SFT, the 1-shot example is removed, Page 1.
26

---

Published as a conference paper at ICLR 2025
If there is no answer for the test case, print -1, Otherwise, the first line of the output should contain SkS (50 Me k We n$) - the minimal number of words in the set which should be reversed. The second line of the oulput should contain $kS distinct integers the indexes of the words in the set which should be reversed. Words are numerated from SIS to $n$ in the order they oppear. If Sk-0S you can skip this line (or you can print an emply line). If there many answers
print any of them.
Example...........
Input
4
4
0001
1000
3
101
2
00000
00001
4
001
0001
00001
Output
1
3
0
2
##tt Response
"test case "3'n3''m 10 1n 1'"1111", "test case 1n 110",
IMW Function Calling
НA## Programming Problem
Given single positive integer x, we will write an expression of the form x (opl) (op2) x (op3) x where each operator opl, op2, ctc. is cither addition, subtraction, multiplication, or division (+, or /). For example, with x 3, we might write 3 3/33 + 3 which is a value of When writing such an expression, we adhere to the following conventions:
The division operator (/) returns rutional numbers.
There parentheses placed anywhere. We use the usual order of operations: multiplication and division happens before addition and subtraction. It's not allowed to the unary negation operator (-). For example, "x-x" valid expression it only uses subtraction, but "-x x" is not because it uses negation.
1-shot competition-levee code generation prompt. When being applied to SFT, the 1-shot example is removed, Page 2.
Figure 10:
27

---

Published as a conference paper at ICLR 2025
We would like write an expression with least number of operators the expression equals the given target. Return the least number operutors used.
Example 1:
Input: - 3, langet - 19
Output:
Explanation: 3 + 3 3/3. The expression contains 5 operations.
Example 2:
Input: x 5, langet 501
Output: 8
Explanation: 5 5 5 5 5/5, The expression contains 8 operations.
Example 3:
Input: 100, target 100000000
Output:
Explanation: 100 100 100 100. The expression contains operations.
Note:
2 -x-x- 100 target <= 10'8
class Solution:
def PesstOOppiippressiirttftet x: int, target: int) -> int:
#A#H Response
"test case 0": Heeasttpssixppess//agrtt 19)".
"test case 1";
32)".
"test case 2"; "leustOpslixpressTinnn666 100)".
IM Gitt Started
Note that in the ubove examples, omit inputs. You should return **10a* groups of inputs object' format.
#aиe Programming Problem
[[Question]]
#AN Response
Figure 11: 1-shot competition-lewe code generation prompt. When being applied to SFT, the 1-shot example is removed, Page 3.
28


Reviews:
[{"review_id": "3jYMASeLZ3", "reviewer": "Reviewer_EPjK", "summary": "The authors propose a technique to generate pseudo-feedback using test cases. For math problem-solving, the test cases are unique (since each math problem has a singular, numerical answer, and a model either produces or doesn’t produce that answer). For code generation, multiple test cases are produced: the paper explores both test case generation using a frontier LLM and using the model being trained.\n\nThis approach provides a new method that does not rely on the high-quality human-verified data required for DPO. Implementing this approach with Llama-3.1-8B, Mathstral-7B, and Deepseek-coder-7B-v1.5-Instruct for math and code reasoning respectively, shows improvements from 5 to over 10% across various benchmarks.", "questions": "* Same as the weakness — I’d love to see whether performance on a subset of more challenging problems (which could be measured by looking at problems where the base models provided an incorrect answer even after applying self-consistency) improved over time with self-generated pseudo-feedback.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Creates the pseudo feedback framework in a way that combines both math and code reasoning tasks.\n* Because the pipeline for generating pseudo feedback is entirely automated, the process is scalable and less expensive than recreating such a process with humans would be.\n* The experiments are robust, conducted with three open LLMs, and with multiple combinations of techniques across two domains.\n* The results of the experiments are not only significant, but also well-explained. The authors do a good job of properly explaining any particularly interesting/surprising results.\n* The approach using frontier LLMs removes the need for ground-truth labels, enabling better use of unlabeled reasoning data for model improvement.\n* The authors provide an appendix with more interesting insights and all the necessary details needed to recreate their work/expand on it, making for a meaningful contribution to the field.", "weaknesses": "* The primary weaknesses I see with this approach are the limitations caused by self-consistency over self-generated pseudo feedback. I could see improvement over iterations being marginal for more challenging math/coding problems, as the self-consistent \"answer\" for them is likely incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HQ5j155zo6", "reviewer": "Reviewer_NHWB", "summary": "This paper introduces PFPO (Pseudo-Feedback Preference Optimization) that creates pseudo feedback for reasoning tasks by framing the labeling of solutions to reason problems as an evaluation against associated test cases. They explore two forms of pseudo feedback based on test cases: one generated by frontier LLMs and the other by extending self-consistency to multi-test-case. Experiments are conducted on math reasoning and code generation and observe performance improvements.", "questions": "* The method relies on frontier LLMs to generate pseudo feedback. How does the performance of the proposed method degrade if the frontier LLMs are not available or their performance is suboptimal?\n* How does the quality of pseudo feedback (from both frontier LLMs and self-consistency) impact the final performance of the LLM? Is there a way to measure or improve this quality?\n* Could the authors provide more details on the computational requirements and potential optimizations for the iterative training process?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper presents a novel approach to preference optimization by leveraging pseudo feedback.\n* The experiments are well-designed and comprehensive, covering both mathematical reasoning and coding tasks. The results show substantial improvements over baseline models and even surpass some state-of-the-art models, indicating the high quality of the proposed methods.\n* The paper is well-structured and clearly written. The methodology is explained in detail, and the experimental setup and results are presented in a way that is easy to follow.", "weaknesses": "Please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WcQ2ENh8w5", "reviewer": "Reviewer_dJUL", "summary": "This paper proposes Pseudo-Feedback Preference Optimization (PFPO), a novel approach for preference optimization in large language models (LLMs) tailored to reasoning tasks in mathematical and coding domains. PFPO sidesteps the dependency on human-verified labels by generating pseudo feedback through test cases, either created by frontier LLMs or using self-consistency from the policy LLM itself. Using Llama-3.1-8B/Mathstral-7B for mathematical reasoning and Deepseek-coder-7B-v1.5 for code generation, PFPO achieved notable improvements on the MATH, GSM8K, College Math, and LiveCodeBench benchmarks.", "questions": "1. PFPO with self-iteration clearly plateaus after several iterations. Can the authors provide more insights on what stage causes this, such as whether the quality of generated test cases plateaus or the policy model stops learning?\n2. In Table 1, the baseline score for Llama-3.1-8B-base seems missing. Could this be clarified?\n3. In section 4.2.2, synthetic test cases outperform ground truth, likely due to greater quantity. Since generated inputs and ground truth programs are available, why not run the programs on generated inputs to match the number of ground truth test cases, enabling a fairer comparison?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive experiments across a range of reasoning tasks and LLM models.\n2. Clear presentation and good formalization.\n3. Empirical results show the method’s effectiveness.", "weaknesses": "1. Limited Novelty at ICLR Level:  \n   The technique of leveraging unit tests as feedback for direct preference optimization (DPO/PPO) is already established in existing research. Previous works, such as CodeRL [1], and more recent studies [2][3], have incorporated unit tests and compiler feedback within reinforcement learning (RL)/PPO frameworks for code generation. Similarly, fine-grained feedback with DPO has been applied in mathematical reasoning [4], and LLM-based feedback with DPO has been explored for code generation [5]. Although PFPO Self-iter demonstrates strong performance, its approach of utilizing LLM-generated feedback signals for DPO closely aligns with these prior works, which limits its methodological originality.\n\n   - [1] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. C. H. (2022). CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. *Advances in Neural Information Processing Systems*, 35, 21314-21328.\n   - [2] Liu, J., Zhu, Y., Xiao, K., Fu, Q., Han, X., Yang, W., & Ye, D. (2023). Rltf: Reinforcement learning from unit test feedback. arXiv preprint arXiv:2307.04349.\n   - [3] Dou, S., et al. (2024). Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391.\n   - [4] Lai, X., et al. (2024). Step-dpo: Step-wise preference optimization for long-chain reasoning of LLMs. arXiv preprint arXiv:2406.18629.\n   - [5] Weyssow, M., Kamanda, A., & Sahraoui, H. (2024). CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences. arXiv preprint arXiv:2403.09032.\n\n2. Insufficient Discussion of Limitations in Self-Generated Pseudo Labels:  \n   The paper could benefit from a more thorough examination of the potential biases and risks associated with self-generated pseudo labels, such as model bias and the tendency to overfit synthetic feedback. A clearer analysis would provide a more robust perspective on PFPO's limitations. Additionally, connecting this limitation to the plateau effect noted in Question 1 would be insightful, as similar issues with DPO setups have been documented, as seen in [4]. Empirical evaluation or discussion on the effects of varying types and quantities (quesiton 3) of synthetic feedback could strengthen this analysis further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)