Iterative Reasoning Preference Optimization
Authors: Richard Yuanzhe Pang
Weizhe Yuan
He He
Kyunghyun Cho
Sainbayar Sukhbaatar
Jason E Weston
Venue: NeurIPS 2024 poster (2024)
Keywords: NLP
LLM
language models
iterative learning
reasoning
Primary area: natural_language_processing

Abstract: Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks. In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps. We train using a modified DPO loss with an additional negative log-likelihood term, which we find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy on GSM8K, MATH, and ARC-Challenge for Llama-2-70B-Chat, outperforming other Llama-2-based models not relying on additionally sourced datasets. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.

Iterative Reasoning Preference Optimization
Richard Yuanzhe Pangl?
Weizhe Yuan 1.2
Kyunghyun Cho?
He He
Sainbayar Sukhbaatar Jason Weston 1,2+
Meta FAIR
New York University
Abstract
Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks [Yuan et al., 2024, Chen et al., 2024]. In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thoughe (CoT) candidates by optimizing for winning steps. We train using a modified DPO loss [Rafailvv et al., 2023] with an additional negative log- log-likelihood term, which find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy GSM8K, MATH, and ARC- ARC-Challenge for 2-7ma-2-70B-hatt outperforming other Llama-2-bant models relying on additionally sourced datasets. For example, improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
losing reasoning
large
Introduction
Preference optimization has proven to give large gains when aligning pre-trained language models to human requirements compared to supervised fine-tuning alone [Ziegler et al., 2019, Stiennon et al., 2020]. Offline methods such as DPO [Rafailov et al., 2023] are becoming more popular for their simplicity and efficiency. Recent results have shown that iterative application of such an offline procedure is beneficial, whereby the updated model is used to construct that are more informative, and hence improve results further. These methods include Iterative DPO [Xu al., 2023, Xiong et al., 2023], 2024), and other methods [Rosset et al., 2024). Common to these approaches that they have been shown perform well general instruction tuning tasks, but they either make only moderate gains or even decrease the performance on standard reasoning tasks. While other kinds of iterative training methods have been applied successfully reasoning, particularly involving the iteration of supervised fine-tuning (SFT) such as STaR [Zelikman et al., 2022], R.Mt [Singh et al., 2024], and V-STaR [Hosseini et al., 2024]', using preference optimization train the generative reasoning model is not applied in these methods.
preference relations
LLMs [Yuan et al., 2024], SPIN [Chen et al.,
In this work, develop an approach apply iterative preference optimization to reasoning tasks, (CoT) reasoning [Wu et al., 2023]. On each iteration we
with particular focus on Chain-of-Thowe sample multiple mhai--of-though reasoning steps and final answers construct preference pairs such that pair winners have correct answers and pair losers have wrong answers. We then train variant of DPO that includes negative log-likeelin likeeleoeoodd (NLL) loss term for the pair winners, which also proves crucial for performance. Given the newly trained model, we then iterate the procedure by generating new pairs, and training again, starting from the previously trained
training prompts, and then
Equal contribution.
V-S does use preference optimization, but for training separate verifier model.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Chain-of- Thought Answer Generation
Preference Oplimization
Training prompts (m::)
Seed model (for (+1)
Generale
Generale
Compute rewards
Preference pairss
M
B
TOWARd ITOOM
select
(ct ki 10 14.
training
M
N
Next Iteration model
Figure Iterative Reasoning Preference Optimization. Our iterative preference optimization method consists of two steps: (i) Chain-of- ThTughhtt & Answer Generation: training prompts used to generate candidate reasoning steps and answers from model Mr. and then the answers are evaluated for correctness by given reward model. (ii) Preference Optimization: preference pairs selected from the generated data, which are used for training via a DPO+NLL oPo+tLLobjeective. resulting in model M++1. This whole procedure is then iterated resulting in improved reasoning ability on the next iteration, until performance saturates.
model. We find that reasoning performance improves over multiple iterations until it eventually saturates.
We show that our approach, termed Iterative Reasoning Preference Optimization (Iterative RPO), outperforms number of baselines, including SFT or applying standard DPO, well as other baselines from the literature. We see an improvement from 55.6% of zero-shot performance on GSM8K to 81.6% after our Iterative RPO training (or from 70.7% to 88.7% with majority voting out of 32 samples), from 77.8% to 86.7% on ARC-Challenge (without using the provided ARC Corpus), and from 12.5% to 20.8% of 32 samples), without using the provided pretraining corpus in MATH. We provide ablations that indicate the components that lead these improvements. We also present analysis objectives influence the probabilities of training sequences, which helps explain the success of our method. Overall, our method provides simple recipe that has the potential to improve the reasoning ability of LLMs over a wide range of tasks, as shown on the three tasks we consider.
MATH (or from 18.8% to 29.1% with majority voting out
how different
2 Iterative Reasoning Preference Optimization
Our approach first assumes access to a base, typically pretrained or instruction-ture language model, of training inputs, and the ability to judge the correctness of the final outputs. Given a training input, the language model is expected to generate (i) set of reasoning steps (Chain-off Chain-of-Thought) followed by (ii a final answer to the given problem. We assume that we have access to correctness measure for the final answer, and not for the correctness of the reasoning steps used to reach that answer. In our experiments, we thus consider datasets where gold labels are provided for training inputs, and a binary reward is derived by the exact match between these labels and the final answer generations. However, our approach could also be applied to settings with more general reward models.
On each iteration, our method consists of two steps, (i) Chain-of-Thought & Answer Generation and (ii) Preference Optimization, as shown in Figure 1. For the tth iteration, we use the current model M, step (i) to generate new data for training the next iteration's 's model M,41 in step (ii).
We assume containing questions T; and their correct answers y:- The model will be trained and updated at each iteration, resulting in models Mo. M,
Initialization.
given initial model My, and training set D {Ix,, y:))
Chain-of-thought & answer generation. Given the current model Mf, we generate N different responses for every input, where each response consists of CoT reasoning followed by final answer
(c' ( ~ Mc(x:) for all x4 E D and [N],
where we use [N] denote (1,2....,N)

---

In the general version of our approach, one then computes the reward r for each of these responses based on the correctness of their answers, i... r" R(y y:). In our experiments this simply corresponds to r = if yr" Vi. and 0 otherwise; i.e., whether the prediction matches the answer provided in the training dataset. Thus we have constructed a set of generated responses augmented with rewards:
G = {c' 3/ J
Preference optimization. In the next step, we first construct dataset of response pairs Dr based on the generations G; from the current model Mr The paired data is constructed such that chosen (winning) responses have higher rewards than rejected (losing) responses. This data is then used for preference optimization. In general, this can be done by selecting two responses for the same input, such that one has higher reward than the other, and setting the one with higher reward as the winner. In the binary reward case, we can split the generated responses G; into two sets based on their rewards:
G = (c' 1f - 1"
Next we build a dataset of preference pairs by selecting a winner response (o(" from G., and loser response (c/, v/) from G., In particular, we simply iterate over G and G simultaneously to produce K pairs of indices {(wk, (2)]. in order to ensure use much of the data as possible.
Drarr = ((,"",,, ")"",, (c w x, E D and & E [K]]-
Given the preference pairs, we can now train a new model Mg that will become our next model M The parameters 0 are initialized from model M,, and updated with a loss function that combines the DPO loss [Rafailov et al., 2023] for learning from the preference pairs, and the negative log- likelihood (NLL) loss for learning over the winning response from each pair. The loss corresponding to each preference pair is as follows:
LDPO-NLL = LDDO (c"", Ub cl. xalxe) + aLLLL(C, ( (x;))
(llog Mo(e," (.))):)) Blog Mo(c), yf uflara)) M.(e, v/(ors) ) log (c /.-l p0:)) (1)
= =- log
Micc"' ,/11),)
lo." 1v0"!
Here (z) denotes the probability of sequence under the model M, and o is the sigmoid function. We use the previous iteration's 's model Mt as the reference model in the denominator the DPO term. Note that the NLL term is normalized by the total response length. The hyperparameter balances the two loss terms. For brevity we omit the pair index k, but we optimize this loss on each of the k [K] pairs generated for every input sample. At the end of this training, we thus obtain our next model M,+1 = Mo. which will be then used to build data for the subsequent iteration.
Iterative training. Our overall procedure trains series of models MI,..., My where each successive model + uses preference data Drair created by the th model.
In our experiments, we define the models and the training data they use as follows:
Mo Base LLM; our experiments we initialize with fine-tuned instruction following model.
Mi Initialized with Mo. then trained with Deair using LDPONNLI.
M2 Initialized with Mj, then trained with pairs using CDPOINLL
M₃ Initialized with M2, then trained with D using LDPOINLL.
M4 Initialized with M3, then trained with Dgair using LDPO
This approach can be seen similar, but simpler, instance of the Self-Rewarding LLM training scheme proposed in Yuan et al. [2024], with three differences. Firstly, on each iteration in Self- Rewarding new set of prompts is created explore the input distribution, but in approach we use the same fixed set of prompts. Secondly, due to this choice our experimental setup does
II the iteration reaches the end of set, it restarts from the first element. If one of the sets is empty, then that input will be ignored.
3

---

not require a sophisticated reward model to judge the model generations, we assume the training prompts have provided gold labels which we compare to. These two omitted steps are challenging for reasoning tasks because they require a language model to verify correctness, which is known to be difficult [Huang et al., 2024]. Thirdly, we show that our DPO+NIL objective important for our LLM has used the standard DPO objective.
reasoning tasks, whereas
Our approach also related to the iterative training in the Self-Taught Reasoning (STaR) method [Ze- likman et al., 2022], except that their approach uses SFT training, rather than preference optimization using DPO-like training. Preference optimization allows the use of negative examples of reasoning chains and answers, which we show improves performance. See Section 4 for more discussion of related work.
3. Experiments
3.1 Math Word Problems: GSM8K
In our first set of experiments, we use the GSM8K dataset [Cobbe al., 2021]² that contains real grade-school math word problems. For example the question: "Natalia sold clips to 48 her friends in April, and then she sold half as many clips May. How many clips did Natalia sell altogether in April and May?". These questions typically require the model to perform intermediate reasoning, i.c., generating chain-of-thought before answering, otherwise performance poor. Each problem contains a question F,, gold ofann-of-thought solution G, and a final numerical answer y;- For our entire training process, we only use the training set of around 7.5k problems without any extra questions.
Experimental setup. As a seed model Mo we use the chat version of Llama-2 70B model [Touvron al., 2023], which is instruction fine-tuned. We use a zero-shot prompt containing the question logether with instructions to produce a chain-of-thought and to follow specific format so the final answer can be easily extracted (the exact prompt given in Appendix B.2). In each iteration, we generate N- = 30 solutions per problem using sampling with temperature 0.8 for iterations 1-2 and temperature 1.3 for iterations 3-4 (hoping that there is significant number of incorrect generations in later iterations). Since some problems might not have any model-genent geder-geeee correct solution, we include the gold human written solution (G,, yt) in the winning set G so not empty. Then we generate K 10 pairs per problem for training with our loss in Equation 1, and filter out examples that were too long terms of overflowing the context length or else do not have any incorrect generations. This procedure gives around 55-60k pairs for training, per iteration."
In total, we perform four iterations, producing models Mi, M2. Ma. and M4. For each iteration, we train a maximum of 5000 steps, and then select the best checkpoint using a held-out Ik samples from the training set. We then retrain while including those 1k samples for the selected number of steps. The coefficient a is tuned in [0.25, 0.5, 1, 2 when training M1, and we end up using I for all experiments in the paper. The coefficient | in the DPO loss is tuned in (0.05, 0.1, 0.5, 1.0), and we end up using 0.1 in this experiment. We use a batch size of 16 and learning rate 7e-7 using the AdamW W optimizer. Throughout this paper, all generation is done using one node containing eight V100 GPUs (32G memory). To do inference efficiently, we use vLLM [Kwon et al., 2023]. All training is done using eight nodes each containing eight A 1000 GPUs (80G memory).
Overall results are given in Table 1, where we give the exact match accuracy on the GSM8K test set.
Iterative RPO improves over baselines. We find that Iterative RPO outperforms zero-shot CoT, CoT solutions, and variants of DPO by a
supervised fine-tuning (SFT) on the gold
wide margin. SFT gives boost in performance compared zero-shot CoT from 55.6% to 63.5 63.5% but still far from the 31.6% of Iterative RPO. We apply standard DPO to the same set of preference pairs D as used in the first iteration of our method. Whether initializing from alama-2--70--n- (Mp) or from SET training on the chosen (winner) examples, we find that DPO performance, while
We have contirmed that the licenses of the datasets used in this paper (MIT for GSM8K and MATH, CC BY-SA 4.0 for ARC) are respected. 'If after filtering the number of pairs is larger than 60k. then we randomly select around 60k examples. This number is fixed because do not want to introduce another source of variability in experiments.

---

Table 1: GSM8K results comparing Iterative Reasoning Preference Optimization (Iterative RPO)
against other baselines that are based on the same base model and training data. We report the exact match accuracy from a single generation (using greedy decoding), as well as majority voting over 32 generations (through sampling with temperature 0.8).
Model
Test Accuracy (%)
Iterative RPO (initialized from Llama-2-70b-chhe
Iteration I
73.1
Iteration 2
78,0
Iteration 3
81.1
w/ majority voting using samples
88.2
Iteration 4
81.6
w/ majority voting using samples
88.7
Other Llama-2- 707---oat-inittiine
methods
Zero-shot CoT
55.6
w/ majority voting using 32 samples
70.7
DPO initialized from Llama-2-70b-chat
61.8
DPO initialized from SFT trained on Iteration I chosen seqs
60.3
SFT on gold CoT examples
63.5
STaR (I iteration)
65.2
STaR (I iteration, but on twice as much data)
66.9
Iterative RPO (1 iteration, but initialized from SFT trained on chosen seqs)
73.1
Iterative RPO (1 iteration, but on twice as much data)
74.8
-200-
-200
Log probatillty
Log probalality
-400
-400
-600
SFT chosen rejected
-600
gold, legproh SFT on gold, logprob measured on chosen gold. logprob measured rejected
DPO-NLL chosen
-800
DPO-NLL relected
-800
3000
1000
1500 2000
Training steps
Training steps
(a) SFT trained on chosen seqs; init from Llama
SFT trained gold CoTs; init from Llama
Figure 2: Effect of SFT training. (a) Although SFT training (solid green) is on chosen sequences (DDairs from iterative RPO iteration 1) only, the rejected sequence log probabilities (dotted green) also increase and are close to the chosen sequence probabilities. contrast, our DPO+NLL training (blue) manages to decrease the rejected probabilities while increasing the chosen probabilities. This observation could potentially help explain why SFT-only performance lags significantly behind Iterative RPO Iteration performance. (b) We show similar plot but where SFT is trained gold (dataset-provee provided)) CoTs. Chosen and rejected sequence probabilities (which are from D are still close to each other, but with a slightly bigger gap. Another observation is that the chosen sequence probabilities barely increase.
being better than zero-shot CoT, is no better than the SET model, with accuracies of 61.8% or 60.3% respectively.
also show that SFT on only the chosen CoT solutions, which corresponds to the first iteration of the STaR method, improves results 65.2% over SFT on the gold solutions alone, but still falls short of the performance of the first iteration of Iterative RPO. One hypothesis for these improvements the necessity of including the rejected sequences in the training objective; otherwise their probability
5

---

-200
-200
probatillty
Log probalality
-400
4400
-600
DPO chosen
-600
DPQ chosen
DPO chosen
D chosen
-800
DPO-NLL rejected
-800
DPO+NLL rejected
500
1000
1500
2000
3000
1000
1500
3000
Training steps
Training steps
Initialized from Llama
(b) Initialized from SET trained chosen seqs
Figure 3: Efffcc of NLL loss term on DPO training for GSM8K. In our GSM8K experiments we observe the log probability of chosen sequences in standard DPO without NLL loss (solid orange) decreases over training steps, especially if the model is initialized from SFT training chosen sequences (right). However, they increase over training steps when using DPO with NLL loss (solid blue). In all four settings, the margin between the two curves continues increasing. We find that DPO+NLL loss gives superior test accuracy in our experiments.
increases along with the chosen samples; see Figure 2. We note this observation has also been reported in concurrent work [Hong et al., 2024].
All of the results reported above using a single generation test time using greedy decoding. If we use majority voting over 32 samples (sampling with temperature 0.8), a standard approach to improve performance the literature, we can improve the accuracy of our approach from 81.1% to 88.2% for iteration 3, and from 81 81.6% to 88. 78 for iteration 4 of Iterative RPO. While performance much improved using majority vote, this should be compared majority vote baseline, where we find similarly large improvement over the zero-shot chain-of- offthoughtt with majority vote, which obtains an accuracy of 70.7%.
Iterations of Iterative RPO yield improved reasoning. We observe that Iterative RPO provides
improvements over its training iterations, increasing the base model accuracy by 47% (from 55.6% to 81.6%) in total. In contrast, supervised training using the gold CoT only brings about 14% accuracy boost. We see performance improves across each iteration, from 73. 5. to 78. 78.0% to 81. 31.1% to 81.6%. However, the gain decays across the iterations (17.5%. 4.9%. 3.1,, 0.5%). indicating an upper limit on learning across iterations, especially as we are iterating across a fixed number of prompts, i.c., only from the training samples.
We also show that it the iterations of updating the model (i.e., initializing from the previous model) that are helping, not just because there is more data in the form of new pairs generated from the fixed training set. To test this statement, we run the first iteration of Iterative RPO but on twice as much paired data by doubling K, and we run the STaR method first iteration with twice as much data as well.5 In both cases performance improves compared to less data, but not as much as performing two iterations. Iterative RPO with twice as much data obtains 74.8% (an improvement over 73. % using the original dataset size); however, training for two iterations obtains 78.0%. For STaR, training on twice much data obtains 66.9%, compared to 65.2% with the original data, which is still much lower performance than Iterative RPO.
NLL loss necessary in our method: DPO with NLL DPO without NLL. The first iteration of our method can be compared to standard DPO training, which uses the same preference data, as reported in Table 1. We a large performance drop (73.1 % 61.8%) using DPO compared to our method after one iteration. The gap remains large even when the standard DPO training starts from the superior SFT-tuned model, which it has been argued improves DPO's performance [Rafailov et al., 2023, 2024]. Our results support the need of the NLL loss term in our training, not just using SFT for initialization. To further understand this result, we plot the sequence- log probability seeuenceee
'I particular, for "twice as much data" in Table 1, we generated more synthetic responses, so examples are not simple duplication.

---

over training steps for these methods in Figure 3. We see that for DPO without NLL loss there is a decrease over training for the chosen sequences, whereas for DPO with NLL there not, which may help explain the improved performance of the latter. Related observations have been made elsewhere in various settings [Pal et al., 2024, Xu et al., 2024, Hong et al., 2024]. Further, we note that whether we initialize with -Jama-2--70-chhn or SFT on chosen for Iterative RPO, accuracy results of first iteration training do not seem to deviate (both obtain the same score 73.1%). This is another advantage of our method as the training process is simpler without the SFT step.
Other results in the literature. We can compare our results to others in the literature, even if their experiments are in different settings. Touvron et al. [2023] reports an accuracy of 56.8% for 8-shot Llama-2-70b, which is close our zero-shot CoT results for Alama-2-70b-chhe of closed-source proprietary language models, some results are superior to ours, while others are not; for example GPT-4 obtains 92.0% (5-shot chain-of-thought) [Achiam et al., 2023], Claude obtains 88.0% [Anthropic Team, 2023], PaLM 2 obtains 80.7% [Anil et al., 2023], while GPT-3.5 obtains 57.1% (5-shot) [Achiam et al., 2023]. We note that the size (number of parameters) and the makeup of the training set of some of these models have not been fully disclosed. For results that use the same size and class model, Llama-2-70b, MetaMath [Yu et al., 2024] reports an accuracy of 82.3%, while WizardMath reports 81.6% [Luo et al., 2023]. These last two results use additional augmented training data, whereas our method does not use additional prompts. Such approaches should be orthogonal to ours, and both can provide benefits.
In terms
3.2 ARC-Challenge Task
To test reasoning capabilities outside of mathematics, we employ ARC [Clark et al., 2018] which covers multiple science subjects. Questions are multiple-choice for example: "A fold observed in layers of sedimentary rock most likely resulted from" " with four possible answers, e.g., "(A) cooling of flowing magma, (B) converging of crustal plates, (C) deposition of river sediments, or (D) solution of carbonute minerals". 'he training dataset contains 7.7k questions split into easy and challenge sets. We report results on the ARC-Challenge test set which has 1172 examples. There is no gold chain-of- of-nouht reasoning provided for training examples in this task. Our method does not have that requirement and hence can still be applied as we only compute rewards based on the final answer. One consequence however is that if there is model gener-ggeeeraed correct solution for a question, then that question is not included in our training. We follow the same setup as before to first generate reasoning and then a final answer by the models (see Appendix B.2 for prompt) to construct data for iterations of Iterative RPO. We only train on the training set (both easy and challenge sets) and do not utilize the supporting ARC Corpus.
Specifically, in each iteration, we generate N' = 30 solutions per problem using sampling with temperature 0.8 for iterations 1-2 and temperature 1.3 for iteration 3. We select K = 20 pairs of solutions per problem. We end up with around 20k example pairs for iteration 1, 1 1k example pairs for iteration 2, and 5k example pairs for iteration 3. The decrease in the number of examples is due to the lack of incorrect samples for a number of questions in later iterations. Each iteration is trained on maximum of 4000 steps. The hyperparameter tuning relies the provided development set.
We hence perform experiments using very similar setup to the one previously described for GSM8K. Overall results are given in Table 2. We again find that Iterative RPO provides increased performance across iterations (84.8%, 86.2%, 36.7%) over three iterations. Majority voting using the model in the third iteration (32 samples, temperature 0.8) leads to another small boost (87.9%). These results outperform zero-shot CoT (77.8%), SFT on chosen sequences (79.8%) and standard DPO (83.5%). We arrive at similar observations in Figure 4a compared to Figure 3: when training with DPO without NLL loss, the log probabilities of chosen sequences barely increase over training; when training with DPO with NLL loss, the log probabilities increase noticeably.
Even though we arrive at similar conclusions to the ones from GSM8K, we find these results especially noteworthy due to the multiple-chote choice nature of the task. As there are typically only four possible answers, the generated data step (i) of Iterative RPO may provide a CoT and a final answer that is correct by luck (as random guessing is correct 25% of the time). Hence, the nature of the task may introduce significant amount of noise in the CoT generations used preference optimization in step (ii. Nevertheless, the method seems robust to this issue and we still observe performance gains.
7

---

Table 2: ARC and MATH results. We compare Iterative Reasoning Preference Optimization (Iterative RPO) against other baselines that are based on the same base model and training data.
ARC-Challenge (0-shot)
MATH
Model
(4-shot)
Test Accuracy (%
Test Accuracy (%)
Iterative RPO (initialized from 1aamm----70b-chan
Iteration /
84.8
17.7
Iteration 2
86.2
19,9
Iteration 3
86.7
20.8
w/ majority voting using 32 samples
87.9
29.1
Other (lama-2-70bccht-iniitieee CoT
methods
77.8
12.5
w/ majority voting using 32 samples SET on chosen sequences DPO initialized from Llama-2-700 2-1o--2--b-htt
82.9
18.8
79.8
16.8
82.8
12.4
DPO init from SFT model trained on chosen seqs
83.5
10.5
200
Log probability
-400
400
-600
Log
-600
DPO chosen
rejected
DPO+NLL chosen
-1000
800
-1200
3000
500
1000
1500
2000
2500
3000
Training steps
Training steps
(a) ERC-Challenge
(b) MATH
Figure 4: Effect of NLL loss term DPO training for ARC and MATH. The legend on the right plot omitted due to space constraint, but GSM8K, in ARC-Challenge and MATH, increase over training steps when training with DPO. However, when training with DPO with NLL loss. the log probabilities increase over training steps.
same as the legend the left plot. Similar to that the log probabilities of chosen sequences barely
3.3 MATH Task
We also experiment with more advanced math problems using the MATH [Hendrycks et al., 2021] dataset that is composed of 12,500 competition problems, for example the question: "Tom has red marble, a green marble, a blue marble, and three identical yellow marbles. How many different groups of two marbles can Tom choose?". While this may look superficially similar to the GSM8K task, features substantially harder questions, as will be shown by the baseline performance. The test set has 5,000 examples. Similar to the GSM8K dataset, gold CoT solution is provided for each problem, and the gold answers can be matched uniquely to predicted answers after normalization to compute rewards. We do not use the accompanying pretraining data. For each MATH question, use few-shot prompt given in Appendix B.2 as the input to the language model. In particular, the prompt includes four fixed in-context examples chosen from the training set. The language model needs these demonstrations so that the final answers can be properly formatted 18TEX.
In each iteration, we generate N 20 solutions per problem using sampling with temperature 0.8 for 15 pairs of solutions per problem,
iterations 1-2 and temperature 1.0 for iteration We select and after filtering out pairs with overly long generations, for each iteration we end up with around 75k example pairs. We train maximum of 5000 steps per iteration; other details are similar GSM8K setups.

---

Results are given in Table 2. We again find that Iterative RPO provides increased performance across iterations, from 17.7% 19.9% to 20.8% over three iterations. Majority voting (32 samples. temperature 0.8) leads to significant boost in performance (29.1%). These results outperform few-shot CoT (12.5%), SFT on chosen sequences (16.8%) and standard DPO (12.4%). In particular, DPO degrades the performance compared to initialization. Similar to the previous tasks, we show the si-probabilities during training in Figure 4b.
Overall, we find on all three distinct tasks we tried, from simpler to more difficult, similar observations on performance gains are exhibited by our method.
4 Related Work
General iterative alignment methods. Several works have implemented iterative reinforcement learning from human feedback (RLHF) with human-in -aaa----tth---oop to provide additional labels to retrain the reward model at each iteration, e.g., via Proximal Policy Optimization (PPO) [Schulman al., 2017]. reporting improvements across iterations [Bai et al., 2022, Touvron et al., 2023]. Recently, approaches have been proposed to perform iterative alignment without a human-in-the-lowe Iterative DPO [Xu et al., 2023, Xiong et al., 2023] optimizes preference pairs using DPO [Rafailov et al., 2023] at each iteration, and then constructs new preference pairs for the next iteration by generating them using the updated model, and scoring them using a reward model. Other iterative methods than DPO exist well, such the Cringe loss [Adoiphs et al., 2023], Pairwise Cringe Loss [Xu et al., 2023], and ReST [Gulcehre et al., 2023].
SPIN [Chen et al., 2024] is an Iterative DPO- like framework that uses human labels as the winning response in pair, and the last iteration's generations as the losing response the pair. The authors note this has the limitation that once the model generations reach human performance, they are bottlenecked. Further, each input prompt is required to have human-annotated generation. In contrast, our work only requires the final answer, but not the reasoning steps, and crucially uses the model to generate both winning and losing Chain-of-Thoughlee Only modest gains on reasoning tasks are reported in their work.
Self-Rewarding LLMs [Yuan et al., 2024] also use Iterative DPO with the LLM itself used reward model to construct pairs for each successive iteration. Both that work and the work of Rosset et al. [2024] and Snorkel AI Team [2023], which do similar iterations but with external reward models, show sigmificant gains on general instruction following tasks. However, again, only modest gains on reasoning tasks are reported.
Methods improving reasoning ability. While a number of approaches have been developed to curate or distill training data for reasoning tasks [Yu et al., 2024, Toshniwal et al., 2024], this work we focus on learning algorithms which is an orthogonal axis. Expert Iteration assumes a reward model, and repeatedly uses rejection sampling to filter generations and train on them, which is found to match the sample complexity of PPO [Havrilla et al., 2024]. STaR [Zelikman et al., 2022] relies on similar loop: generate rationales to answer many questions, prompted with a few rationale examples; if the generated answers are wrong, try again to generate a rationale given the correct answer; and then fine-tune on all the rationales that ultimately yielded correct answers; and repeat. ReSTEM [Singh et al., 2024] assumes a ground truth verifier and also fine-tunes on filtered samples in repeated fashion. All these methods rely on finding high-quality samples for SFT-like training, rather than using DPO-like pairwise preference optimization in our work.
The V-STaR method [Hosseini et al., 2024] trains a verifier using DPO and uses this to filter the generations of a model trained by SFT, rather than using DPO to train the generator, as we do. MAPO [She et al., 2024] also recently utilizes DPO but for multilingual reasoning tasks, where they translate across languages.
5 Conclusion
We propose an iterative training algorithm, Iterative Reasoning Preference Optimization, for improv- ing
reasoning task performance in LLMss. In each iteration, we generate multiple responses and build preference pairs based on the correctness of their final answers, and then use modified DPO loss with additional NLL term for training. Our method does not

---

require human- in-the-loop or extra training data, and remains simple and efficient to implement. The experimental results show large improvements on GMS8K, MATH, and ARC-Challenge over various baselines using the same base model and training data. These results indicate the effectiveness of our recipe of iterative training in improving the reasoning capabilities of L.MMs.
Acknowledgments
We thank colleagues at Meta and NYU for valuable discussion: in particular, Angelica Chen, Jing Xu, Abulhair Saparov, Vishakh Padmakumar, Nicholas Lourie, Nitish Joshi, Ilia Kulikov, and Mark Hou.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.08777 08774, 2023.
Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. Learning what language not model. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 8854-8874, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2223.acll acl-oogg...-- URL attps://aclanthollo. org/2023........ooo....o........ acl acl-long.44 493.
The CRINGE loss:
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaI.M 2 technical report. arXiv preprini arXiv:2305. /0403, 2023.
Anthropic Team. Claude 2, 2023. URL attps://wwt http///www.tttrpp.c.... www./ooww/oaaaeeeeeeeeee 2.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0 05862, 2022.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Forty-first International Conference on Machine Learning. 2024. UtLpstts:: httpl::s///pppeeviieeee www.iiiwweeeeeeeeeeeeeeeee.eeeee wwt.:nnnnnuueeee. wow.omidd-04cHtteeeee
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint arXiv: 080355 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 14168, 2021.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (ReST) for language modeling. arXiv preprint arXiv:2308.0800 2023.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models reason with reinforcement learning. arXiv preprint arXiv:2403. 24444420.. 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track met:/foruniid--Byrttnnnes
(Round 2), 2021. URL bectps://openrew///
Jiwoo Hong, Noah Lee, and James Thome. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024.
10

---

Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training verifiers for self-taught reasoners. In First Conference on Language Modeling, 2024. URL. htttps://openrevieer net/foruum''d=stmqBBBi
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models camnot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024, URL htttpp///opnrrvvee net/ forum?1d=T<<D33KKD
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arX1v:2308.09583 2023.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO-positive. arXiv preprint arXiv:2402. 13228, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:/ http://www...eeeetttd/a/aaeaeeea
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to Q': Your language model is secretly a Q-function. First Conference on Language Modeling, 2024. URL https:// i http:/fforumiddkkEeiietttee kEVcNxxtqXX
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct Nash optimization: Teaching language models self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707006777 2017.
Shuaijie She, Wei Zou, Shujian Huang, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. htwpgggulllliiiiaeena---rrrreeee
opti- mization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10015-10007, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.186653/1/2022.aaall
MAPO: Advancing multilingual reasoning through
URL https: https://aclannhoologg org/2024 acl mocl-long... 539.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling telf-iraining for problem-solving with language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httphtts//opeereiiieeeee http://ppnrrviie...eee/rrrrrrr
net hett//oruu//dd-MMMyygggFFCF Expert Certification.
Snorkel AI Team. monnkellmisttrll-piimm..
attps://huggingface co/snorkelai/ 2023. Accessed: 2024-04-15.
Moankell-Ksstral-Paiiiea
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei. and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3000 30201 2020.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Git- man. OpenMathinsttuettl:: A 1.8 million math instruction tuning dataset. arXiv preprint 10176 2024.
11

---

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXin:2307.092888 2023.
Dingjun Wu, Jing Zhang, and Xinmei Huang. Chain of thought prompting elicits knowledge augmentation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL 2023, pages 6519-6534 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18633/v1/2023 fin.:66n/i.22...nnnnnnn----- findings-acl.eeee acl.nns. URL https://aclaatthllo00 org org/2023.... findings-aaee acl 408.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sam- pling from human feedback: A provable KL-constrained framework for RLHF. arXiv preprint arXiv: 1456, 2023.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, 2024. URL https::t////. http::tttt:///pett
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.1668,, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference Learning Representations, 2024. URL https:tt///// /ttp:::s///ppeevvviiee
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024. URL https: /ttpps://ppnttt /opp///eeeeeeeeeee//eeee net/ ftffmorrddd 1iriidd--Ohpoooe Cnggu.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping reasoning with reasoning. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: ptttts://ooennevieee net/ Siiiiiddd=EEEdd2ase
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1rXxv:1909.08593 2019.
12

---

A Limitations
On experiments: When training iteration t using iterative RPO, we not make use of the collected data in previous iterations. Utilizing those data could potentially boost the performance even more. We leave this point to future work, as not central our theses. In addition, we have experimented on three tasks. It is unclear how the approach would perform general instruction tuning tasks without a clear best answer, but we argue that positive results on the three tasks in this paper can already prove the method useful. The current recipe requires correct answers, and a clear metric for comparing a generated response with this correct answer.
Regarding our loss function: The NLL loss is shown to be helpful in our case. Our iterative RPO algorithm requires training data to be mostly collected from the previous iteration of the model. Therefore, the chosen and rejected sequences all have reasonably high probability under the model distribution. When training sequences are arbitrary (e.g., sampled from other models), it is unclear whether the NLL loss is necessary (although this setting does not fall under the umbrella of the iterative RPO procedure this paper).
B
More Details on Experimental Setup
B.1 More Details Hyperparameters
For baseline hyperparameter selection: We conducted a grid search for SFT, iteration 1 of DPO, and iteration 1 of RPO. For iteration>>,, I, we use the same hyperparameters as iteration 1, except for the number of training steps which is selected individually for each iteration. For all baselines, learning rates all tuned from the range of Se-7 to 5e-6. DPO and RPO tuned in the same sets of hyperparameters (if the hyperparameter exists in both methods).
B.2 Prompts
GSMSK. For each GSM8K question, we use the following prompt as the input the language model:
Your task is answer the question below. Give step by step reasoning before you answer, and when you're re ready answer, please use the format "Final answer: ...
Question:
[question here]
Solution:
MATH.
For each MATH question, we use the following prompt as the input to the language model. In particular, the prompt includes four fixed in-context examples chosen from the training set of MATH. The language model needs these demonstrations so that the final answers can be properly formatted in I8TX..
Your task is to answer the last question below. Give step by step reasoning before answer, and when you're re ready answer, please wrap your answer in \boxed, and conclude using the format "Final answer: ...
Question: [question for the first example]
Solution: [solution for the first example]
Final answer: [answer (e.g., number, formula) here]
Question: [question for the second example]
Solution: [solution for the sccond example]
Final answer: [answer here]
Question: [question for the third example]
Solution: [solution for third example]
Final answer: [answer here]
Question: (question for the fourth example] Solution: [solution for the fourth example]
Final answer: [answer here]
13

---

Question:
[the question to be solved]
Solution:
For each ARC question, we use the following prompt as the input to the language model, assuming the question has four options (each question has three to five options).
ARC.
Your task answer the question below. Give step by step reasoning before you answer, and when you' re ready to answer, conclude using the format "Final answer: (insert letter here)"
Question: [question here]
(A) [option here]
(B) loption here]
(C) loption here]
(D) Loption D here]
Solution:
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and the introduction cover the paper's contributions algorithm, performance, ablations, and further analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix A for detailed discussion on limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
is an empirical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental details are provided at the beginning of each subsection in the Experiments section. The prompts used are included in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: This decision still pending. However, our experiment mostly involves relatively straightforward modifications of the DPO algorithm, and all important details included in the beginning of each subsection Experiments. Prompts are also included in the appendix. Given these details, it should be straightforward to replicate the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submíssion guidelines (https://nips.. cc/ public/ge htte//uudd///eeeeeee "CodeSubmissionn
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wwtpppccc/ppccctcccccccccccccctt wwwpc//ggg///eeeeeeeeeeeeeeeeeee /tceeuuuiddeeeeeuuubmiiiiii SeseeeuummsssionnP..
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity. the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details provided in the beginning of each subsection in Experiments. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments are run once given the cost of fine-tuning 70B models. The difference from baseline performance often very large as indicated in the discussion the Experiments section. In addition, we are following the norms of existing publications that experiment on these datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Mentioned toward the start of the GSM8K subsection Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / //eurips....e//
Question:
Answer: [Yes]
Justification: The paper proposes a method improve the performance on reasoning benchmarks. We do not see relevant violations as listed on the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not see significant positive or negative societal impacts, as the work provides a training algorithm that is applicable reasoning tasks like mathematics and commonsense.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The benchmarks are popular and low-risk. We are not planning to release models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The licenses mentioned in a footnote near the beginning of Experiments. We have made sure that the licenses are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "aYKwKlmwC0", "reviewer": "Reviewer_jeVs", "summary": "The paper introduces a novel approach to improve the performance of language models on reasoning tasks through iterative preference optimization. It proposes an iterative method that generates multiple reasoning steps and final answers, constructs preference pairs based on the correctness of the answers, and then optimizes these pairs using a modified Direct Preference Optimization (DPO) loss combined with a negative log-likelihood (NLL) term. This iterative process results in progressively improved model performance. The approach demonstrates significant accuracy improvements on the GSM8K, MATH, and ARC-Challenge datasets using the Llama-2-70B-Chat model, outperforming other models that do not rely on additional datasets. Key contributions include the iterative application of DPO with an NLL term for reasoning tasks, comprehensive experimental validation, and performance gains without the need for additional human-annotated data.", "questions": "Although the authors have discussed other related iterative preference optimization methods (e.x. Iterative DPO, Self-rewarding, SPIN), why not include their results in the evaluation tables?", "limitations": "See Questions.", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written, and easy to follow.\n2. The idea is clean and the contribution is clear.\n3. The evaluation and improvements are convincing and significant.", "weaknesses": "I do not see significant weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and the support! \n\nWe presented DPO results in the paper. We showed that DPO in iteration 1 does significantly worse than our RPO’s iteration 1; hence we did not try further iterations of standard DPO (aka Iterative DPO). The reviewer raises a good point that we can check what future iteration results are like. SPIN doesn’t include the NLL term in IRPO, so it’s closer to iterative DPO than IRPO; moreover, SPIN assumes always using the reference CoT but we do not; they also report only modest gains in reasoning tasks.   \n\nThe reviewer raises a good point about self-rewarding LM. The original algorithm requires generating prompts and evaluating the sampled responses using the LM itself. IRPO assumes knowing what the correct answer is (e.g., the answer may be 7.5 for a math question), but we wouldn’t know the correct answers for augmented prompts. LMs are quite bad at generating evaluations of a response (e.g., according to RewardBench – although there has been some progress in the past few weeks). So we believe that self-rewarding LM cannot be successfully directly applied using llama-2-70b-chat, but using IRPO in an unsupervised fashion is a really promising research avenue to keep exploring!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVNcpXvFFb", "reviewer": "Reviewer_9NDK", "summary": "This paper proposes a novel method for preference optimization for reasoning tasks. Their method involves prompting models to generate the CoT reasoning steps and answers for a set of reasoning task inputs, then labeling samples as correct or incorrect using the ground truth outputs, and training the preference model. The authors use standard DPO loss with the addition of a negative sampling loss using samples for which the model generates incorrect outputs. They train models iteratively, using the previous generation as the base model to generate new outputs at each step. They observe that using this method they are able to improve significantly over DPO for three reasoning tasks.", "questions": "1. What hyperparameters are used for DPO and SFT? Are they tuned with the same care as those for RPO?\n\n2. Do you explore the reasoning steps generated by models? How often are they actually correct?\n\n3. What is the cost of training models using this method?", "limitations": "The authors address limitations in the appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "RPO demonstrates performance boosts on a variety of tasks, outperforming DPO and SFT. Improving model performance for reasoning tasks has shown to be a challenging task, and this method shows promise for improving model performance in this area.", "weaknesses": "1. Although the authors present ablations on the number of data points for RPO, they do not present experiments with simply training the model for longer. This is true for the comparisons to DPO as well. Though iterative training likely is playing a role in the performance of the model, for the sake of careful analysis, it would be good to compare the performance of a model trained for multiple generations vs a model trained for only one generation but an equivalent number of steps.\n\n2. I was not able to find details on the hyperparameters used for DPO or SFT or how optimized they were. In contrast, the hyperparameters for RPO are chosen carefully. While it's unlikely all performance boosts are due to this, it should be clear how hyperparameters were chosen for the sake of comparison and reproducibility.\n\n3.  Though the comparison of DPO with and without NLL shows that the loss for rejected samples decreases when NLL is applied, it does not demonstrate why this is important for model improvement. A more effective analysis would compare the mistakes made by the model with and without this loss (e.g. FNR/TNR).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and suggestions!\n\n\n**Weakness 1 (longer training)**: Thank you for pointing out this issue. In each iteration, we do train longer (5000 steps) but end up selecting an earlier checkpoint – the selected checkpoints (by validation accuracy) are usually trained for 2000 or 3000 steps (e.g., for GSM8K, M1 is trained for 2000 steps and M2 for 2000 steps). This is because training longer makes the model overfit to the training and hurts validation performance. We use the same checkpoint selection process for both RPO and DPO. We’ll include more detailed discussion in the revision.  \n\nImportantly, in Table 1 of the submission, we also included results where the model is trained on twice as much data instead of two iterations (using STaR and using iterative RPO, also for a max of 5000 steps). As shown in Table 1, this strategy (the last row) doesn’t match the performance of doing two separate iterations (2nd row). This highlights the advantage of iterations over longer training.\n\nAnother piece of evidence (that iterative training is helpful) is that recent work (Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data; https://arxiv.org/abs/2404.14367) has demonstrated the importance for on-policy data (i.e., iterations). We’ll make sure to discuss this issue more thoroughly. \n\n\n**Weakness 2 (hyperparameters for baselines)**: We conducted a grid search for SFT, iteration 1 of DPO, and iteration 1 of RPO. For iteration>1, we use the same hyperparameter as iteration=1 (except for num_training_step which is selected individually for each iteration). All experiments share a similar config file; therefore, for all baselines, learning rates are all tuned from the range of 5e-7 to 5e-6. DPO and RPO are tuned in the same sets of hyperparameters (if the hyperparameter exists in both methods). We will clarify this issue, and thanks for pointing this out.\n\n\n**Weakness 3**: With DPO training, the probability of rejected samples decreases regardless of NLL (see Fig3 a). Addition of NLL affects the chosen samples and makes them go up in probability. In contrast, their probability doesn’t go up much in DPO training without NLL. The chosen samples are correct solutions, so their higher probability means the model is likely to generate that correct solution. The direct way to measure if the model is generating correct solutions or not is to look at the test set accuracy, which we report in the paper.\n\nIn other words, our hypothesis is that without NLL, both chosen and rejected probabilities decrease, and given that probabilities sum to one over sequences, the probabilities might be going to unforeseen low-quality sequences. \n\n\nQ1: Yes. Please see weakness 2 above. Thanks again for raising this point.\n\nQ2: We have eyeballed ~30 generations for each dataset; to the best of our ability (given that MATH is difficult), if the answer matches, then our generated CoT is almost always correct for these particular datasets, especially GSM8K and MATH. \n\nQ3: Generation is relatively cheap these days given the low-precision toolkits (e.g., vLLM; https://github.com/vllm-project/vllm) – using 8 V100 (32GB) GPUs, we can sample a few thousand responses (for our tasks) from 70B models in just a few minutes. All training in this paper is done using eight nodes, each containing eight A100 GPUs (briefly mentioned in Section 3); each 10 steps take around 2 minutes; training 5000 steps (plus validation and checkpoint saving) takes less than a day.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQGHt2hJSU", "reviewer": "Reviewer_E9gS", "summary": "The authors propose iterative RPO method for reasoning tasks. In particular, iteratively, the model at hand will be prompted to generate many CoT reasoning, and the ones that align with the true answers will be used as chosen, the other generated samples as rejected, for a DPO+NLL loss. The authors conduct experiments on GSM8K, ARC, and MATH to demonstrate the superiority of the proposed algorithm.", "questions": "1. My main concern is where the improvement comes from. In iterative RPO, there are (at least) three sources of improvements (a) more data, and (b) the model is updated iteratively, and (c) preference optimization is used in addition to SFT.\n    -  More data. While in table 1, iterative RPO with twice as much data is compared to STaR. It seems that the data is simply duplicated, but what if we use extra data generated by the model? Currently we are simply training more epochs.\n    -  What if we take the positive and negative samples generated by iterative RPO, and used it to train DPO or SFT? This tells us how much (b) or (c) helps, respectively.\n    - Currently, is the DPO baseline trained on just (x, y) or (c, x, y)?\n\n2. A less important question is the importance of NLL. My intuition on using NLL is that it regularizes the model from hacking the reward too much, but conceptually, choosing a larger $\\beta$ does the same thing -- it enforces a larger KL regularization between the SFT model and the trained model. If you vary $\\beta$, will figure 2 still hold?", "limitations": "Limitations have been discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The reasoning task at hand is important to the community, and the proposed algorithm shows promising improvements.", "weaknesses": "The novelty between the proposed algorithm and self-rewarding language models seems to be marginal. It also occurs to me that more ablation studies should be conducted, see **questions** for detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and insights.\n\nRe: novelty. \n- Our method is not self-rewarding because the reward signals come from the ground-truth labels instead of the model itself. More specifically, self-rewarding LM requires generating prompts, and using LMs to evaluate the sampled generations. But LMs are horrible at evaluating reasoning-related generations (e.g., according to RewardBench – although there has been some progress in the past few weeks). So we believe that self-rewarding LMs cannot be successfully directly applied using llama-2-70b-chat, but using IRPO in an unsupervised fashion is a really promising research avenue!\n- We are proposing a loss that integrates NLL while the self-rewarding LM uses standard DPO training. Through our experiments (in this IRPO submission), we demonstrate the importance of the NLL term on positive examples. \n- The focus of self-rewarding was general instruction tuning and our paper focuses on reasoning, math, and science reasoning in particular. In fact, the self-rewarding did not bring improvement in the math category.\n\nQ1:\n- IMPORTANT CLARIFICATION on “more data”: The “twice as much data” baseline means that we actually *generated* twice as much data – there is no duplication. We apologize for the confusion and will emphasize this point in the paper.\n- On the second bullet point: “Using the positive & negative samples for DPO” is actually our DPO baselines in the tables. “Using positive samples for SFT” is actually the STaR baseline in the tables. \n- On the third bullet point: (c, x, y). The DPO baseline is essentially RPO but without the NLL term. \n\nQ2: Yes. We experimented with different beta’s (0.05, 0.1, 0.5, 1.0) and the probability trends are similar within each dataset. We are investigating why sometimes naive DPO leads to decreasing chosen AND rejected probabilities like in Figure 2. Our current hypothesis is that it’s related to the fact that both current and rejected generations are sampled from the most recent model – both generations have high probabilities under the most recent model distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8vjUbN5Tez", "reviewer": "Reviewer_DMPg", "summary": "This work proposes an iterative training algorithm that enhances a model's Chain-of-Thought (COT) capabilities in reasoning tasks by combining self-improvement with preference optimization. The algorithm employs ground truth labels as supervision signals to evaluate model-generated responses, which are then incorporated into subsequent training iterations. Experiments on the GSM8K, MATH, and ARC benchmarks show significant improvements compared to the baseline, with continuous enhancement observed over multiple iterations.", "questions": "Please refer to the weaknesses section.", "limitations": "--", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces preference optimization into the self-improvement of reasoning tasks, yielding excellent results.", "weaknesses": "1. In the experiments, this work conducts experiments on the training sets of the GSM8K, MATH, and ARC benchmarks, and evaluates models on the corresponding test sets. All tests were completed on held-in data, with no performance results provided for the model on held-out tasks.\n2. This paper emphasizes the importance of preference optimization, distinguishing it from other iterative training methods like Rest-EM [1], which should be considered as a baseline.\n\n\n[1] Singh, Avi, et al. \"Beyond human data: Scaling self-training for problem-solving with language models.\" arXiv preprint arXiv:2312.06585 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments.\n\n\nWeakness 1: Correct; we trained using IRPO by leveraging GSM8K/MATH/ARC training set, and we tested on the respective test sets, but not other datasets. This paper doesn’t focus on generalization to other datasets, but the reviewer raises a valid point. We believe that generalizing to other datasets might need a larger number of prompts (e.g., by leveraging a dozen different datasets for IRPO training) and it would be an effort to find good-quality test data as well. \n\n\nWeakness 2: We were not aware of this paper at the time of writing, and we will cite this paper. But as explained below, in fact, this algorithm is already considered in our baseline. \n\nThis paper is similar to STaR (Zelikman et al.) from 2022. Our submission includes a variant of STaR baseline that uses temperature sampling without rationalization (i.e., without providing the correct answer first before generating the problem). In Rest-EM, in the binary reward case, when the solution has the wrong final answer, r=0; in this case we do not take any gradient. Therefore, the Rest-EM’s approach is essentially our variant of STaR. We’ll clarify this piece of related literature and our baselines in our revision, and thank you for suggesting!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a method that iteratively preference optimization based on the correctness of the generated answers. In other words, they optimize the preference between competing generated CoT candidates and this progressively contrastive appraoch leads to rapid gains when the model is already well-trained. The authors adopt a modified DPO objective with a NLL term. It lead to significant performance improvements while not using external data, boosting the GSM8k result from 55.6 to 88.7. Some reviewers had concerns about the sources of improvements and whether it comes from using more generated data. The concerns were addressed by the authors.
Decision: Accept (poster)