Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision
Authors: Yaowen Ye
Cassidy Laidlaw
Jacob Steinhardt
Venue: ICLR 2025 Spotlight (2025)
Keywords: unreliable human supervision
language model post-training
scalable oversight
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We find SFT+DPO breaks down under unreliable supervision and show that it is better to direct unreliable feedback towards improving the training *data* rather than continually training the *model*.

Abstract: Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference comparisons for reinforcement learning from human feedback (RLHF). As LMs become more capable, the tasks they are given become harder to supervise. Will post-training remain effective under unreliable supervision? To test this, we simulate unreliable demonstrations and comparison feedback using small LMs and time-constrained humans. We find that in the presence of unreliable supervision, SFT still retains some effectiveness, but DPO (a common RLHF algorithm) fails to improve the model beyond SFT. To address this, we propose *iterative label refinement* (ILR) as an alternative to RLHF. ILR improves the SFT data by using comparison feedback to decide whether human demonstrations should be replaced by model-generated alternatives, then retrains the model via SFT on the updated data. SFT+ILR outperforms SFT+DPO on several tasks with unreliable supervision (math, coding, and safe instruction-following). Our findings suggest that as LMs are used for complex tasks where human supervision is unreliable, RLHF may no longer be the best use of human comparison feedback; instead, it is better to direct feedback towards improving the training *data* rather than continually training the *model*. Our code and data are available at https://github.com/helloelwin/iterative-label-refinement.

Published as a conference paper at ICLR 2025
ITERATIVE LABEL REFINEMENT MATTERS MORE THAN PREFERENCE OPTIMIZATION UNDER WEAK SUPERVISION
Yaowen Ye"
Cassidy Laidlaw +
Jacob Steinhardt
The University of Hong Kong el alwin@connect.. hhu. hk
University of California, Berkeley
@lassidy_laidllae jsteinhardt) (be @barkeley edu
ABSTRACT
Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference com- parisons for reinforcement learning from human feedback (RLHF). As LMs be- come more capable, the tasks they given become harder supervise. Will post-training remain effective under unreliable supervision? To test this, we sim- ulate unreliable demonstrations and comparison feedback using small LMs and time-constrained humans. We find that in the presence of unreliable supervision, SFT still retains some effectiveness, but DPO (a common RLHF algorithm) fails to improve the model beyond SFT. To address this, we propose iterative label refinement (ILR) as an alternative to RLHF. ILR improves the SFT data by us- ing comparison feedback to decide whether human demonstrations should be replaced by model-generated alternatives, then retrains the model via SFT on the updated data. SFT+ILR outperforms SFT+DPO on several tasks with unre- liable supervision (math, coding, and safe instruction-folloving) Our findings suggest that as LMs are used for complex tasks where human supervision is un- reliable, RLHF may no longer be the best use of human comparison feedback; instead, it is better to direct feedback towards improving the training data rather than continually training the model. Our code and data are available at https: / //gi thub. com/he loell weeewinniii wwnn/teeaae ve- label. 1 re nement.
INTRODUCTION
Language models (LMs) learn rich knowledge when pretrained on internet-scale corpora (Achiam al., 2023; Dubey et al., 2024). To elicit their full capabilities and align them human values, they are typically post-trained with two types of human supervision: task demonstrations for the initial supervised finetuning (SFT) stage, followed by preference comparisons used in reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022) via algorithms like proximal policy optimization (PPO) (Schulman et al., 2017) or direct preference optimization (DPO) (Rafailov et al., 2024; Dubey et al., 2024).
As LMs continue advance, they are trained to solve complex tasks that are difficult for humans to supervise (Amodei et al., 2016; Leike & Sutskever, 2023; Wen et al., 2024). For example, humans are often imperfect at coding, producing bugs that models like Github Copilot learn to imitate (Asare et al., 2023). Similarly, industrial-level ChatGPT training data rated as "flawless" has recently been found to contain flaws (McAleese et al., 2024). As task complexity increases, human supervision may become even less reliable. Thus, is imperative to understand how both stages of current post-training pipelines (SFT+ + RLHF) perform under unreliable supervision.
Studying this is difficult because tasks where human supervision is unreliable are by nature difficult to obtain ground truth for. Ideally, we want tasks with known ground truth where we can also collect human-like unreliable data. We employ two approaches simulate this unreliable supervision. First, we use smaller LMs that often make mistakes, in line with Burns et al. (2023) and Dubois et al. (2024). Second, we recruit human workers to label data under time constraints. For either of these two types, we can use it to simulate either unreliable demonstrations or unreliable comparisons (e.g. for
"qqual contribution

---

Published as a conference paper at ICLR 2025
Typical Post-Training Pipeline
relablle SFT dora relitle feedbass
uncelabll SFT deth relabile feechark
ureelaable
replace
unoellatiie fredhark
model neiporos
A-
31%
SFT
SFT
sorted respormoes
model meggonoe
22%
Iterative Label Refinement
33%
model peperee
A
381 data
SFT
refined
2%
data
realace
Rounds baaning
(a)
(b)
Figure I: In contrast RLHE, which iteratively updates the SFT model, IIR directs comparison feedback towards improving the SFT dataset. (b) With reliable supervision, DPO effectively improves the SFT model and ILR is unnecessary. However, DPO becomes less effective with unreliable demonstrations and breaks down completely when comparison feedback is also unreliable. In contrast, ILR consistently improves on the initial SFT model with unreliable supervision, in the most extreme
DPO). We thus simulate post-training by running SFT on unreliable demonstrations followed by DPO on unreliable comparisons.
We first find that SFT with unreliable demonstrations performs adequately, in that the learned SFT model more reliable than the demonstrations themselves, phenomenon termed weak-to-strong generalization (W2SG) that has been previously observed by Burns et al. (2023) in classification tasks. However, since these models inevitably learn to imitate errors in their unreliable training data, the resulting performance is still far from their full capability when trained on ground truth. Thus, ideally, the subsequent DPO stage should improve these SFT models.
Unfortunately, we find DPO breaks down under unreliable supervision. With unreliable demonstra- tions and comparisons, DPO offers little to no performance improvement on top of SFT (Figure lb). Our experiments suggest that this may be due to the tendency of DPO overoptimize, similar to what is observed in Gao et al. (2023). To prevent overoptinization, DPO implicitly penalizes KL divergence from the SFT model, and we find that a large penalty is necessary when feedback is unreliable. However, this large penalty prevents the large changes in the model that are necessary to correct the errors learned from the unreliable demonstrations during SFT (Section 4.1).
To address this, we propose iterative label refinement (ILR). In contrast to RLHF, which iteratively updates the SFT model, ILR updates the SFT dataset using the same type of comparison feedback. Specifically, ILR uses this feedback to compare demonstration in the SFT dataset with an alternative one written by the SFT model; if the model-written demonstration chosen, replaces the original demonstration in the SFT data. This improves the SFT data by replacing low-quality incorrect demonstrations with better model-written ones. Thus, when a new SFT model is trained on the updated dataset, it should perform better than the original SFT model, facilitating further improvements the dataset via additional rounds of ILR (Figure la). Unlike DPO, we find that updating the SFT data via ILR leads to significant improvements from one round to the next (Figure 1b).
ILR improves over SFT+DPO on several tasks (math, coding, and safe instruction following) in LM-based simulation. These results are further confirmed by our human study, where recruit workers to provide demonstrations and comparison feedback under time constraints for the A Alpaca (Taori et al., 2023) instruction-followin task. In this setting with more realistic human errors, ILR continues to improve the initial unreliable human demonstrations and outperforms DPO.
In summary, our findings suggest that preference-based post-training like RLHF may no longer be the best use of comparison feedback as we train LMs to solve complex tasks where human supervision unreliable. Instead, better to direct human feedback towards fixing errors in the unreliable training data rather than continually training the model.
2 RELATED WORK
Scalable oversight and weak-to-strong generalization. Scalable oversight aims to develop methods to supervise AI systems tasks that are difficult for humans (Amodei et al., 2016; Bowman et al.,
2

---

Published as a conference paper at ICLR 2025
2022). The primary focus in scalable oversight has been designing human-AI collaboration mechanisms that enhance humans' ability to evaluate Al outputs more accurately (Christiano et al., 2018; Irving al., 2018; Michael et al., 2023; Khan et al., 2024; Kenton et al., 2024) and with less cognitive burden (Saunders et al., 2022; McAleese et al., 2024; Kirchner et al., 2024). In contrast, recent work W2SG (Burns et al., 2023) explores a complementary direction that develops learning algorithms to make models generalize correctly from weak supervision as if they were trained on higher-quality supervision or even ground truth. The SFT stage our LM-based simulation employs a similar setting as the setup used for studying W2SG in Burns et al. (2023) but extended to text generation tasks; Burns et al. (2023) only consider classification tasks.
Language model post-training. Pretrained language models (PLMs) possess a rich understanding of language but require post-training to align their behavior with human preferences. This gener- ally involves supervised finetuning (SFT) (Wei et al., 2021) 1) on human- written demonstrations and reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Bai et al., 2022). RLHF is typically done by algorithms like PPO (Schulman et al., 2017), which uses an explicit reward model, or DPO (Rafailov et al., 2024; Dubey et al., 2024) and its variants (Liu et al., 2023; Azar et al., 2024), which uses an implicit reward model. Recent work has also investigated the impact of noisy preference data on RLHF (Chowdhury et al., 2024; Fisch et al., 2024; Gao et al., 2024), while our work addresses more challenging case where both SFT data and preference data are unreliable.
Learning with imperfect supervision. Research in weakly-supervised learning has focused on training models with noisy data (Reed et al., 2014; Rolnick al., 2017; Song et al., 2022). Many works study classification with random label flips, proposing methods like data filtering and robust losses (Zhou, 2018; Frénay & Verleysen, 2013), while some also consider noise in structured prediction tasks like parsing and word alignment (Ganchev, 2010). Our setting differs as we study unreliable supervision from LMs or time-constrained humans for complex lext-generation tasks. In cetpeaaaaaiiriiiiiiiig from reasoning failures or social biases rrateer than
this setting, errors tend to be random, and so may not be well-addressed by traditional methods. Our cross-labeling framework shares conceptual similarities with co-training (Blum & Mitchell, 1998; Zhou et al., 2005) and co-teaching (Han et al., 2018), although these methods primarily focus on classífication settings and rely on model confidence scores.
PROBLEM DEFINITION: POST-TRAINING WITH UNRELIABLE SUPERVISION
Training LMs to be useful assistants involves two stages: pre-training and post-training, During pre-training, the LM is trained to autoregressively predict a large corpus of text, typically taken from the internet and books (Radford al., 2019: Brown al., 2020). Afterwards, the pretrained language model (PLM) has acquired knowledge and skills from its pre-training data, but cannot yet be easily used for realistic applications. To elicit the PLM's capabilities for some class of tasks, post-training techniques are used (Wei et al., 2021; Dubey et al., 2024; Ouyang et al., 2022; Bai et al., 2022). We denote an LM as a conditional distribution p x) over responses y given a prompt x.
Standard post-training pipeline. Post-training PLM typically involves two stages. In the initial SET stage. a PLM is trained on human task demonstrations. That is, given an SFT dataset DEFT (ASrT, DSFT) of prompts XSFT and human--ritted responses Verr, the PLM is trained to minimize [Pseil (( llogp(y p(y z). We call the result of this stage the SFT model, which has typically learned the correct input/output format for the given task and has partially elicited the PLM's capabilities.
Subsequently, to further improve performance, the SFT model undergoes RLHF via algorithms like PPO (Ouyang et al., 2022; Bai al., 2022: Schulman al., 2017), DPO (Rafailov et al., 2024; Dubey et al., 2024), or related methods. In this stage, new dataset is constructed using a set of prompts E XRLIF and response pairs y' sampled from the SFT model. Each prompt and pair of responses shown to human annotator that picks the response they prefer, creating dataset of triples (z, y-) E DRLHF, where y+ is preferred to y This dataset is used for preference optimization via methods like PPO, which trains reward model from DRLMI for online RL, or DPO, which uses an implicit reward model. In this work, we focus on DPO due to its computational efficiency and stability. We call the output of the this stage the SFT+DPO model.
Simulating unreliable human supervision To study how current post-training pipelines perform under unreliable supervision, we need tasks with known ground truth so that we can evaluate
3

---

Published as a conference paper at ICLR 2025
performance, along with realistic forms of unreliable supervision. We employ two approaches to simulate unreliable supervision: small language models (Section 4 and 5) and eme-ccontraiieed humans (Section 6). Most of our experiments focus on the small-LM case, in line with Burns et al. (2023) and Dubois et al. (2024), and we then verify the results on our instruction-folloiing dataset using time-constrained human annotators.
For both forms of supervision, we need to simulate both unreliable demonstrations and unreliable comparisons. For humans, we do this simply by querying them. For small LMs, we follow the procedure below (more details in Appendix B):
1. Unreliable task demonstrations: We finetune a small PLM p on ground-trute -ruth demonstrations and use it to generate responses for a held-out set of prompts as unreliable demonstrations.
2. Unreliable comparison feedback: We finetune classification model : to select the better response when given two responses y1, y2 and a prompt H, where q(y1 > 3/2 ()) denotes the probability that 31 is preferred to y2- Note that g is based the same PLM as p but with an additional classification head. Each element of ÿ's training data consists of prompt and two responses: one response is from the ground-truth demonstrations used train ý and the other is a lower-quality output generated by intermediate checkpoints of p. is trained to choose the ground truth using a standard binary classification loss.
SFT+DPO with unreliable supervision. We start by finetuning larger PLM on an unreliable SFT dataset SSFT = (X))) to obtain an initial SFT model SIT: where 5 is a set of unreliable demonstrations given by p or time-constrainet humans. We then perform K rounds of DPO training. In each round k, we: sample completions y1, 1/ from the current SFT+DPO model P SET-DPO for each prompt x X' (where P SITTDDO = ser)); collect unreliable comparisons using either á or time-constraiined humans to construct an unreliable preference dataset DRLIESS and, use this train k-1 SFT+DPO via DPO, resulting in p SPT+DPO The final model after K rounds is denoted as p SFT+DPO or simply P SFT+ Further implementation and hyperparameter details can be found in Appendix C.
K
3.1 TASKS AND MODELS
Datasets. We test SFT+DPO with unreliable feedback on three text generation tasks: mathematical problem-solving using GSMSK (Cobbe et al., 2021), SQL code generation with BIRD (Li et al., 2024), and safe instruction following with SaferPaca (Bianchi et al., 2023), which is a mix of the Alpaca dataset (Taori et al., 2023) and refusal demonstrations to unsafe instructions. All datasets are formatted as question-anseee answer pairs following the same template.
Models. We use three open-source PLMs of varying sizes in our experiments: Gemma 2B (Team et al., 2024), Mistral 7B (Jiang al., 2023), and Meta Llama 3 70B (Dubey al., 2024). Our LM-simulated experiments include four settings that use a smaller model to supervise a larger model. In our simulation with time-constrained humans, we only experiment with the largest 70B model.
Evaluation metrics. Each dataset requires a specific approach to evaluate model outputs and deter- mine performance. For GSM8K, we parse the numerical answer following "# # # #" the end of each response and compute exact match accuracy by comparing it with the ground truth. For BIRD, we measure execution accuracy by running the generated code corresponding test databases, following Li et al. (2024). For SaferPaca, we follow Li et al. (2023) and use GPT-4o (OpenAI, 2024) to compute win rate against reference answers.
Further implementation details on datasets, evaluation metrics, model architectures, training and inference configurations are provided in Appendix A, B, and C.
DPO IS INEFFECTIVE WITH UNRELIABLE SUPERVISION
In this section, we present results of SFT+DPO under LM-simulated unreliable supervision. We find that SFT still retains some effectiveness, but DPO fails to improve the SFT model. We further show that the failure of DPO appears be caused by its tendency to overoptimize given unreliable comparison feedback, which motivates our alternative approach introduced in the next section.
SFT shows limited robustness to unreliable demonstrations. We first investigate the effectiveness of SFT under LM-simulated unreliable supervision. When finetuned D with unreliable demon-
4

---

Published as a conference paper at ICLR 2025
Superdior (8)
SFT Unnelistie Demorndrations l6srl
Gemund- Truth Demonstrations (Par)
6S48K (28- 7B)
GSMSK (78 - 7081
BIRD (26 708)
SaferPace (28 78)
0.60
0.50
0.60
0.80
0.55
0.55
0.55
0.75
0.40
0.70
0.40
0.60
0.35
0.65
0.35
0.55
0.20
0.20
Figure 2: Models finctuned on unreliable demonstrations generated by a smaller supervisor model show higher accuracy than the supervisor, demonstrating SFT's robustness unreliable supervision. However, models' full capability is only partially recovered with unreliable demonstrations. A - denotes using demonstrations generated by the model with size A to finetune the model with size B.
KL Regulariztionn (ESMeK
$-0.00
20 TrAToo 0122 Shurr
7.5%
5.0%
1.0%
2+3
ILS eack compatisats
DPO w ande compersons (p=
0.0%
DPO ande comparsons ip- 0.3)
-23%
8-03
DPO oace comparsons (p- 0.03)
-.00
ILA und ub compar sers
-5.0%
-.0%
DPO urre vrrelaale can cumpa bons 0.3)
DPO w urre compa bons (5-
9-0.0
DPO unrelazle cam pa bons (5- 0.03)
-7.5%
1211
20
10
-3.0%
0.0
0.5
1.0
2.0
[Coeffccant Regularzation)
verpence homm
(a) DPO effectively improves SFT models with ora- cle comparisons, especially with weak regularization. However, with unreliable feedback, improvements are limited narrow range regularization strengths because weak regularization causes overoptimization.
(b) Strong regularization in DPO limits useful model updates, while small regularization leads to overop- timization of unreliable feedhack. In contrast, IIR facilitates large model updates, allowing faster im- provement and efficient use of comparison feedback.
DPO struggles avoid overoptimization of noisy preferences while also updating the initial suboptimal SFT model sufficiently to improve performance.
Figure 3:
strations generated by D. the larger model SET consistently outperforms ý across all tasks, showing SFT's robustness imperfect demonstrations (Figure 2).
However, models finetuned on unreliable demonstrations inevitably learn to imitate errors them. When applying SFT to the same PLM using ground-truth demonstrations, which denote PSFT, find that it significantly outperforms P SFT (Figure 2). This shows that SFT on unreliable demonstra- tions only partially recovers model's full capability, leaving large performance gap between Pser and SPT- Thus, ideally, the subsequent DPO stage should further improve upon SFT--
DPO does not improve (Rafailov et al., 2024; Dubey et al., 2024), we find that it consistently fails to improve performance SFT when both demonstrations and comparison feedback unreliable (Figure 4). That is, when we further finetune P SPT via multiple rounds of DPO on comparison feedback given by the resultant model P SET+DPO shows little no performance gain (or performance even declines).
SFT with unreliable comparisons. Contrary to DPO's past success
To investigate the degree to which this is due to the unreliable comparison feedback versus the unreliable SFT data, we run DPO in other settings on the GSM8K task. In the first, initialize training with P SPT but use an oracle evaluator for preference comparisons that always selects the better answer by comparing model outputs ground truth. In the second, we again use the oracle evaluator, but start from an SFT model trained on a filtered subset D that only contains correct demonstrations. As shown in Figure lb, DPO with the oracle evaluator does manage improve over the starting SFT model in both cases. These results reveal that DPO's effectiveness heavily relies on high-quality supervision.
5

---

Published as a conference paper at ICLR 2025
G7B 708)
+708)
SaterPaca [28
0.44
0.885
SFT+0R0
0.65
0.300
0.71
0.65
2.372
3.41
5.770-
0.69
0.59
0.02
0.500
0.67
0.63
0.60
0.350
0.36
Bound
Bound
Round
found
Figure 4: ILR consistently provides more improvements for the SFT model than DPO in four settings with LM-simulated unreliable demonstrations and comparison feedback. Round represents accuracy of SPT-
4.1 DPO WITH UNRELIABLE FEEDBACK EXHIBITS OVEROPTLMIZATIO
We hypothesize that DPO's failure under unreliable supervision stems from the problem of overopti- mization that preference-bar based learning methods often suffer from (Gao al., 2023). Optimizing an explicit reward function (with PPO) or an implicit one (with DPO) based on preference feedback can initially lead to an increase in performance but eventually causes decline the reward function diverges from the true objective. To combat this, both PPO and DPO regularize based on the KL divergence between the initial and updated policies.
We find that the optimal amount of regularization for DPO, which is controlled by the hyperparameter B, depends strongly on the reliability of comparison feedback. This dependency creates critical trade-off: stronger regularization is needed to prevent overoptimization with unreliable feedback, but this constrains DPO's s ability to improve upon the suboptimal SFT model. To demonstrate this, we measure DPO's s relative improvement over SFT under varying feedback quality levels and KL regularization strengths (Figure 3). Full results for each training epoch are presented in Figure
With unreliable comparison feedback, DPO is especially prone to overoptimization and needs very strong regularization. In Figure 3a, we consider three levels of feedback quality: unreliable feedback from q, mixed feedback (50% q. 50% oracle), and pure oracle feedback. With oracle feedback, smaller 3 values consistently yield better performance. However, as feedback quality decreases, smaller values lead worse performance while only suitably large 3 enables positive improvements. This suggests that preference optimization under unreliable feedback needs heavy regularization prevent overoptimization.
Strong regularization limits useful model updates during DPO training. Figure 3b demonstrates that larger divergence correlates with higher accuracy increases the case oracle feedback (colored circles in upper left). This indicates that setting small 3 to allow substantial model updates is crucial for improving upon the initial suboptimal SFT model. However, with unreliable feedback, this not possible: as we observed above, using insufficient regularization with unreliable comparisons leads to overoptimization.
These two observations suggest a tension between preventing overoptimization from unreliable com- parison feedback and making large enough model updates. Since SFT on unreliable demonstrations produces a model P SET that is suboptimal, we need large updates to P SFT to recover performance close to PSFT. However, with unreliable comparison feedback, we need significant regularization (i.e., large 3) for DPO, which prevents the large updates that are needed. Given these issues, we seek an alternative algorithm that better leverages comparison feedback to improve over the SFT model.
ITERATIVE LABEL REFINEMENT
To overcome the limitations of preference optimization discussed in Section 4. our method needs to be robust to unreliable comparisons while still allowing large updates from the SFT model. To achieve this, we propose iterative label refinement (ILR). In contrast to RLHF methods, which iteratively update the SFT model. ILR focuses improving the unreliable demonstrations in the initial SFT dataset. It uses comparison feedback to decide whether the initial SFT data should be replaced by model-generated alternatives, and then retrains the model from scratch via SFT on the

---

Published as a conference paper at ICLR 2025
new data (Figure la). As we will show, ILR avoids the regularization dilemma discussed in Section 4.1 that impedes DPO with unreliable feedback. In ILR, retraining the model from scratch at each iteration allows for large changes to the SFT model but avoids overoptimization. We introduce the detailed methodology of ILR in Section 5.1, and demonstrate that outperforms DPO both in the LM-simulated setting (Section 5.2) and with time-constrained humans (Section 6).
5.1 METHODOLOGY
ILR consists of several iterations; during each iteration, the SFT dataset is improved by replacing some of the demonstrations. The first step of each iteration is to gather proposed demonstrations that may replace unreliable demonstrations in the current SFT data. These proposals are generated using models trained via SFT the current dataset. We showed in Section 4 that SFT models often outperform their unreliable training data. Thus, by replacing some of the existing dataset with the improved output of the SFT model, the overall quality and accuracy of the dataset should increase.
However, our findings in Section 4 only show that SFT models outperform their training data on held-out prompts; if an SFT model is tested contain mistakes imitated or memorized from the training data. Thus, if we use a model trained on the entire current SFT dataset generate proposals, those proposed responses are less likely to improve over the current responses in the dataset. To ensure that the proposed replacements are generated on held-out prompts, we implement ILR by training two SFT models on different halves of the SFT data; then, these models cross-label the half they were not trained on. This makes it more likely the gener--geeertte responses are different and better than the existing responses, enabling improvement of the dataset.
train prompt, it is likely output responses that
Once proposal responses are generated, we selectively update the SFT dataset with them by leveraging comparison feedback. We ask the annotator (i.e., the small LM time-constrained humans) to compare an existing response in DSFT with model--gnerated proposal for the same prompt. If the proposal is preferred by the annotator, it replaces the original response. After these comparisons are complete, new SFT model is trained on the updated dataset. As long as the annotator chooses better responses more than half the time, the overall accuracy of the SFT data will increase, enabling the new SFT model to outperform the initial one.
Formally, each iteration of ILR consists of the following steps:
Data splitting: We start with a dataset of task demonstrations Dj = (2,)))), where is the current iteration and Do DSTT. The dataset Dj is evenly split into two disjoint subsets, D = (x1, (x' ,3%) and D = (z)
2. Model training: Two models, p SPT and PST.. are finetuned on Dj and Dj respectively.
3. Cross-labeling: Each model generates label proposals for the subset it wasn't trained on, i.e., we sample 27 ( x²) and 2 P X)).
Proposal evaluation: The unreliable supervisor provides comparison feedback on the new proposal 2, and original label Y, for prompt æ; to decide which of them preferred; this feedback determines whether Ys should be replaced by 2,.
5. Label updating: Based on the comparisons in step 4, we form an updated dataset D = (x1)).)) where each element of V4+1 is either the accepted new proposal the retained original label. We control the label refinement speed by setting a hyperparameter a E (0, 1]. allowing most a X labels to be updated in each iteration. When more than a| proposals are accepted in step 4, we choose the ones with higher evaluation confidence, determined by either q's log probability human annotators' self-reported confidence. In all experiments set = 0.15. Further analysis of a is presented in Appendix E.3.
At the end of each iteration, we finetune new model starting from the base PLM of γ SFT using the entire refined dataset DJ. resulting in the SFT+ILR model 5 P This process is repeated for K iterations and the final model is SFT+ILR
In the LM-simulated setting, we only consider proposals that are sufficiently different from the original labels during step 4. We determine proposal is different enough if it has a different final
7

---

Published as a conference paper at ICLR 2025
GSMBK (26-)) -
GSMBK 781 w/ Oracle Feedback
0.65
Noroo!
Improvements
0.15
0.65
Truth SPT Model Have?
Inprovetuents
0.60
3P1 Labels
0.50
SPT Labels
2.10
0.55
0.55
0.3
0.05
0.50
0.45
0.40
0.40
0.35
0.35
0.0
Round
Round
(a)
Figure 5: SFT label accuracy increases across rounds during ILR. With high-quality feedback, ILR can lead to labels that approach even surpass the accuracy of the model trained ground truth (psEr).
answer (GSM8K), different execution result (BIRD), large embedding distance (SaferPaca). We present the pseudo-code (Algorithm 1) and more implementation details of ILR in Appendix C.
5.2 LM-SIMULATED EXPERIMENT RESULTS
SFT+ILR leverages unreliable supervision better than SFT+DPO. We compare SFT+ILR and SFT+DPO across four settings with LM-simulated unreliable demonstrations and comparison feed- back (further validation using Lime-constraines human data is presented in Section 6). As shown in Figure 4, SFT+ILR consistently outperforms SFT+DPO all scenarios under unreliable supervision. In GSM8K, ILR shows more significant improvements as model size increases from 7B to 70B, suggesting that it benefits from model scaling and may remain effective for future models that are even more capable.
Comparison feedback is effective for improving demonstration quality. ILR relies on refining SFT data using comparison feedback, which is often easier to obtain than high-quality demonstrations, especially for complex tasks that humans struggle with. Figure 5a shows that unreliable comparison feedback guides this refinement process effectively: the accuracy of the SFT data steadily increases with more rounds of IIR. As suggested, this is likely because evaluation of Al output is typically easier than the demonstration an ideal output (Leike et al., 2018), allowing even imperfect feedback to contribute to meaningful improvements in the SFT data. With reliable feedback (Figure 5b), the SFT demonstrations can be improved even further, approaching or even surpassing the accuracy of the model P SFT trained ground truth. This highlights ILR's 's potential when combined with other scalable oversight techniques that enhance humans' evaluation capability.
ILR enables larger model updates and more efficient of comparison feedback. As illustrated in Figure 3b, models trained with ILR exhibit significantly higher KL divergence from the initial SFT model compared those trained with DPO
after a single round. Moreover, when comparison feedback is reliable while SFT data not, ILR shows much higher efficiency than DPO, using the same amount of feedback enable larger improvement over the initial SFT model. These can be attributed to ILR's direct modification of the unreliable SFT data, which fundamentally alters models' training dynamics during SFT. By doing so, ILR allows each subsequent model learn from new data independently and from scratch, without inheriting the errors of previous models, unlike the continual preference-baste based training seen DPO.
Supervision for refinement is necessary. Since the SFT models can generate proposal responses that are more accurate than their training data, appears that one can simply replace the initial SFT data with new proposals without using any comparison feedback. To understand the importance of comparison feedback ILR's refinement process, we compare it to a naive approach that directly replaces fraction of the original labels with new proposals in each round. As shown in Figure (Appendix E.2), this naive method leads to performance degradation, showing that supervision the refinement process,
if unreliable, is necessary. It could be that training model outputs without any curation leads to model collapse, a phenomenon observed when training generative models with synthetic data (Shumailov et al., 2023; Ren et al., 2024; Gerstgrasser et al., 2024).
8

---

Published as conference paper at ICLR 2025
Human
Comparing
100
CPI Aryne's
10
Human Answers Mean
Median
0.8
Accurozy
0.45
0.44
0.2
STT+
SFi+DDO (g=0
0.0
Round
Tound2
0.42
SFT+DRD (g-3.)
STT+DRD (5-0.5)
SIT+Doo (p-
Worl Count
(c)
Figure 6: (a) Distribution of word of human written demonstrations compared to GPT4-generae answers collected by (Peng ct al., 2023). (b) In both ILR rounds, participants are at least modcrately confident choosing newly proposed labels over original human demonstrations 20% of the time, but they tend to update less in later rounds. Participants also show higher overall confidence when choosing human demonstrations. These suggest that W2SG is effective enough consistently produce better outputs that AI oulputs becoming complex cime-conttrained participants to evaluate. (In second round, we only count preference pairs with (c) ILR enables more improvements on top of
least one human-written label when calculating the frequencies) the initial SFT model, while DPO plateaus similar to what is observed in LM A--simulated settings.
Question
Human Demonstration
Chosen AI output
Below an instruction that describes a task... Instruction: Explain the concept of random for-
A random forest is forest, usually in nature, that not accounted for with maps or other docu- mentation.
A random forest a ma- chine learning method for predication clas- sification using multiple decision trees...
Figure Example lable refinement ILR: When instructed explain "random forest," an inaccurate human demonstration misinterpreted it as a natural forest. Human workers then chose to replace this with a more accurate moodel-generatee explanation, improving the SFT datasct quality.
TIME-CONSTRAINED HUMAN STUDY
To validate our findings from LM-simulated settings, conduct human study where we recruit workers through CloudResearch Connect' to provide both task demonstrations and comparison feedback for impose time constraints to create annotator noise and obtain unreliable data; this simulates the challenges that may arise humans supervise Al on complex tasks that are difficult for humans themselves.
wowtiuitiioooooooiinn task. Although the task is not beyond human capability,
Task and data collection. For simplicity, focus on the Alpaca instruction-following dataset (Taori et al., 2023; Peng et al., 2023) without unsafe instructions. We evaluate models using the complete test of AlpacaEvval (Li et al., 2023) and GPT-4o judge, similar evaluation in our SaferPaca setting. We collect human demonstrations for 1,000 randomly sampled instructions, with workers instructed spend 1-2 minutes write each response. Each worker assigned 15 questions, plus 3 attention checks used to filter out low-quality responses. As shown in Figure 6a, human demonstrations are generally shorter than GPT-4-generated responses, suggesting suboptimal quality due to time constraints. For the comparison feedback used in DPO and ILR, instruct workers to spend 30 seconds to minute per question, with 50 questions assigned per worker and additional questions for quality checks. We collect 1,000 comparisons for each algorithm in each round and conduct two rounds of both DPO (3 0.1) and ILR (a = 0.1). We also use the comparison data collected for DPO with B 0.1 run DPO with several other 33 values. More details of our human study setup are provided in Appendix D.
httt:://commee..cuuddrcccc..cm

---

Published as a conference paper at ICLR 2025
ILR outperforms DPO with time-constrained human supervision. With time-constrained unreli- able human supervision, ILR demonstrates fast and consistent improvement compared to DPO (Figure 6c). While DPO with strong regularization (large 3) improves slowly, DPO with less regularization (small (3) improves quickly in the beginning but then plateaus in performance. We hypothesize that DPO may be overoptimizing with insufficient regularization, similarly our LM-simulated experi- ments. Notably, the results of ILR and DPO with the collected con-sconsra human data are most similar to LM-simulated scenarios with unreliable demonstrations but reliable comparison feedback, as seen in Figure 3. This suggests that comparing Alpaca outputs remains relatively straightforward for humans even under time pressure. Future work could explore more complex tasks or even shorter time constraints to better simulate more unreliable human feedback.
ILR effectively improves SFT data because models finetuned on unreliable demonstrations can surpass human supervisors. We find humans are at least moderately confident in choosing model-generated outputs over original human demonstrations for more than 20% of the questions in both rounds of ILR (Figure 6b). This further confirms the W2SG phenomenon, where models trained on unreliable supervision can surpass their supervisor. For example, Figure 7 demonstrates that a participant chose to replace human-writter misinterpretation of "random forest" with more correct and detailed model-generater explanation.
However, the effect of ILR may decrease over rounds because participants tend to select fewer Al-generated responses in later rounds (Figure 6b), which may result in fewer opportunities for label refinement. This could be due to reasons. First, participants may become less confident in choosing AI outputs because in later rounds these outputs are more likely to surpass human level in complexity. Second, W2SG may not effective enough consistently produce strictly better Al outputs. To address these challenges, future work could explore better refinement mechanisms that combine human demonstrations with AI outputs, potentially allowing for more nuanced refinement in cases where both human and AI responses are partially correct in complementary ways.
7 DISCUSSION
In this work, we study the effectiveness of a typical SFT+DPO pipeline for language model post- training under unreliable supervision, as simulated by small LMs and time-limited humans. We show that in this setting, unlike with reliable feedback, SFT+DPO fails to improve upon SFT. Our analysis suggests that DPO struggles to avoid overoptimization with unreliable comparison feedback and requires heavy regularization, preventing it from correcting significant errors in the SFT model. To address this, propose ILR redirect comparison feedback towards improving unreliable demonstrations in the SFT dataset. ILR enables larger model updates without the overoptimization risks in preference optimization and consistently outperforms DPO under unreliable supervision.
Limitations. Our focus on RLHF via DPO may not fully capture the nuances of RLHF pipelines that use reward modeling and PPO. Future research should verify whether our findings generalize to these more complex RLHF setups. Also, our human study using time- cons-consta data collection may not perfectly simulate human errors that arise from more realistic capability constraints. Exploring more challenging tasks such as competition math or coding could potentially provide a closer analogy to supervising AI on tasks that humans truly struggle with.
Future work. Our findings open several exciting directions for future research. For example, one could explore hybrid approaches that combine ILR with RLHF. Applying PPO or DPO after sev- eral rounds of ILR could potentially yield more improvements, since the problem of imitating errors in unreliable demonstrations relieved after improving the SFT data with IIR Additionally, investigating more sophisticated label refinement strategies, such as synthesizing human demonstra- tions with model-generater proposals or implementing ensemble techniques through more than two cross-labeling splits, could potentially further enhance the effectiveness of ILR.
Broader impact. Our findings suggest a potential paradigm shift in how we approach human oversight of AI systems under unreliable supervision. We show that the canonical RLHF post-training pipeline may no longer be the best use of human comparison feedback. Instead, we need methods like ILR that leverage model outputs to improve and learn from unreliable human supervision.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Thanks to Alex Shypula, Eli Bronstein, Meena Jagadeesan, Ruiqi Zhong, Xinyan Hu, and members of JS's group for helpful feedback, and to Tianyi Qiu, Xiaoyuan Zhu, Yitao Liu, Meitong Liu, Han Li for valuable discussions. This work is supported by the Open Philanthropy Action Fund through funding provided to JS. CL is supported by an Open Philanthropy AI Fellowship. Additional thanks to Yuchen for her personal support and encouragement.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087: 087744, 2023.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565. 2016. 1,2
Owura Asare, Meiyappan Nagappan, and N Asokan. Is github's copilot as bad as humans at introducing vulnerabilities in code? Empirical Software Engineering. 28(6):129, 2023.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024. 3, 22
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862,, 2022. 1,3
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger. Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07875 2023. 4, 15
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the eleventh annual conference Computational learning theory, pp. 92-100, 1998. 3
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilé Lukošiūtē, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:221 1tfi:22211.005 1.03540, 2022. 2
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners Advances in neural information processing systems, 13:1877-1901, 2000 3
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strongg generalization: Bliciting strong capabilities with weak supervision. arXiv preprint 2rXiv:2312.093300 2023. 3, 4, 16
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.004099 2024. 3, 22
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv: 181000881 2018. 3
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2xxv:2110.14166 2021. 4, 15
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint axXiv:2407.21783 2024. 4,

---

Published as a conference paper at ICLR 2025
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024. 1,4 4
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405./93/6 2024. 3
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE fransactions on neural networks and learning systems, 25(5):845-865 2013.
Kuzman Ganchev. Posterior regularization for learning with side information and weak supervision. PhD) thesis, University of Pennsylvania, 2010.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp- 10835-10866 PMLR, 2023. 2, 6
Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. arXiv preprint arXiv:2404.09824 2024.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, al. Is model collapse in- evitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.011 01413, 2024. 8, 21
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018. 3
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.0668. 09685, 2021. 17
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00889 2018. 3
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023. 4
Zachary Kenton, Noah Y Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D Goodman, et al. On scalable oversight with weak Ilms judging strong 1lms. arXiv preprint arXiv:2407.04622 2024.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Ed- ward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive Ilms leads to more truthful answers. arXiv preprint
2024. 3
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 142.69900 2014. 17
Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of Ilm outputs. arXiv preprint arXiv:2407.13692 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposiun on Operating Systems Principles, 2023. 15
Jan Leike and Ilya Sutskever. Introducing superalignment, Jul 2023, URL https: openai.. com/bl roduci -up gnment..
12

---

Published as a conference paper at ICLR 2025
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07877 2018.
Jinyang Li. Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? a big bench for large-scale database grounded text-to-slls. Advances in Neural Information Processing Systems, 36, 2024. 15
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cneeructtoo-follwwiii models. https: /ttp:////tthhle www/tatsu-naaaaaeaaecc http//alacaa__eeee
2023. 4, 9, 15
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.066 069006677 2023. 3
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm crities help catch Ilm bugs. arXiv preprint arXiv:2407.002/55 2024. 1,
Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R Bowman. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.0870.. 2023. 3
Eric Mitchell. A note dpo with noisy preferences & relationship to ipo, 2023. 22
OpenAl. Hello gpt-4o, 2024. URL https: http://ppeeaaa..ccmmneeeeeeeeeee www.:/ddee//eeeeeecccce gpt 40/. Ac- cessed: 2024-09-22, 4, 15
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022. 1,3
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 2023. 9, 15
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 3
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 3,
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv: 142.6514 2014.
Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, and Danica J Sutherland. Language model evolution: An iterated learning perspective. arXiv preprint N.Ni:22404.02288 2024. 21
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv: 1705. 1705./06944 2017. 3
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiqinng models for assisting human evaluators. arXiv preprint arXiv:2206.05802. 2022. 3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017. 1,3
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604, PMLR, 2018. 17
13

---

Published as a conference paper at ICLR 2025
Ili Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17495, 2023. 8, 21
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey, IEEE transactions on neural networks and learning systems, (4(11):8135-881) 13533 2022. 3
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://rrfm. stanford. cdu/2023/03/13aappeeee html, 3(6)77, 2023. 2, 4, 9, 15, 18
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprini arXiv:2403.08225, 2024. 4
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. 3
Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Boman, He He, and Shi Feng. Language models learn mislead humans via rlhf, 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020. 18
Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National science review, 5(1):
44-53, 2018.
Zhi-Huz Zhou, Ming Li, et al. semi-supervised regression with co-training. In IJCAI, volume 5, pp. 908-913, 2005.
14

---

Published as a conference paper at ICLR 2025
A TASKS AND EVALUATION
A.1 DATASETS
GSM8K (Cobbe et al., 2021): This dataset contains math word problems widely used evaluate language models' mathematical problem solving capabilities. In total, there are 8.5K question-answwr pairs for training and 3K for testing.
BIRD (Li et al., 2024): In this task, models required to generate SQL query that answer question of interest, given context of database descriptions. We filter out questions with token lengths greater than 1024 and obtained 8.5K examples for training and 1.4K for testing.
SaferPaca (Bianchi et al., 2023): This dataset is a mixture of instruction following questions from (Taori et al., 2023) and demonstrations of refusals to unsafe instructions. requires a model serve as an assistant that follows human users' instructions while generating responsible responses and rejecting harmful prompts. For efficiency, we use subset that contains 9500 general instructions with responses labeled by GPT-4 (collected by Peng et al. (2023)) and 500 safety-related instructions with the original labels (Bianchi et al., 2023). The test we use contains 300 questions and reference answers taken from AlpacaFval (Li et al., 2023), alongside 278 safety questions and simple rejections ("Sorry, cannot help with that.") ") of the I-Ma iiiiiiii ciuusuttie Inst ruct ions and I-CoNA datasets used in (Bianchi al., 2023).
format all datasets into question-answer pairs following the same template:
Prompt template for all datasets
USER: \n infquestion hww..soso.........SSSSTTTTT (answer) An
A.2 EVALUATION
Sampling. For all tasks and models, we sample one response for each prompt using vLLM (Kwon et al., 2023) with beam search enabled and beam size set to 4.
Metries. For GSM8K, simply parse the final numerical answer and compute exact match accuracy. For BIRD, we download all databases and execute the generated code on them to compute execution accuracy, following (Li al., 2024). We ensure all models' generated code is executed on the same machine for fair comparison. For SaferPaca, we adopt the CoT prompt (shown below) from (Li et al., 2023) and use GPT-4o (OpenAI, 2024) to compute model answers' win rates against the reference answers. Specifically, model's s answer over the reference answer.
compute GPT-4o's mean probability of generating token that chooses the
Prompt template for GPT-40 evaluation
Select the output A or B that best matches the given instruction. Choose your preferred output, which should first contain concise explanation and then shculd end with ONLY the token: 'A" or 'B* (no det, no backticks, no new line, no quotes, no 'it depends', no "both' ...), because we will use 'output [-1] to extract the best output.
subjective. Your answer
Here's s an example:
Example:
# Instruction:
Give a description the following job: "aphthalmologisst"
## Output A:
An ophthalmelogist is medical coctor who specializes in the diagnosis and treatment of eye ciseases and conditions.
" Output B:
15

---

Published as a conference paper at ICLR 2025
An ophthalmologist is medical coctor who pokes and prods at your eyes while asking you letters from chart.
# Concise explanation followed by 'A" or 'B'
### Concise explanation
provides comprehensive and accurate description of the job of an ophthalmologist. In contrast, is more of a joke.
### Which is best, 'A' or 'B'
Task:
Now is the real task.
## Instruction: (instruction)
## Output 'A": (A)
## Output 'B": (B) Concise explanation followed by A or 18"
## Concise explanation
B LM-BASED SIMULATION DETAILS
For GSM8K, we experiment both with using the 2B model to supervise the 7B model, and using the 7B model to supervise the 70B model. For SaferPaca and BIRD, we consider 2B supervising 7B and 2B supervising 70B, respectively, to balance limited compute while maintaining sufficient performance gaps between models.
In each task, split the training set into halves. We use the first half as ground truth to train and 7. and then use ý to generate labels for the second half or use q to generate comparison feedback for answers questions the second half. This is similar to the setup in (Burns al., 2023), despite that we are operating on text generation tasks with both task demonstrations and comparison feedback.
To collect training data for q. we evenly save 10 intermediate checkpoints when training P and use them to sample answers for questions in their training set. Then, we pair these low-quality answers with ground truth answers, and train 7 to select the ground truth with standard binary classification loss. In SaferPaca, we balance refusals and acceptances (i.e., following the instruction) in the training data to avoid class bias in q. This is done by first training two á on safe and unsafe instructions in SaferPaca respectively and sampling answers from them for each instruction. In this way, we obtain both acceptance and refusal responses for each instruction, and then pair them up with the ground truth as q's balanced training data.
We use the following template compose two answers and the question into a prompt for q:
Prompt template for ğ
QUESTION: [qquestion)
ANSWER (A))
[[answer_a]
ANSWER (B)
16

---

Published as a conference paper at ICLR 2025
Table I: Training Hyperparameter for Each Dataset
Dataset
Epoch
Batch Size
Max Answer Token
GSM8K
2
32
256
BIRD
2
32
256
SaferPaca
4
32
512
Algorithm 1: Iterative Label Refinement (ILR)
Input Initial SFT dataset Du = (x', )%)
Comparison feedback annotator !
Maximum iterations K
Refinement
Output: :upput:Reinngg Refined dataset Dx
K Final model p SFT+ILR
for k = ( to K 1 do
Split D3 into two disjoint subsets:
DD = (x'1 11 and D = (x2, (),, where X' U Ux2 = X
Train models SET on Dj and STT on D
foreach I; E X do
L Sample one proposal 2, P 12,)
foreach 2; X do L Sample one proposal ~ (+,,))
foreach I; E X' do
Collect comparison feedback for (zi, i),,s) using ④ Proposal zạ is accepted if the feedback is 2, > Mk.
11
12
Let A be the set of indices where 2; is accepted
Limit |A| I ai|A'| by selecting proposals with highest confidence
14
Update labels:
if E A, otherwise
UK+111
UK
Form the updated dataset Da+1 = (X, (e+1)
16
17
Train model p k+1
SPT+ILR via SFT Dk+1
return DK. PSFT-HRR [K]
((answer_b))
You gave the anawer (A) to the question. Do you accept tc change it
(B) proposea another model?
Note that the prompt states order of "your answer" and "answer proposed by another model" due to historical reasons. In practice, we randomize the two answers' order to avoide position bias.
C IMPLEMENTATION DETAILS OF SFT, DPO, AND ILR
In all our experiments, we apply Low-Ran Row-Rank Adaptation (LoRA) (Hu et al., 2021) for efficient model training. We set r = 64 and a = 128 for all models.
SFT. For each task, we use consistent setting of epoch, batch size, and max answer token for all models (Table 1). We use set learning rate to 5e for Gemma 2B and le for Mistral 7B and Meta Llama 3 70B across all tasks. We use Adam (Kingma, 2014) optimizer for Gemma 2B and Mistral 7B and AdaFactor (Shazeer & Stern, 2018) for Meta Llama 3 70B. We enable gradient checkpointing and use gradient accumulation with mini batch size of 1 for all models.
17

---

Published as a conference paper at ICLR 2025
ILR. We present the pseudo-code of ILR in Algorithm 1. Each round of ILR uses the same training configuration as the initial SFT, since the only difference is that the dataset is refined. When training the half-data models used for generating new proposals, we also adopt the same SFT training configuration for simplicity.
In LM-simulated experiments, only collect comparison feedback for proposals and initial demonstrations that are sufficiently different. In GSM8K and BIRD, this is determined by having different final numerical answer or execution result. For SaferPaca, API to compute the embeddings for the two responses, and then take dot product to compute their embedding distance. We only collect comparison feedback for pairs with the top 50% largest embedding distance.
OpenAl Embeddings
DPO. We use the DPOTrainer in HuggingFace Transformers (Wolf al., 2020) to perform DPO training. In all tasks, we use the suggested rmsprop optimizer with learning rate set to le and we train for the same number of epochs as SFT. For all LM-simulated tasks, we sample 6 completion for each prompt, create 3 pairs with them, and then gather comparison feedback using q. We subsample the top 15% most confident feedback (measured by 14(y1 > y2 x) 0.5l) when constructing DRLHF. since we find this performs better than using all feedback generated by the unreliable é in DPO. Specifically, in GSM8K (2B + 7B), model accuracy after round of DPO is 0.3220.0003 when using all feedback and 0.36+0.0032 when using the 15% most confident feedback.
D HUMAN STUDY DETAILS
We use CloudResearch Connect to recruit online workers to write human demonstrations for 1000 Alpaca (Taori et al., 2023) instructions and provide comparison feedback that used in two rounds of ILR and DPO.
D.1 COLLECTING HUMAN DEMONSTRATIONS
In our survey for collecting human demonstrations, assign each worker 15 questions with additional screening questions formatted in the same way for filtering out low-quality responses. Specifically, the screening questions and filtering critiria are:
1. Instruction: How many letter R's does the word 'STRAWBERRY" have? Expected answer: Anything containing 3 or three.
2. Instruction: Weng earns $12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn?
Expected answer: Anything containing 10 or ten.
3. Instruction: want to know what dish can cook with these ingredients: eggs, tomatoes, salt, oil. Also, please give me list of steps cook it.
Expected answer: Anything containing the mentioned ingredients and least steps for cooking them.
The task instruction we provided at the beginning of the survey is:
Task instruction for writing response demonstrations
Thank you for participating in study! We would like your help in training Al assistant that follow users' instructions helpful, honest, and harmless way.
In this survey, you will be presented with 15 questions posed by human users to an Al chathot. For cach question, please take more than minute and less than 2 minutes to write response that you think is as helpful, honest, and harmless as possible. Please aim for high-quality detailed responses within the given time.
Additionally, there will be randomly placed quality check questions. These questions are super easy and you will definitely get them right if you attentive. We will manually check your answers to
htnp::/coomeet..onuteeeccc.o.m
18

---

Published as a conference paper at ICLR 2025
these questions, and if you don't pass, your response may not be qualified. However, you will still receive base payment of $0.75. In such case, please manually return the assignment ensure can pay you the base amount, Please let us know you find our judgement incorrect.
To get full compensation, please complete all questions. Otherwise, you will only receive base payment of $0.75.
At the end, we welcome your feedback on improving the survey for future participants!
NOTE; Please refrain from searching for answers during the survey. No use of external sources (e.g.. Google ChatGPT) is allowed.
D.2 COLLECTING HUMAN COMPARISON FEEDBACK
In each round of ILR and DPO, we collect human comparison feedback for all 1000 questions. For DPO, model completions are generated by models trained with 3 0.1. In our survey for collecting human comparison feedback, we assign each worker 50 questions with 3 additional screening questions formatted the same way for filtering out low-quality responses. Specifically, the screening questions and filtering critiria are:
1. Instruction: Where is water that has its salt removed before it can be used as drinking water most likely to have come from?
Response A: Water that has its salt removed before it can be used as drinking water is most likely to have come from a sea.
Response B: Water that has its salt removed before it can be used drinking water is likely to have come from a lake.
Expected answer: with any confidence level.
2. Instruction: Help me solve this math problem. Input: How can compute the of circle with radius 5?
Response A: The area of it is 25T.
Response B: Note that the area of a circle with radius r is x 73, Therefore, the area of circle with radius 5 * * 5 25...
Expected answer: B with any confidence level.
3. Instruction:
Immediately before and after running a 50 metre race, your pulse and breathing rates are taken. What changes would you expect to find?
Response A: After rumning 50-metre race, an increase in pulse and breathing rate is the changes one would expect to find immediately before and after. Response B: After running a 50 metre race, you would expect to find an increase in pulse but no change in breathing rate when your pulse and breathing rates are taken immediately before and after the race.
Expected answer: with any confidence level.
The task instruction we provided at the beginning of the survey is:
Task instruction for providing comparison feedback
Thank you much for participating in our study! We would like your help training Al agent that both helpful and honest.
First, you will be shown three screening questions, and if you get them correct, you will be able to participate in the rest of our survey. If you do not qualify. you will be eligible to re- ceive base payment of $0.75, and you will be automatically redirected to Google Form. In case, please manually return assignment otherwise we will be unable to pay you the base amount.
For the main task, you will he shown fifty request-response groups: each request is posed by a user an Al chatbot, and you will see two potential responses that the Al chatbot has generated. Please select the scale which of the answers you believe is honest and helpful. For instance, if you believe one choices more helpful, select the bubble corresponding one the two
19

---

Published as a conference paper at ICLR 2025
Rugulbrization (GMMXX-
Accuracy incresse vs. RL Divesperic
25.0%
2.2
2.6
-435
0.0
DPO Ohar Nerlooo
DFO
125
Unvilaaee
0.00
2.5
(a) DPO effectively improves SFT models with acle feedback, espccially with weak regularization. However, unreliable feedback. improvements are limited to a narrow range of regularization strengths because weak regularization causes overoptimization.
(b) Strong regularization in DPO limits useful model updates, while small regularization leads overop- timization unreliable feedback. In contrast, ILR facilitates large model updates, allowing for faster im- provement and efficient use of comparison feedback.
Figure DPO struggles avoid overoptimization noisy preferences while also updating the imitial suboptimal SET model sufficiently to improve performance.
extremes the scale, but if believe the two choices similar each other, select bubble the center of the scale. You can also think of this scale as your confidence the selection you made-the more confident you are, the closer your selection should to one of extremes. Option- ally, you can explain why you chose the response that you did in the scratch space provided on each page.
You should spend around 30 seconds and limit of this survey 50 minutes. Please read these instructions for more details how evaluate helpfulness and honesty. You may open the instructions in a separte tab for reference whenever you want. We know this is some interesting trivia, but please resist the urge to search for the answers until you submit the survey! No use of external sources, including Google or ChatGPT, is permitted.
than minute for each question. The total time
At the of survey, please give feedback on we can improve the survey experience future evaluators!
NOTE: Please do not take this survey more than once! You will not be compensated for more than attempt.
Some of the statements that you will see are factually inaccurate. Please do not rely on reference of this information in the future, and do not spread misinformation about the topics covered.
Disclaimer:
D.3 DPO AND ILR HYPEPERAMETERS
In both DPO and ILR, we only use comparison data with at least moderate confidence. We tested the first round of DPO using the top 10%, 20%, 30%, 40% and 80% most confident comparisons and found 30% yields the best performance, hence this is applied throughout the two rounds. We also tested 3 (0.01. 0,1, 0.5, 0.1] for the first round and found B 0.1 performs the best. For ILR, tested a E (0.1, 0.2, 0.33 for the first round and applied the best-performing a = 0.1 across the two rounds.
E ADDITIONAL EXPERIMENTS
E. KL REGULARIZATION DPO
In Figure 8, we present the complete experiment results when using different levels of KL regulariza- tion and feedback quality for DPO.
20

---

Published as a conference paper at ICLR 2025
G7B
SaferFaca (28
0.73
0.44
0.66
9.42
0.70
0.04
0.40
0.35
0.57
Round
Bound
Round
found
Figure 9: Naively replacing initial SFT labels with new model's proposals lead to performance degradation across all lasks, suggesting the importance of refinement oversight.
ILR Model Accuracy over Rounds
ILR Label Accuracy over Rounds
0.385
0.41
15%
15%
0.380
30%
Model Accuraoy
0.375
0.39
0.370
0.37
0.365
0.35
0.355
3 Rounds
5
Rounds
Figure Left: Overall, not particularly sensitive choice a. although larger may lead to less stable performance, while small a does not allow enough improvement. Right: When a is large enough, SFT data label accuracy converge to similar level. When is overly small, label accuracy is not efficiently improved.
E.2 FAILURE OF NAIVE ILR: SUPERVISON FOR REFINEMENT NECESSARY
To understand the importance of comparison feedback within ILR's refinement process, we compare to a naive approach that directly replaces an a fraction of the original labels with new proposals without any feedback. As shown in Figure 9, this naive method leads to performance degradation in tasks, showing that supervision in the refinement process, even unreliable, is necessary. could be that training model outputs without any curation leads to model collapse, phenomenon observed when training generative models with synthetic data (Shumailov et al., 2023; Ren et al., 2024; Gerstgrasser al., 2024).
E.3 EFFECT OF CONTROLLING REFINEMENT SPEED IN ILR
The refinement speed in ILR, controlled by a, plays a crucial role in maintining stability. Figure 10 illustrates the impact of different a values on model performance in GSM8K. We find a = 0.05 too small for effective improvement of label accuracy, while large a's make the refinement process less stable. According to this result, we set a 0.15 for all other experiments in LM-based simulation and find that this choice leads relatively stable performance across all tasks, showing that not particularly sensitive to change in data distribution. Future work can explore adaptive refinement speed to remove the potential need for manual hyperparameter tuning.
E.4 IMPROVING DPO WITH ILR MODULES
Several modules ILR can also be applied to DPO. In sDPO, we use two models trained on disjoint data subsets generate samples for the preference dataset, similar to how we generate proposals with the labss-labeling framework in ILR. wsDPO, we construct the preference dataset by comparing model-generatee responses with original unreliable labels, similar to the mechanism of using feedback to decide label refinement in ILR. We test these two variants of DPO in the GSM8K (2B - 7B) setting. As shown Table 2, these methods provide minimal improvements upon DPO. We conjecture
21

---

Published as a conference paper at ICLR 2025
Table 2: Accuracies of DPO variants with ILR modules across 4 rounds. The highest value in each column highlighted bold.
Method
Round
Round
Round
Round 4
DPO
0.3591
0.3520
0.3462
0.3412
sDPO
0.3624
0.3561
0.3457
0.3343
wsDPO
0.3626
0.3624
0.3538
0.3482
ILR
0.3647
0.3738
0.3851
0.3788
Table 3: Accuracies of DPO variants with improved loss functions across rounds. The highest value in each column is highlighted bold.
Method
Round
Round
Round
Round 4
DPO 0.3591
0.3520
0.3462
0.3412
IPO
0.3604
0.3571
0.3503
0.3460
cDPO 0.3589
0.3536
0.3563
0.3530
rDPO 0.3568
0.3487
0.3386
0.3391
ILR 0.3647
0.3738
0.3851
0.3788
that they still struggles due to similar reasons to standard DPO: Errors learned during the initial SFT stage being hard to correct through preference optimization, and limitations caused by the overoptimization issues discussed in Section 4.1.
E.5 ROBUST DPO LOSSES
Recent work has proposed several DPO losses that address preference noise. In Table 3, we compare IPO (Azar et al., 2024), cDPO (Mitchell, 2023), and rDPO (Chowdhury et al., 2024) to standard DPO and ILR. While these methods may help with random preference noise, they are less effective for handling unreliable supervision in our setting. This could be because both task demonstrations used in SFT and comparison feedback are unreliable in our setting. Moreover, noise in demonstrations or comparison feedback are systematic (Section 2) instead of random, unlike the random label flip assumed in some theoretical results (Mitchell, 2023; Chowdhury et al., 2024). Such systematic errors are particularly challenging because they may not be easily modeled using standard robust learning techniques.
22


Reviews:
[{"review_id": "BgOruvePt3", "reviewer": "Reviewer_R8qz", "summary": "The authors examine how to improve model performance when SFT and preference data is unreliable (contains some proportion of mistakes). They find that doing SFT training still results in improvements with unreliable data, but DPO does not improve. They propose a new method, iterative label refinement, which involves iteratively training models on data subsets, and then using the models to relabel the SFT data (with an unreliable supervisor deciding if the new label should be accepted), and retraining. The proposed method outperforms DPO when applied to SFT models both in an artificial setup (using small models to train larger ones) and a time-constrained human-labeller setup.", "questions": "1. I was a bit unclear on how the unreliable DPO dataset is constructed: so, you generate responses from a model checkpoint, and train a classifier to pick the original ground truth label instead of the model response, and then use this classifier to pick between two samples from the same model checkpoint? Or is the classifier the same across rounds?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I think the experimental setup is good, looking at a variety of models and tasks, and the human labeling experiment is also interesting.\n- The analysis of why DPO does not perform well with unreliable data is interesting, and provides reasonable evidence that DPO can fit too hard to the unreliable preferences.\n- The performance of the proposed method seems robust, with it being tested both in multiple settings and with (time-constrained) human annotators. The performance gains seem robust across the tasks and models tested.", "weaknesses": "- Missing Baseline: it would be good to compare against just doing SFT on the chosen sample or unreliable DPO pairs (my understanding is that this is not the same as training only on the model’s proposals because there is an additional trained classifier that may pick the right answer, albeit unreliably).\n- I wonder if the comparison between DPO and SFT/ILR methods are entirely fair since the DPO method doesn’t use the original ground truth answer, just two model generations. Could the DPO results perform better if the original ground truth label was also provided as an option when picking chosen and rejected pairs (as it is for the ILR approach)? Would it be possible to run the same ILR algorithm outlined in section 5.1, but rather than replace labels, just use the accepted/rejected proposals to construct chosen/rejected pairs instead?\n- Figure 6b is quite hard to parse, as the label box overlaps with a large chunk of the figure, and the differences between round 1 and 2 are somewhat small.\n- Did the authors try longer DPO training with a high beta? They clearly show that using a larger beta with unreliable feedback can improve a bit, I wonder if training for longer with higher beta results in further improvements with DPO or if it then starts to over-optimize as well (potentially a different way that the issues with DPO could be alleviated).\n\nOverall, I think the paper presents an interesting issue (that DPO doesn't seem to work well with unreliable label) and proposes an interesting alternative method that appears to work well in across a few models and tasks, although I think the comparisons between DPO and ILR could be made a bit tighter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mmqk5VSCh0", "reviewer": "Reviewer_J4qi", "summary": "This paper proposes iterative label refinement (ILR) as an alternative to reinforcement learning from human feedback (RLHF) for language model post-training under unreliable supervision. The authors argue that in settings where human supervision is unreliable, methods such as direct preference optimization (DPO) become less effective. Instead, ILR iteratively refines the supervised fine-tuning (SFT) dataset by replacing unreliable human demonstrations with better model-generated responses, thus improving the model’s training data. Experiments show that ILR outperforms DPO in tasks such as math, coding, and safe instruction-following, especially when human supervision is noisy or unreliable.", "questions": "1. How does ILR compare to having annotators review each sample in the SFT data to remove low-quality ones? This approach seems less expensive, as it avoids generating new responses with the SFT model and lets the annotators avoid pairwise comparisons.\n\n2. How frequently are the model-generated responses considered better than the SFT data depending on the size of the model?\n\n3. Typos: “Ground Truth” in Figure 5.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Motivation.** Safe post-training of language models with noisy fine-tuning data is a highly practical problem with a lot of relevant research done to address the issue.\n\n2. **Experiments and analyses.** The paper conducts a range of experiments, simulating unreliable supervision using small LMs and time-constrained human evaluators to evaluate ILR under multiple settings.\n\n3. **Performance.** ILR demonstrates improvements over DPO under the unreliable supervision settings considered in the work.", "weaknesses": "1. **Limited evaluation of alternatives.** Several approaches exist for handling noisy feedback data, including conservative DPO [1], which combines regular DPO with label smoothing, IPO [2], MMPO [3], and filtering noisy labels, among others. However, empirical evaluations of some of these methods in the unreliable supervision setting considered are missing.\n\n2. **Unreliable supervision not well-defined.** The concept of unreliability is not quantitatively well-defined. For example, it is unclear whether a small LM trained as a classification model actually provides unreliable comparison feedback, and if so, to what extent the labels are considered inaccurate. Relatedly, it is unclear how unreliable the “time-constrained” human supervision really is.\n\n3. **Questionable assumptions.** For the ILR framework to be effective, it seems that the SFT model needs to generate responses that are often better than those in the SFT dataset, and the annotator must be able to identify these better responses. While the latter assumption is discussed in the paper, the former depends on the SFT model being sufficiently capable and not overfitted to the SFT data, so that it can generate responses than the SFT data at least some of the time. This depends on factors such as the size of the SFT dataset, the proportion of unreliable data, and the size of the SFT model, raising questions about how generally applicable ILR is.\n\n---\n[1] Mitchell. \"A note on DPO with noisy preferences & relationship to IPO.\" (2023). \\\n[2] Azar et al. \"A general theoretical paradigm to understand learning from human preferences.\" (2024). \\\n[3] Kim et al. “Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback.” (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "srOXoTncus", "reviewer": "Reviewer_bW5s", "summary": "This paper proposes a novel approach, Iterative Label Refinement (ILR), aimed at improving language model (LM) post-training when demonstration and human supervision is unreliable. The authors identify that RLHF (specifically DPO) struggles under unreliable feedback due to overoptimization, and they propose ILR as an alternative that improves the supervised fine-tuning (SFT) dataset itself. The approach replaces low-quality human demonstrations with (small) model-generated alternatives based on comparison feedback, which is then used to retrain the model. The paper provides evidence that ILR outperforms DPO in various tasks, including math, coding and safe instruction-following, under both LM-simulated and human supervision settings.\n\nComments:\n- Scalability and Computational Cost: The paper could include a more detailed discussion on the computational trade-offs involved in using ILR versus DPO. Given that ILR requires multiple rounds of refinement, it would be useful to know how this affects training time and computational resources.\n- Clarification on Theoretical Boundaries: Theoretical analysis could be expanded to explore the long-term behavior of ILR. For example, will the refinement process converge effectively without introducing new errors or biases into the dataset over time?\n- Additional Visualizations: It would be helpful to see additional visualizations that show the progression of ILR versus DPO over multiple training rounds, especially in terms of how the refined dataset improves in quality and how the model’s performance evolves.\n- Task Extension: Future work could explore extending ILR to more diverse task types, including more subjective ones or tasks where human demonstrations are more error-prone (e.g., complex visual tasks or tasks requiring nuanced judgment).", "questions": "- How does ILR compare to other RLHF methods like PPO? Would similar overoptimization issues arise in other preference optimization methods, or are they more resistant to unreliable supervision?\n- Could ILR be extended or adapted to address tasks that involve more subjective or open-ended feedback, such as creative writing or opinion-based tasks?\n- Does ILR introduce any risks of bias through its iterative process, particularly as models continue to generate and refine their own data? How would you address or mitigate such risks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Novelty and Significance: ILR introduces a fresh approach to addressing the key challenge of unreliable human supervision in language model post-training. This is quite crucial as RLHF is becoming increasingly important for scaling LMs in practical applications.\n- Comprehensive Evaluation: The experiments cover different tasks including math, coding, safe instruction-following and use both LM and real human feedback, ensuring the findings are well-supported. The robustness of ILR under different conditions and tasks is thoroughly tested and clearly explained.\n- Clear Motivation: The paper presents a strong investigation and reasoning for why RLHF, especially DPO, struggles under weak supervision. The overoptimization problem is well-identified and investigated, which motivates the need for a method like ILR, which focuses on refining the training data rather than the model directly.\n- Effective Use of Feedback: The authors highlight the importance of directing comparison feedback towards improving the dataset, not just the model. This is a significant insight and could influence future work in RLHF, encouraging researchers to rethink more about how to utilize human feedback in post-training.", "weaknesses": "- Potential ensembling effect that may explain performance gain: 5.1 describes training one model on each half of the SFT data, then using these models to cross-label during ILR. There does not seem to be an ablation either. I think a fair comparison would be against a two-model ensemble, as opposed to a single model baseline. \n- Limited Comparison with other RLHF Methods: While the paper focuses on comparing ILR to DPO, it does not address how ILR might perform compared to other RLHF algorithms, like PPO. Including more comparisons could make the paper’s conclusions more robust and introduce a broader impact.\n- Computational Cost and Scalability: The ILR approach, which requires iterative refinement and cross-evaluation of datasets, which means  both SFT model and DPO needs to be retrained for every iteration. This might introduce higher computational complexity compared to DPO or traditional RLHF methods. The paper could benefit from a more detailed discussion of the trade-offs between performance improvement and computational cost.\n- Task Diversity: While the paper tests ILR on math, coding, and safe instruction-following, it would be beneficial to see it applied to a broader range of tasks, particularly more complex or open-ended ones where human feedback is highly subjective or unreliable (e.g., creative writing or subjective question-answering or toxicity).\n- Theoretical Depth: The theoretical tracking of ILR is solid, but the paper could elaborate more on the theoretical limits of ILR. In particular, the long-term stability of ILR, and whether it risks introducing new forms of bias through the iterative refinement process, could be explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HRXIkezTy", "reviewer": "Reviewer_mmnq", "summary": "The paper shows that DPO is ineffective in the presence of unreliable feedback. The paper introduces a new method that replaces data samples in the SFT demonstrations dataset with synthetic demonstrations if the reward model prefers the synthetic demonstration. This process is iterated.", "questions": "- Is it specifically necessary to do 1:1 replacement of training data points with new synthetic ones? Does it make sense to decouple the number of data points removed due to low quality, and the number of new synthetic data points accepted? How much of the effectiveness is due to the filtering vs the addition of synthetic data points?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Demonstrating the failure of DPO via overoptimization in the presence of unreliable feedback is important, given the prevalence of unreliable feedback.\n- The introduced method is simple and performs clearly better.\n- Validation on human setting increases confidence that the results on synthetic setting are reasonable", "weaknesses": "- Doesn't compare against PPO", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)