Your Weak LLM is Secretly a Strong Teacher for Alignment
Authors: Leitian Tao
Yixuan Li
Venue: ICLR 2025 Poster (2025)
Keywords: large language models; alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The burgeoning capabilities of large language models (LLMs) have underscored the need for alignment to ensure these models act in accordance with human values and intentions. Existing alignment frameworks present constraints either in the form of expensive human effort or high computational costs. This paper explores a promising middle ground, where we employ a weak LLM that is significantly less resource-intensive than top-tier models, yet offers more automation than purely human feedback. We present a systematic study to evaluate and understand weak LLM's ability to generate feedback for alignment. Our empirical findings demonstrate that weak LLMs can provide feedback that rivals or even exceeds that of fully human-annotated data. Our study indicates a minimized impact of model size on feedback efficacy, shedding light on a scalable and sustainable alignment strategy. To deepen our understanding of alignment under weak LLM feedback, we conduct a series of qualitative and quantitative analyses, offering novel insights into the quality discrepancies between human feedback vs. weak LLM feedback. Code is publicly available at https://github.com/deeplearning-wisc/weak_llm_teacher.

Published as a conference paper at ICLR 2025
YOUR WEAK LLM IS SECRETLY A STRONG TEACHER FOR ALIGNMENT
Leitian Tao, Yixuan Li
Department of Computer Sciences, University of Wisconsin-Madison (leitiantao, sharon1f @css wi wisc.edu
ABSTRACT
The burgeoning capabilities of large language models (LLMs) have underscored the need for alignment to ensure these models act in accordance with human values and intentions. Existing alignment frameworks present constraints either in the form of expensive human effort or high computational costs. This pa- per explores promising middle ground, where we employ a weak LLM that is significantly less resource-intensive than top-tier models, yet offers more au- tomation than purely human feedback. We present a systematic study to evaluate and understand weak LLM's ability to generate feedback for alignment. Our empirical findings demonstrate that weak L.LMs can provide feedback that ri- vals or even exceeds that of fully human-annotated data. Our study indicates a minimized impact of model size on feedback efficacy, shedding light on a scalable and sustainable alignment strategy. To deepen our understanding of alignment under weak LLM feedback, we conduct a series of qualitative and quantitative analyses, offering novel insights into the quality discrepancies be- tween human feedback vs. weak LLM feedback. Code is publicly available at https: / / thub. cww/deepll ea en rning- wi /ww//ekkk........ 1m teacher.
INTRODUCTION
As we observe the impressive capabilities of large language models (LLMs) across diverse applica- tions (Brown al., 2020; Achiam al., 2023; Bubeck et al., 2023; Team al., 2023; Anthropic, 2023), there emerges critical need to ensure Al systems are helpful and harmless. Al alignment aims to harmonize Al behaviors with human intentions and values and ensure safe and desirable behavior. A key recipe to achieve alignment involves presenting pairs of responses and collecting binary feedback (e.g., preferred, less preferred) based on the comparative quality of these responses. The prevailing methods in alignment can be categorized based on the source of the feedback. For example, the popular framework Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022) relies pure human judgments, which often involves considerable human labor and manual effort. On the other end of the spectrum, framework such Reinforcement Learning from AI Feedback (RLAIF) (Bai et al., 2022a; Lee et al., 2023) har- nesses feedback from high-capacin LLMs to annotate preference datasets, often incurring significant computational and financial costs, along with the need for heavy prompt engineering.
These contrasting frameworks highlight two extremes of a feedback spectrum, raising critical ques- tions about the largely untapped middle ground that leverages the strengths of both while alleviating their respective drawbacks. As shown in Figure 1, this middle ground entails using significantly smaller LLMs that are less snsource-intensin than top-tier models, yet offer more automation than purely human feedback. For instance, while a model like GPT-4 might have trillions of parameters, a weak I.IM might only have hundreds of millions or even fewer. This smaller size means they inherently demand less computational power, thus reducing operational costs and enabling faster iter- ation cycles in development. This approach not only aligns with the practical demands deploying scalable AI solutions but also addresses the need for sustainable development within AI research, making it prudent choice for tasks where the ultra-high capabilities of matte-o-thhe-re models do not necessarily translate significant improvements in feedback quality.
Despite the appeal, the research community still lacks a systematic evaluation and understanding of weak L.LMs' capability generate feedback for alignment. Motivated by this gap, undertake

---

Published as a conference paper at ICLR 2025
Question: What kind vegetables offer the moat health benefits? Response I'm doctor, but my guess that kale, broccoli, cauliflwwrr and cabbage good sources of vitamins and liber. Response B: Most health benefits, or more specifically, most nutrients?
Pure human feedback
Weak LLM feedback
Ultra-large LLM feedback
Lebor intersive
More aulomation losa rosource insensive insouroo-iininiiie
Computationaly expensive, reflace prompt ongiooring
-100M parsmeteess
>1 Tillion pirsmalara
RLHF (Ouyang et al. 2022)
Our focus
RLAIF (Lee ot al. 2023)
Spectrum feedback alignment
Figure 1: spectrum of feedback for aligning LLMs. ranging from labor-intensier human annotations (e.g.. RLHF (Ouyang et al., 2022)) to highly automated, resource-intensin LLM feedback (e.g.. RLAIF (Bai et al., 2022a; Lee et al., 2023)). Our work explores the largely untapped middle ground, evaluating and understanding the use of weak LLM feedback for alignment.
comprehensive investigation of alignment via weak LLM feedback. Specifically, this paper makes three key contributions:
Contribution 1: A framework for evaluating alignment via weak LLM feedback. We formalize a learning and evaluation workflow utilizing feedback from weak LLM instead of traditional human annotations. Our framework operates the combination of labeled and unlabeled preference datasets. Different from the existing RLHF framework, we leverage data comprising of unlabeled triplet (x, y1, 12), where both y1 and y/2 are responses corresponding prompt but the preference is unknown. In practice, the unlabeled triplets can be collected in large volumes without the need for human annotations. weak LLM trained on labeled data will provide preference feedback on the large unlabeled data. Finally, we train target L.IM policy based on the weak I.L.'s feedback. The framework introduces novel perspective by connecting semi-supervised learning to alignment- field that has yet to be thoroughly explored, especially in the context of using weak AI feedback.
Contribution 2: comprehensive evaluation and novel empirical findings (Section 3). We employ the framework to evaluate the impact of weak LLM feedback alignment across a variety of model scales and diverse model families. Intriguingly, our results reveal that using a weak LLM, match
with size small as 125M (Zhang al., 2022), provide preference label for alignment or even exceed the performance of using full human feedback (see Figure 2 and Figure 3). Moreover, systematically evaluate alignment performance when the feedback is provided by L.MM of varying capacities: a weak supervisor (Llama-3-8B), and very strong supervisor (GPT-4). We found that the performance under weak, moderate, and strong supervisors is nearly comparable, suggesting that the supervisor model's size has minimal impact on feedback effectiveness. Notably, the weak LLM, OPT-125M, outperforms the more advanced GPT-4, indicating that task-specific weak LLM can provide more effective feedback than larger, more powerful LLM that relies solely on prompt engineering.
a moderate supervisor (OPT-1.3)) 1.3B), strong supervisor
Contribution 3: An in-depth analysis on the quality of weak LLM's feedback (Section 4). To deepen our understanding of alignment under weak L.M feedback and reason our observations, we conduct series of qualitative and quantitative analyses. A pivotal aspect of our study lies in the examination of quality discrepancies between human feedback vs. weak LLM feedback. threefold: (1) when weak LLM's chosen response contradicts human's
Our key observations choice, nearly half of these responses exhibit higher quality, suggesting the lack of reliability in human feedback and weak LLM can surpass human judgments; (2) weak LLM's feedback can be qualitatively similar to human feedback in contexts with a clear gap. yet they exhibit uncertainty in judgments when the distinctions in response quality are subtle; and (3) advanced L.LMs like GPT-4
2

---

Published as a conference paper at ICLR 2025
also show increased feedback inconsistency in scenarios where response distinctions minimal, highlighting the challenges faced by strong LLMs in providing feedback.
PRELIMINARIES ON LLM ALIGNMENT
We denote xo as a language model policy parameterized by 0, which takes in an input prompt æ, and outputs discrete probability distribution To over the vocabulary space V. To (y/ar) refers to the model's probability outputting response y given input prompt z. Alignment algorithms operate on comparative data, where pairs of responses are presented, and the model is trained to produce the preferred response given a query. Formally, we define the preference data below.
Definition 2.1 (Preference data) Consider two responses ye: Yr for an input prompt 2, we denote We yr if ye is preferred over yr- We call Ye the chosen preferred response and 1/r the rejected response. Each triplet (z, yes Yr) is referred to as a preference. Furthermore, the empirical dataset D = [1 Vo,ks Vr.a) 1n consists such triplets sampled from preference distribution.
Reinforcement Learning from Human Feedback. RLHF a widely used paradigm for fine- tuning language models based on human preferences (Christiano et al., 2017; Ziegler et al., 2019; Ouyang al., 2022; Bai et al., 2022a). The key stages in RLHF are reward modeling and reinforce- ment learning with the learned reward function.
Reward modeling learns function mapping, which takes in the prompt and response y and outputs scalar value r(z, y) signifying the reward. A preferred response should receive higher reward, and vice versa. Based on the Bradley-Terr model (Bradley Terry, 1952), the reward function optimized over dataset of human preferences, with the following objective:
LR WE (r(x,, ye) r(r,,r))))
where denotes the sigmoid function. Using the learned reward function, the model is further fine-tuned with reinforcement learning to maximize the expected rewards, thus promoting exploration and adherence to learned preferences. The optimization objective is formulated as follows:
max Eg) 0) -llgg
We
Jeer (ū)z))
where i represents the response generated by the current policy for the prompt x. indicates the reference policy or the initial policy before running RL optimization, and 3 serves hyperparameter to regulate the KI. divergence.
Training with RLHF can be computationally expensive due to the use of multiple models. As an alternative, Rafailov et al. (2023) proposed to directly optimize for the policy best satisfying the preferences with a simple objective:
Cpeo Neet D) -B(.....-)DD log (3 (1(1og TT(Belz) log
To T (x) Teef (3+ (x) )
(3)
Tiee (yec
Rafailov et al. (2023) showed that under mild assumptions, the optimal policy under the DPO objective (3) is equivalent to the optimal policy under the RLHF objective (2). This objective facilitates more direct reflection of human preference judgments within the optimization framework.
HOw GOOD IS ALIGNMENT WITH WEAK LLM FEEDBACK?
In existing alignment approaches described above, models are often trained on fully supervised data where each preference label is hand-annotated by humans. To systematically evaluate the reliability of using feedback from weak I.LM instead of human feedback, we outline the training workflow in Section 3.1, experimental setup in Section 3.2, and present our main findings in Section 3.3.
3.1 ALIGNMENT VIA WEAK LLM FEEDBACK
Consider two empirical datasets, D, and D.,, representing a labeled preference dataset and an unlabeled dataset respectively. The labeled dataset DD comprises of triplets (xr, Mes yr), where
3

---

Published as a conference paper at ICLR 2025
human annotators have indicated known preference Be yr- Du. the other hand, comprises of unlabeled triplet (z, 91, 922,, where both 1/1 and 1/2 are responses corresponding to x but the preference is unknown. A weak LLM, trained on Dr. will provide preference feedback on the unlabeled data DD,- This setup offers practical advantages since unlabeled triplets can be collected in large volumes without the need for human preference annotations. For example, by querying varying LLMs with the same prompt. Below, we describe the alignment process using feedback from the weak LLM.
generate multiple responses
Preference feedback from the weak LLM. We first train a weak language policy Tw based on the labeled preference dataset Dr. We use the subscript to indicate "weak" in the remainder of the paper. Specifically, the weak LLM is optimized using the DPO loss (cf. Equation 3), under which the optimal policy is equivalent to that of RLHF:
W argmin LDPO (Tws π SFT. Dt),
where " signifies the policy for the weak model, trained with DPO loss. TSFT is the reference model or the initialization, which is an SFT model fine-tuned on the preferred question-answer pairs (zz Ve) in Dj. Compared to directly using the untuned base model reference model, performing SFT enhances the model's 's ability to generate desired responses to questions (see Appendix C).
We then generate the weak feedback for unlabeled data Du., leveraging the weak language model Th For each triplet (x. 3/2 E 'D., we compute the reward Far (Ir, y1) and Tw (I. y2) according to DPO's implicit reward:
Tu (y)
Tw (z. 3) = log
(5)
T (y|æ)
SFT
We then assign the preference label ÿe for the response with higher predicted reward, and dr for the response with a lower predicted reward. Mathematically:
81
Tw (I, > Tw (z, 122
De
y2
Tw (z, 31) S Tw (2, V2)
We denote the resulting weakly labeled dataset as Dweak = {1x, Dc: Dr)) where [Dweak| = |Dul]
Alignment with feedback from the weak LLM. Finally, we train an LLM policy mo based on weak LLM feedback D The model is aligned using the following objective:
T argmin (TP To
:D
(6)
where Th the resulting policy based on the weak LLM feedback. To obtain the reference model Tg we fine-tune the base model using question answer pairs (I, ÿc) in Dwesk. Under the training workflow, a central unresolved question we address s in this paper is how effectively LLMs can be aligned using feedback from weak LLMS as opposed relying solely on human feedback. The training workflow thus serves as the foundation to explore our core research question. To understand this, we conduct a systematic evaluation in the next section.
-answer
3.2 EXPERIMENTAL SETUP
Dataset. To evaluate the performance, we use the Anthropic HH-RLHF (Helpful and Harmless) dataset (Bai et al., 2022a), which the most commonly used dataset for alignment. The dataset consists of 12,000 training samples and 12,500 test samples and is publicly available. Each sample includes prompt and two responses, with one being preferred over the other. The selected responses annotated based the opinions of crowd workers, who assess which response is more helpful and harmless. We preprocess the dataset by filtering out samples with token lengths greater than 512, which yields 100,000 training samples and 11,000 test samples. We split the training data into two disjoint sets. The first subset is used as labeled data D;, and the remainder is used as the unlabeled data D., (by disregarding the preference labels). We will vary the size of D; in our ablation. We also evaluate on the Reddit TL.DR (TL;DR) summarization dataset from Stiennon et al. (2020), which consists of a Reddit post and several short summaries, judged for quality and informativeness by human evaluators.
4

---

Published as a conference paper at ICLR 2025
Different size of student model
Different sources feedback
weak LLM feedback
human feecbbck
6.65
6.19
6
5.585,39
6
Gold reward
4. 4.63
Gold reward G P.
4.63
4,84
4.78
4.91
3-
1
GPT-1.3b
OFT-2.76
0PT-6.7b
OPT-13b
human
weak LLM
moderate LLM
strong LLM
very strong (GPF-4)
Figure 2: (a) Alignment with feedback from weak LLM (OPT- 125M) outperform human feedback. (b) Alignment performance on OPT-1.3B model under varying capability supervisor. See Section for details.
Models. For main experiment, utilize the OPT family model introduced by Zhang et al. (2022), which provides a spectrum of model sizes. This versatility enables us to evaluate the performance across different levels of model capability. Specifically, varying sizes (1.3B, 2.7B, 6.7B, and 13B) our LLM policy models, trained with feedback from weak LLM. Additionally, to validate the efficacy of our experiments, we consider more advanced open-sourced models, including Llama-2-7B (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023a) and Gemma-7B (Team al., 2024). For a comprehensive description of the hyper-parametent employed in our experiments, please refer to Appendix A.
employ OPT models of
Evaluation metrics. Given the test set Drest we evaluate the generation performance under two policies: z (with weak LLM feedback), and ni (with human feedback). デ is trained with DPO loss on the same set of triplets in D... except for using the original preference label provided by human annotators. This allows us estimate the performance achieved with fully supervised data. For fair comparison, T& and गं, always share the same model capacity and only differ in the source of feedback. We assess the generation performance using the following metrics:
Gold reward: Previous studies (Gao et al., 2023; Coste et al., 2024; Xiong et al., 2023) commonly utilize gold reward a metric for assessing the generation quality of language models. Gold reward is desirable due to the high cost associated with obtaining ground truth gold rewards from human annotators. We employ the output of large auxiliary gold reward model, denoted as r Tgold- to evaluate the quality of generated responses. For each test input prompt from Dust, we generate a response ý according to a given language policy, and then compute its gold reward as (ar Ú)- A higher gold reward signifies that the model's responses hetter align with desired preferences. Our results are consistent under alternative gold reward models; see Appendix C for details.
GPT-4 win-rate: We employ GPT-4 proxy for human evaluation by instructing it to review two responses (from two different policies) to the same prompt. It then assigns a rating on scale from 1 to 10. A higher win rate indicates that policy on average produces more favorable answers.
3.3 MAIN RESULTS
Alignment with weak LLM feedback can outperform human feedback. In Figure 2(a), evaluate the alignment performance using feedback from the weak LLM vs. human. Employing OPT-125M as our weak LLM to provide supervision (with the lowest capacity in the model family). we align student models of varying capacities (OPT-1.3B OPT-2.7B, OPT-6.7B, and 03T-13B). Interestingly, the alignment performance using weak L.M feedback closely matches or even surpasses that of using human feedback. The similar alignment performance between тे and πī has not been observed in previous studies. Moreover, we provide additional evaluation in Appendix C, and show that the observation holds for different weak LLMs such as GPT-Neo- 125M (Gao et al., 2020) and Pythia- 160M (Biderman et al., 2023). Given that the primary objective of alignment is to enhance the model's 's ability to generate content that better resonates with human goals, our findings suggest that leveraging weak LLM feedback is a promising route without access to full human supervision. For an in-depth analysis of the unexpectedly high efficacy of weak LLM feedback, we defer to Section 4.
5

---

Published as a conference paper at ICLR 2025
D ffferen model families
GPT-4 evaluation
12
Win
Tie
Lose
weak LLM feedback human feedback
8.52
7.90
8.20
Llama-78 7B
7.28
OFT-6.78
Gold
OPT-2.78
OFT-1.38
Mistral-78 Llama-2-78
Gemma-78
100
Percentage
Figure 3: (a) Results different model families. (b) GPT-4 evaluation for different models aligned with weak LLM feedback vs. human feedback.
Alignment performance under the varying capability of supervisor. Existing frameworks learning from AT feedback, such as RLAIF (Lee et al., 2023), employ more advanced LLMs (e.g., GPT-4) annotate preference datasets for the alignment of student models. To understand the alignment performance under different supervisor models, we systematically consider spectrum of supervisor LLMs of varying capabilities: (1) weak supervisor based on OPT-125M, (2) moderate supervisor based on OPT-1.3B, (3) strong supervisor based on Llama-3-8B. and (4) very strong supervisor based on GPT-4. For variants (1) through (3), we apply the same training workflow described in Section 3. For variant (4), we utilize GPT-4 as the LLM labeler, following the prompt suggested by Lee et al. (2023). Based on these four different sources of feedback, we compare the alignment performance on the same policy model OPT-1.3B As shown in Figure 2 (b), the aligmment performance under the weak, moderate, and strong supervisors is nearly comparable. This suggests that the size of the supervisor model plays less impactful role in the effectiveness of feedback. Notably, we observe instances where the weak L.M with a small capacity, OPM,125M, outperform GPT-4 in providing feedback preferences. Although GPT-4 is more advanced model, this finding indicates that a task-specific weak LLM can serve as a more effective supervisor than larger, more powerful LLM that relies solely on prompt engineering.
3.4 ADDITIONAL ABLATIONS
Results on different model families. To further validate our findings, we extend our evaluation to additional model families, including Llama-2-7B (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023a) and Gemma-7B (Team et al., 2024). We employ OPT-125M as the weak LLM to provide preference labels. Figure 3(a) displays the average gold rewards of the generated responses, based policies TH (with weak LLM feedback) and тī (with human feedback). The gold rewards achieved using weak LLM feedback consistently surpass those obtained from human feedback. This underscores the effectiveness of alignment through weak LLM feedback across various LLM families.
Results under GPT-4 evaluation. Beyond gold reward measurement, we assess the win-rate of the model aligned with weak LLM feedback (zz) against the human feedback (Tī)), utilizing GPT-4 to judge the helpfulness and harmlessness within dialogue contexts. We randomly select 100 prompts from the set of HH-RLHF and employed GPT-4 determine whether responses generated under the policy nे were superior to those from -. Considering the diversity of model families and sizes, we evaluated the GPT-4 win-rate across five student model variants: OPT-1.3B, OPT-2.7B, OPT-6. I. llama-2-7B, and Mistral-7B. The weak LLM feedback all comes from the OPT-125m. As illustrated in Figure (b), the win-rate for policy zi (with weak L.LM feedback) approaches 50% for all cases, indicating that its performance competitively matches that of using human feedback イ
Results on different datasets and tasks. To examine the effectiveness of weak LLM feedback for alignment across broader NLP tasks, turn our attention to the Reddit TL;DR dataset (Stiennon et al., 2020), which focuses on summarization a stark contrast to dialogue tasks, which consists of a Reddit post and several short summaries, judged for quality and informativeness by human evaluators. Similar to the setup for HH-RLHF, we use OPT- 1PMM as the weak LLM and assess the performance of weak I.LM feedback human feedback for different student sizes: OPT-1.3B, OPT-2.7B,

---

Published as a conference paper at ICLR 2025
(a) Results the TL:DR
(b) Different data sizes
feedback
hurann feedback
3.76
6.03
3.63
4.17
4.84 4.63
Cold rewears
3.45
2.71
Gold
OPT-1.36
OPT-2.7b
OPT-13B
Student model
Size of the labeled dataset
Figure 4: (a) Results on different tasks (TL;DR). (b) Results under different data sizes.
OPT-6.7B, and OPT-13B. As depicted in Figure (a), the average gold rewards for outputs sampled under the policies Th also outperform Ti across all student model sizes. This finding reinforces the efficacy of weak LLM feedback for alternative linguistic tasks.
Results under different data sizes. In this ablation, we systematically investigate the impact of dataset size. We vary the ratio between the labeled dataset size and the full dataset size: 1/16, 1/8, 1/4, 1/2, with the remaining data serving as the unlabeled subset. We train the weak LLM across different ratios with the same steps. We utilize OPT-125X. as the weak LLM to align the policy model OPT-1.3B. Notably, as shown in Figure 4 (b), even under small labeled dataset size with ratio of 1/16, the performance using weak LLM feedback is favorably close that of using human feedback.
Section 3 key takeaways
1. Using a weak LLM, with size as small as 125M, to provide preference feedback for aligument can match or even exceed the performance of using pure human feedback.
Alignment performance can similar using feedback from supervisor LLMs of varying capabilities- from weak (125M), moderate (1.3B), strong (8B) to very strong (GPT-4). This suggests that the size of the supervisor LLM plays a less impactful role in providing feedback for alignment.
Our finding consistently holds across different model families, evaluation metries, and tasks.
4 IN-DEPTH UNDERSTANDING OF WEAK LLM'S FEEDBACK
When weak LLM feedback matches/missatche human feedback. In this ablation, we investi- gate the impact of samples either matching or mismatching human feedback. Adopting OPT- 125M as the weak LLM to provide feedback for the OPT-1.3B model alignment, we design three controlled settings for this ablation:
Deaak: Use all samples, {(x, Des ú.)). for training. This is the same as our main setting.
Dmmachh Use samples with weak labels matching the human feedback, i.e., (rw (z, Ve) Fw (1 vr)) Dmismatch, Use samples with labels mismatching the human feedback, i.e., (r (35, ye) rmn (2, Ve)).
As shown in Table 1, training with Dmaach (sam- ples with matching labels between weak LLM feedback and human feedback) yields gold re- wards comparable to those obtained from using the full set D Additionally, when examin- ing the alignment performance using weak the gold reward significantly improves relative the pre-trained strong model without any fine- tuning by 1.79. This indicates that weak effectively improves alignment, even when the weak LLM provides feedback that are entirely contrary the human preference. We further confirm that the noise of preference labels in Dweak appears to behave differently from ran- dom noise (see Appendix C). The phenomenon is intriguing and prompts our next analysis.
Table 1: Effect of purifying weak LLM feedback. Num- bers in brackets denote improvement relative to the pre- trained student model without any fine-tuning.
% matching human feedback
Gold reward of aligned model T
D weak
60.6%
4.84 (+2.61)
100%
4.78 (+2.56)
weak Diismmttc
0%
4.01 (+1.79)
7

---

Published as a conference paper at ICLR 2025
Table Qualitative example of the chosen and rejected responses in weok
Prompt
Human: What kind of vegetables ofter the most health benefits?
Assistant:
Chosen response by weak LLM (rejected by human)
I'm not a doctor, but my guess is that kale, brocpoli, caulifiower, and cabbage are good sources of vitamins and fiber. (gold reward: 6.82)
Rejected response by weak LLM (chosen by human) Most health benefits, or more specifically, most nutrients? (gold reward: 3.25)
Dive into the quality of weak LLM vs. hu- man feedback. In Table 3, we provide a qual- itative example from Dimisnatc where the cho- sen response by weak LLM mismatches that of the human feedback. More qualitative ex- amples are in Appendix B. Despite contradict- ing the human preference, the chosen response by the weak LLM has higher quality. We fur- ther measure the gold reward as proxy for response quality, which is indeed higher for the response chosen by the weak L.M,, i.e., "goia ūe) > T'gaa (2,, Ur). Across all the sam- ples in Diismatch, we notice that 44.3% samples display a higher gold reward for the response chosen by the weak LLM. This indicates thut the preferences from weak LLM in Dimismaach are not
Table 2: Summary statistics of average gold re- ward on HH-RLHF dataset. "Chosen" and "Re- jected" indicate the human preference for original dataset Dumman and the weak LLM preference for the Dweak, weak and Deaak
Chosen
Rejected
A
Druman
5.77
4.23
1.52
Dweak Dmatch
5.63
4.39
1.23
5.93
3.66
2.27
Dimimmtch weak
5.17
5.53
-0.36
entirely erroneous, but better than human feedback in nearly half of the cases'. Considering the feedback from weak LLM and human shares Dmatth portion and only differs in Dmismmath the close performance achieved between the two models can be attributed to the fact that human feedback is similarly imperfect.
Building on the observation that weak LLM can sometimes surpass human judgment, we delve into the characteristics of datasets produced under such supervision. Table 2 shows that the gold rewards for chosen responses in Dmiimmtth are average comparable to those for rejected responses, while the gold rewards for chosen responses in Dimatc are significantly higher than the rejected responses. This suggests that the weak LLM primarily errs between subtly different choices, while it remains reliable distinguishing between options with clear quality distinctions. Overall, the discrepancy (A) within D weak is similar to that in D Dhuman indicating that the quality of weak labels is not severely compromised.
More qualitative analysis. To further investigate the quality gap in responses from the dataset where weak LLM feedback and human feedback diverge (Dmissmatch), we conducted a qualitative assessment using GPT-4. Our goal is determine whether GPT-4's preferences for chosen and rejected responses in Dautth and weak remain consistent. We hypothesize that consistent preferences from GPT-4 across multiple evaluations would indicate a significant quality dif- ference between the responses, while fluctuating preferences would suggest that the responses are less distinguishable in quality. To quantify this, we prompt GPT-4 ten times consecutively to identify which answer was more helpful and less harmful. We randomly sample 100 examples from Dmaach and Dmiismatch and measure prefer- ence consistency by calculating the fraction of majority votes for each question, which ranges from 0.5 to 1. For instance, Table 5 presents qualitative example from Dinismatch where the weak LLM and humans have different preferences. We observe that GPT-4 indeed exhibits mixed preferences across the ten evaluations. indicating a low preference consistency (0.5) and thus the indistinguishable nature between the two responses. Indeed, as illustrated in Table 4, the average consistency for
Table 4: The GPT-4 preference consistency.
Dataset
consistency
Dmatch
0.84
Dmiimmrch
0.66
'This is further evidenced by our GPT-4 evaluation, where the response chosen by weak LIM achieved 46% win rate over the rejected response.
8

---

Published as a conference paper at ICLR 2025
Table A qualitative example in Detismath with low preference consistency in GPT-4 eval.
Prompt
Human: I'm visiting the tropics and Id like to go snorkeling in coral reets see fish, but I've never done belore. What equimment do néed?
can probably just up beach and rent snorkeling mask, snorkel, lins, and a life jacket for a very small fec. I'd recommend you try snarkeling on calm, flat water as it's more relaxing.
Assistant: Snorkeling is very comman among
Human: Don't life jackets prevent people from going underwater? Why would one snorkeling underwater? Assistant:
supervisor (rejected by human) A life jackel is an inllalable vest that helps you lloal on the waler. It's not supposed prevent you from going underwater.
Chosen response
Rejected response by weak supervisor (chosen by human)
Right, I was just going to say that you might feel more comfortable weering life jacket until you get comfortable with the water. l've lound that
most advanced snorkelers prefer wear life jackets the beginning.
GPT-4 preferred answer with 10 consecutive trials (consistency: 0.5)
First
Second
Firsi
First
Second
Second
First
Second
Second
First
is lower than that of Dmarch This suggests that advanced LLMs such as GPT-4 experience significantly higher inconsistency when the quality distinctions between two responses minimal, highlighting the challenges faced by both weak and ultra-large LLMs in providing feedback.
samples in weak
Section 4 key takeaways
1. When weak LLM's chosen response contradicts human feedback, nearly half of these responses exhibit higher gold rewards, suggesting that weak LLM sometimes surpass human judgments.
Weak LLM primarily ems between choices that are subtly different, while it remains reliable in distinguishing between options with clear quality distinctions.
Feedback from advanced LLM (e.g.. GPT-4) can exhibit high inconsistency feedback when the distinctions between responses are subtle.
DISCUSSION
Differences w.r.f. Burns al. (2024). Previous work by Burns et al. (2024) explored weak-to- strong generalization, wherein less capable model's weak supervision signals are used to guide a stronger, larger model. However, their study was limited to much simpler learning tasks like reward modeling and binary classification, where the outputs are either scalar or categorical labels. They did not investigate the more challenging task of language generation, which is closely tied alignment and involves a significantly more complex output space. Hence, the potential of weak LLM feedback for alignment remains unexplored in Burns et al. (2024), which is the novel focus of our study.
Contrary to Burns et al. (2024), which employed discriminative metrics such accuracy, our study evaluates the generative performance essential for AI alignment. Although Burns et al. (2024) reported noticeable performance gap between models trained with weak LLM feedback versus those with human feedback, our findings challenge these conclusions. Considering that human preferences noisy and unreliable (Yeh et al., 2024), accuracy on the preference test set is a questionable metric. Our works show contrary conclusions when directly measuring the generation performance of the aligned model, and demonstrate that using feedback from weak LLM can not only match but potentially exceed human feedback in alignment tasks. Our work suggests a promising direction for future research leveraging weak AI signals for alignment.
RELATED WORK
Large language model alignment. The primary objective of model alignment is to steer language models toward human-desired outputs. Numerous studies have leveraged human feedback to refine language models by human preferences (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Lee et al., 2021; Ouyang et al., 2022; Nakano et al., 2022; Glaese et al., 2022; Snell et al., 2023; Yuan et al., 2023; Song et al., 2024; Dong et al., 2023; Bai et al., 2022b; Lee et al., 2024; Munos et al., 2024; Hejna et al., 2024; Dai et al., 2024; Khanov et al., 2024). However, human preference data can be costly to collect and suffer from unreliability issues (Yeh et al., 2024), prompting researchers to explore Al-generated feedback for alignment purposes (Bai al., 2022a; Lee et al., 2023; Ding et al.,

---

Published as a conference paper at ICLR 2025
2023; Gilardi et al., 2023; Guo et al., 2024). Given the computational inefficiency of RLHF, recent shifts in focus favor closed form losses that directly utilize offline preferences, like Direct Preference Optimization (Rafailov et al., 2023) and related methodologies (Gheshlaghi Azar et al., 2024; Pal et al,, 2024; Liu et al., 2024b; Xiong et al., 2023; Tang et al., 2024; Yu et al., 2024; Ethayarajh et al., 2024; Zeng et al., 2024; Calandriello et al., 2024; Muldrew al., 2024; Ray Chowdhury et al., 2024; Liu et al., 2024a; Gao et al., 2024; Yang et al., 2024; Chakraborty et al., 2024; Zhao et al., 2023). Beyond algorithmic approaches, researchers recently have provided theoretical understandings of the learning dynamics (Im & Li, 2024b) and generalization guarantee (Im & Li, 2024a) of DPO. Nonetheless, both RLHF and offline preference- based methods presuppose access to high-quality human, which limits their scalability. Our research endeavor takes initial steps toward understanding the effectiveness of aligning language models under weak LLM feedback. We present a systematic evaluation and in-depth analysis that sheds light on the quality and feasibility of leveraging weak LLM feedback for alignment, and reveal novel empirical findings to the research community.
Large language model as judge. The use of LL--as-a- Judge prompting to evaluate language models has become a common practice (Dubois et al., 2023; Li et al., 2023; Bai et al., 2023; Saha et al., 2023). This approach is also used in collecting preference datasets for alignment. For example, Bai et al. (2022a) employs an LLM to assess responses, refine them, and then use the resulting data to train reward model known "RL from AI Feedback" (RLAIF). Similarly, Lee et al. (2023) demonstrated that employing a strong LLM for LLM-as-a-Judge prompting to create preference datasets yields performance nearly on par with traditional RLHF Yuan et al. (2024) explored the concept of LLM-as-a-Judge prompting to enable models to generate their own rewards during training. While many studies (Jiang et al., 2023b; Li et al., 2024b; Kim et al., 2023; Chen al., 2024; Ye et al., 2024) focus on leveraging advanced LLMs as judges for preference dataset collection, we investigate the effectiveness of feedback from weak L.LMs with significantly smaller capacity. We reveal new insight that task-specific weak LLM can provide more effective preference feedback than an ultra-large LLM (e.g., GPT-4) that relies solely on prompt engineering.
Weak-fo-strong generalization. Weak-to- to-stong generalization refers to the scenario where a weaker teacher model supervises a stronger student model. Unlike traditional teacher-student frame- works such as semi-supervised learning (Laine & Aila, 2017; Tarvainen & Valpola, 2017), domain adaptation (French et al., 2018; Chen et al., 2019), and knowledge distillation (Hinton et al., 2015; Beyer et al., 2022) which generally involve stronger teacher guiding weak student- other studies have explored employing comparably or even lesser-capable teachers to guide student model (Freund & Schapire, 1997; Furlanello al., 2018; Xie et al., 2020; Higuchi et al., 2020; Burns al., 2024; Green Larsen & Ritzert, 2022; Lang et al., 2024; Charikar et al., 2024). Recognizing the broad generalization capacities of large-scale pre-trained models, Burns et al. (2024) introduced the concept of weak-to-strong generalization for LLM, focusing on simpler discriminative tasks such as reward modeling rather than generative tasks such as alignment. A detailed discussion on the differences with Burns et al. (2024) is in Section 5. Different from previous works that explore the effectiveness of weak LLM under the standard supervised fine-tuning (Liu Alahi, 2024; et al., 2024a; Ji et al.,
2024; Hase et al., 2024; Sun et al., 2024; Bansal et al., 2024), we aim to delve deeper into the role of weak LLM could play for alignment. These tasks pose unique challenges due to inherently diverse output and the complexity of performance evaluation but hold promise for advancing superalignment in generative models (Puthumanaillam et al., 2024). Instead of adopting model interpolation for alignment without any additional training (Zheng et al., 2024; Zhou et al., 2024), we explore the three-stage framework for alignment with feedback from weak LLM.
CONCLUSION
In this paper, we systematically explore and evaluate the effectiveness of leveraging weak LLM feedback for alignment. Our findings reveal that using a weak LLM,, with a size as small as 125M, to provide preference feedback for alignment can match or even exceed the performance of using pure human feedback. Through our in-depth analyses, we shed light on the intricacies of alignment under weak LLM feedback and offer valuable insights into the quality discrepancies between human feedback vs. weak LLM feedback. This study not only highlights the feasibility of achieving high- quality alignment with less precise preference data but also suggests directions for future research to further refine alignment methods, ensuring AI systems remain beneficial and aligned with human intentions. We discuss the limitations and implications for future work in Section D.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Research is supported in part by the AFOSR Young Investigator Program under award number FA9550-23-1-0184, National Science Foundation (NSF) Award No. IIS-2237037 & IIS-2331669, and the UL Research Institutes through the Center for Advancing Safety of Machine Intelligence. The authors would like to thank Shawm Im and Wendi Li for the valuable feedback on the work.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Anthropic. Claude: Conversational ai assistant, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212. 08073, 2022b.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. Benchmarking foundation models with language-moddllas an-exxminer. In Seirth-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training IIm reasoners via compute-optimal sampling. arXiv preprint arXiv:2408.167. 2024.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In Computer Vision and Pattern Recognition, pp. 10925-10934, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 3943/4))324-45,, 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Eee Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rX1v:2303.12212 2023.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, llya Sutskever, and Jeffrey Wu. Weak- to-strong generalization: Eliciting strong capabilities with weak supervision. In International Conference Machine Learning, pp. 1971, 2024.
Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, al. Human alignment of large language models through online preference optimisation. In Proceedings of the 41st International Conference on Machine Learning, pp. 5409-5435, 2024.
Souradip Chakraborty et al. Maxmin-rlht. Alignment with diverse human preferences. Forty-first International Conference Machine Learning, 2024.

---

Published as a conference paper at ICLR 2025
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to-strong generalization. Advances in Neural Information Processing Systems, 2024.
Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Computer Vision and Pattern Recognition, pp. 627-636, 2019.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. AlpaCiauus: Training better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. International Conference on Learning Representations, 2024.
Josef Dai al. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.
Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. Is GPT-3 good data annotator? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023.
Hanze Dong, Wei Xiong. Deepanshu Goyal, Yihan Zhang, et al. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint 2r3: 14387, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, al. Model alignment prospect theoretic optimization. Proceedings of the 41st International Conference on Machine Learning, pp. 12634-12651, 2024.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In International Conference on Learning Representations, 2018.
Yoav Freund and Robert E Schapire. A decision decision-theoreeicc generalization on-line learning and an application to boosting. Journal of computer and system sciences, 55(1) 1999999, 1997.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616, 2018.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866, 2023.
Songyang Gao et al. Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback. In Forty-first International Conference on Machine Learning, 2024.
Mohammad Gheshlaghi Azar et al. A general theoretical paradigm understand learning from human preferences. AISTATS, 2024.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 2023.
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment dialogue agents via targeted human judgements. arXiv preprint arXiv: 14375, 2022.
12

---

Published as a conference paper at ICLR 2025
Kasper Green Larsen and Martin Ritzert. Optimal weak strong learning. Advances in Neural Information Processing Systems, 35:32830 35:32830-32841, 2022.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04772, 2024.
Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unrcasonable effectiveness of casy training data for hard tasks. arXiv preprint arXiv:2401.06755 2024.
Joey Hejna et al. Contrastive preference learning: Learning from human feedback without re- inforcement learning. In The Twelfth International Conference on Learning Representations. 2024.
Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi. Mask ctc: Non-autoregressive end-to-end ast with etc and mask predict. Annual Conference of the International Speech Communication Association, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:150..02533 2015.
Shawn Im and Yixuan Li. On the gencralization of preference learning with dpo. arXiv preprint 2408.03459, 2024a.
Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. In International Conference on Machine Learning, 2024b.
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint wwXXi:2442.0221000 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06823 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm- Alm-dlender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.0056/) 2023b.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment seward-guide search. In Proceedings of the International Conference on Learning Representations, 2024.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, al. Prometheus: Inducing ine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08499, 2023.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. International Conference Learning Representations, 2017.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint arXiv:2403. 13787, 2024.
Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to-strong generalization. Advances in Neural Information Processing Systems, 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 2023.
Harrison Lee et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first International Conference Machine Learning, 2024.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, 2021.
13

---

Published as a conference paper at ICLR 2025
Ming Li, Yong Zhang. Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strone toront data filtering for fast cnoruction-tuninng arXiv preprint NMXiv:2402.005300, 2024a.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024b.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloiing models. https: / / thub. com/ tatsu-laal hatpaaappaaaaaaaaa 2023.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, al. Decoding-time realignment of language models. In Proceedings of the 41st International Conference on Machine Learning, pp. 31015-31031, 2024a.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. International Conference on Learning Representations, 2024b.
Yuejiang Liu and Alexandre Alahi. Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture of experts. arXiv preprint arXiv:2402.15505, 2024.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Proceedings of the 41st International Conference on Machine Learning, pp. 36577-36590, 2024.
Remi Munos et al. Nash learning from human feedback. In Forty-first International Conference on Machine Learning, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint
2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Arka Pal et al. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.. 2022222222.2..... 2024.
Gokul Puthumanaillam, Manav Vora, Pranay Thangeda, and Melkior Ornik. A moral imperative: The need for continual superalignment of large language models. arXiv preprint arXiv:2403. 14683, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. In Proceedings of the 41st International Conference Machine Learning, pp. 42258-422777 2024.
Swarnadeep Saha, Omer Levy. Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-merge improves large language model evaluation and generation. arXiv preprint arXiv:2310.15123. 2023.
Charlie Victor Snell, Ilya Kostrikov, Yi Su, Shemyy Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. In The Eleventh International Conference Learning Representations, 2023.
14

---

Published as a conference paper at ICLR 2025
Feifan Song, Bowen Yu, Minghao Li. Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 18990-18998, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 13:3008-3021, 2020.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to- -trd generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.0Pm 09472, 2024.
Yunhao Tang et al. Generalized preference optimization: unified approach to offline alignment. ICML, 2024.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 2017.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre,, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini rescarch and technology. arXiv preprint arXiv:2403.08295 2024.
Ankur P Parikh Thibault Sellam, Dipanjan Das. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099: 09288, 2023.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint 1rXiv:2312./1456, 2023.
Rui Yang et al. Rewards-in-comee Multi-objective alignment of foundation models with dynamic preference adjustment. Forty-first International Conference on Machine Learning. 2024.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo,
James Thorne, Juho Kim, and Minjoon Seo. FLAASK: Fine-grained language model evaluation based on alignment skill sets. In International Conference on Learning Representations, 2024.
Min-Hsuan Yeh, Leitian Tao, Jeffrey Wang. Xuefeng Du, and Yixuan Li. How reliable is human feedback for aligning large language models? arXiv preprint arXhv:2410.01957, 2024.
Meng Yu al. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.147:5 14734, 2024.
Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05300, 2023.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, et al. Token-level direct preference optimization. In Proceedings of the 4|st International Conference Machine Learning, pp. 58348-58363 2024.
15

---

Published as a conference paper at ICLR 2025
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- trairrane transformer language models. arXiv preprint arXiv:2205.01066 2022.
Yao Zhao al. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.1042) 2023.
Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404. 16792, 2024.
Zhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong, Chao Yang, and Yu Qiao. Weak-to-strong search: Align large language models via searching over small language models. Advances in Neural Information Processing Systems, 2024.
Daniel M Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0859) 08593, 2019.
16

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAIL
A.1 TRAINING DETAILS
Software and hardware. Our experiments are conducted on servers equipped with NVIDIA A GPUs, with 80 GB of VRAM. The operating system used is Ubuntu 22.04.2 LTS, supported by NVIDIA CUDA Toolkit version 12.1 and cuDNN version 8.9. All experimental implementations are carried out in Python version 3.11.4, utilizing the PyTorch framework version 1.12.1.
Training hyper-parameters. Based on TRL 2 models with the same hyperparameters as shown in Table 7 and Table 8.
we deploy the training of both teacher and student
Reddit TL;DR dataset preprocessing details. For the Reddit TL.DD dataset, we preprocess the data by filtering out samples with fewer than 512 tokens. This results in 92,000 training samples and 8,000 test samples. We divide the training data into two disjoint sets of 46,000 samples each. The first set serves the labeled dataset, denoted as Dj, while the second set, referred to as the unlabeled dataset, D., disregards the preference labels.
A.2 EVALUATION DETAILS
Table 6: Sample prompt for the GPT-4 evaluation. Text highlighted in orange represents the question, while text in blue represents the responses under comparison.
System prompt
[System]
You are a helpful, harmless, and precise assistant for checking the quality of the answer. We would like to request your Al assistants in response user question displayed above.
feedback on performance
Please rate the harmlessness, helpfulness, and level detail of their responaes. Your evaluation should consider factors such as the helpfulness, harmlessness, relevance, accuracy, depth, creativity, and level of detail of the response. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please first output single line containing only two values indicating the respectively. The two scores are separated by space. In the subsequent line, please provide a comprehensive explanation your evaluation, avoiding any potential presented does not affect your judgment.
for Assistant and
ensuring that the order in which the responses were
User prompt
[Question]
#idHummn: What are some symptoms of caffeine withdrawel? #ditAssistant::
[The Start of Assistant 1's Answer]
When caffeine is removed from someone who has habitually consumed it, the body has natural reaction to produce more adrenaline, order to reduce the need caffeine. This increased adrenaline production causes many different symploms, ranging from jitteriness and restlessness, to lethargy and drowsiness. [The End of Assistant 1's Answer]
[The Start Assistant Answer]
You might feel an overall sense of mental tiredness, decreased ability to concentrate, and problems sleeping, You may also experience headaches and a sense of jitteriness or edginess. There are also some physical symptoms that can appear, such muscle pain and vomiling. [The End of Assistant 2's Answer]
GPT-4 evaluation details. Table 6 presents the prompts and responses in our GPT-4 evaluation, adopted from (Khanov et al., 2024). Each GPT-4 request comprises both a system and a user prompt. The system prompt delineates the proxy's attributes and its specific task, while the user prompt poses a question and provides responses from the two methods.
Gold reward model. For the evaluation of the HH-RLHF, we leverage the ooite--f-the-ann gold reward model Ray2333/ sewarddmoddel. st carra-7B-Bi -7B-insttruct- from the RewardBench leaderboard (Lambert et al., 2024), which fine-tunes on 1i--bleenddrttbiiffeee
f ed-Feedbaact dataset. We also verify our results consistently hold
17

---

Published as a conference paper at ICLR 2025
under different choices of gold reward models in Appendix C. For Reddit TL;DR, take the state- of-the-art gold reward model OpenAssi sitsssstttttt/...e /ttatttteaarddtteeeeeee hat.:aard-mmeel--deert-t-------- on this dataset from the RewardBench leaderboard (Lambert al., 2024) for evaluation.
Hyper-parameters for model generation. To evaluate the response of the model, we adopted the temperature as 0.7 and the max tokens as 256.
Table 7: Summary of training hyperparameters for supervised fine-tuning and direct preference optimization for OPT-family models for HH-RLHE.
Parameters
Value
Number of epochs
1
Learning rate
1 x 10-5
Batch size
32
Supervised fine-tuning
Gradient accumulation steps
1
Maximum sequence length
512
DeepSpeed Zero stage
2
LoRA rank
0
Number of epochs
1
Learning rate
5 x 10-5
3
0.1
Batch size
16
Direct preference optimization
Gradient accumulation steps Maximum sequence length
1
512
DeepSpeed Zero stage
2
LoRA rank
8
Table 8: Summary of training hyperparameters for supervised fine-tuning and direct preference optimization for OPT-family models on Reddit TL;DR.
Parameters
Value
Number of epochs
1
Learning rate
1 x 10-5
Batch size
32
Supervised fine-tuning
Gradient accumulation steps
1
Maximum sequence length
512
DeepSpeed Zero stage
2
LoRA rank
0
Number epochs
1
Learning rate
5 10-6
8
0.5
Batch size
16
Direct preference optimization
Gradient accumulation steps Maximum sequence length
1
512
DeepSpeed Zero stage
2
LoRA rank
8
18

---

Published as a conference paper at ICLR 2025
B ADDITIONAL QUALITATIVE EXAMPLES
In Table 9, provide additional qualitative examples from weak where the chosen response by weak LLM mismatches that the human annotators. Despite contradicting the human preference, the gold reward of the response chosen by the weak LLM can be higher, i.e., rgola( De) Tgela (2; Úr) This indicates that the preferences of the weak LLM can align with the gold reward model, even they differ from human labels.
Table 9: Comparative examples of the chosen and rejected responses in D mismmahh
weak
Example
Prompt
Human: How does pacemaker help the function of the heart?
Assistant:
Chosen response by weak LLM (rejected by human annotators) pacemaker medical device implanted the heart to pump and circulate blood more efficienily and effectively. Pacemakers are generally implanted with catheters by a practitioner surgery. Pacemaler systems usually consist of pulse generator and heart (lead to the right atrium, right ventricle, right-side of the chest, or the right coronary sinus), and the other end which connected the pacemaker. Pacemakers fallure, or at risk of permanent damage from their heart not functioning properly. (gold reward: 1 1.99)
chest
stimulates the heart to
normal rhythm. helping
cardiac catheterization or angiography procedure, or rare cases with open chest
leads, end which sutured to the
can used lifesaving measure patients with heart
Rejected response by weak LLM (chosen by human annotators) A pacemaker is a small electronic device implanted under the skin with electrode attached to heart help control the rate and rhythm of the hearlbeat. The pacemaker senses when heartbeat slow irregular and sends electrical impulses to the heart at a pre defined safe and healthy rate, normalizing the heart beat. (gold reward: 11.76)
Example
Prompt
Human:
Why does someones weight affect their tolerance to alcohol?
Assistant:
Chosen response by weak LLM (rejected by annotators)
Studies suggest that people with a higher body weight may have lower tolerance for alcohol, but this is not well understood. may be due differences body composition or other factors that influence how alcohol is metabolized and how it affects the central nervous system. (gold reward: 8.05)
Rejected response by weak LLM (chosen by human annotators) Well, alcohol is stored as fat in the human body, and so one's tolerance to alcohol is related to one's fat stores. (gold
6.66)
Example
Prompt
How does grid work for powering country's electric grid?
Assistant:
Chosen response by weak LLM (rejected by human annotators)
we're talking about electricity grids, right? One way to describe what's happening to say that grid acts as network for power distribution, where energy flows from various energy sources various consumers. And it's like a huge complex network of pipes and nodes. But this case, the 'nodes' are basically power plants that create and distribute electricity, and the "ipee'' the electrical lines which transport power from sources to the nodes./gold reward: 2.38)
Rejected response by weak LLM (chosen by human annotators)
There's s a bunch different ways to generate electricity. Most time, they boil water to produce steam, which spins turbine, which spins generator, which generates electricity. But there are other ways to do To start, need undersland what you mean by 'smart grid'. Are you talking about a grid that's connected to smarl city, or smarl grid in of power lines, something else? (gold reward: 1.07)
C ADDITIONAL EXPERIMENTS
Evaluation with more gold reward models. To verify the reliability of our conclusions, employ different gold reward models to evaluate the quality of model output. Specifically, we leverage another two competitive gold reward models weqweasdas/ Sateaaaddas/MMMiissss /RM-MMsttraaa77P and OpenAssistant//
from the RewardBench leader- board (Lambert al., 2024). As shown in Figure using these two alternative gold reward models,
19

---

Published as a conference paper at ICLR 2025
can still observe that alignment with weak LLM feedback outperform human feedback. This confirms the reliability of our findings, and its insensitivity to the choice gold reward model.
(a) reward-mode wwwarddmmodll--eeeeeta-----a--a
(b) RM-Mistral-78
weak LLM feedback
weak LLM feedback
hum feodback
0.79 0.76
human feedback
11.01
11.24
0.73
10.2
9.63
9.68
Gold roward
Gold reward
0.54
0.47
0.3
4-
0.2
2-
0.0
OPT-1.36
OPT-2.76 Student model size
OPT-6.7b
OPT-136
OPT-1.36
OPT-2.76
OPT-6.7b
OPT-136
Student model size
Figure 5: (a) Evaluation with the gold reward model Evaluation with the gold reward model RM- RM-Mistrrl-77
(b)
Results on additional weak LLM feedback. To validate our conclusion that weak LLM provides feedback comparable human feedback, extend our analysis beyond the specific case of OPT- 125M. We incorporated GPT-Neo-12555 (Gao et al., 2020) and Pythia- 160M (Biderman et al., 2023) as additional weak LLM provide feedback. We use OPT-1.3B and Mistral-7B as student models to be aligned. Following the alignment with weak LLM feedback process outlined in Section 3. we assess the generalizability of our conclusion. from weak LLMs GPT-Neo- 125M and Pythia- 160M achieved performance comparable to the model aligned with human feedback. This demonstrates that our conclusion is not dependent on a single weak model, but rather applies more broadly across different weak LLMs.
shown Table 10, models aligned with feedback
Table 10: Results on using different weak LLM supervisors: OPT-125M, GPT-Neo- 125M and Aythia-160M.
Human feedback
OPT-125M
GPT-Neo-125M
Pythia-160M
OPT-1.3B
4.63
4.84
4.55
4.87
Mistral-7B
7.19
7.90
7.27
8.01
Correlation of generation quality. In Figure 6, we investigate whether the generation quality correlates between models using weak L.MM feedback vs. human feedhack. In particular, for each test input prompt, we measure the sentence-level similarity between the model's generation ý (using either Ty or T and the chosen response Vu by human annotators from HH-RLHF. Thibault Sellam (2020) proposed a cosine similarity based on the BLUERT embedding. These similarities are denoted as sim,z (ú, Ve) for the model aligned with weak LLM feedback and sim,; (9, ye) for the model aligned with human feedback, represented on the r-axis and y-xis respectively. We observe moderate correlation between the two across all test samples from HH-RLHF.
Additionally, we examine the correlation between the model Tá aligned with weak LLM feedback and the weak LLM itself T" on the right of Figure 6. This analysis helps reveal whether aligning a model under weak feedback could risk imitating the errors inherent that the correlation between & and nh (with R22 = 0.4888) is relatively weaker compared to that between गे and n (with 1t2 0.5789). These results suggest that the model z not only aligns more closely with n but also effectively extrapolates beyond imitating the weaker teacher.
weak L.M.. We observe
Ablation on the impact of SFT. Learning from preference data typically begins by fine-tuning a pre-trained language model with supervised learning on high-quality data for the downstream tasks of interest. We further explore the impact of SFT when aligning the target policy model (c.f. Equation 6). We utilize the OPT- 125M as the weak LLM to provide feedback and the OPT-1.3B as the student model, conducting with weak LLM feedback and the model тं aligned with human feedback, we compare the following two settings: (1) DPO (w/o SFT):: Directly using the pre-trained model the reference model or
experiments
the HH-RLHF dataset. For the model тद aligned
20

---

Published as a conference paper at ICLR 2025
Agreement on ng and my
Agreement on ng and I
1.0
0.8
0.8
0.7
0.6
12
R:: 0.5799
R7: 0.4088
0.2
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Figure 6: Left: Agreement between models using weak LLM feedback (#²) vs. human feedback (mf)). Right: Agreement between the model (Tτ) aligned with weak LLM feedback the weak LLM itself (mw).
initialization. (2) DPO (w SFT): Use fine-tuned model ma as a reference model for DPO training. Compared to directly using the untuned base model reference model, performing SFT enhances the model's ability generate desired responses to questions.
Table 11:
Ablation on the impact of supervised fine-tuning (SFT) as initialization for DPO. We report the average gold reward. Numbers in brackets denote improvement relative to the wre-trained model (OPT-1.3B) without any fine-tuning.
Method
Weak LLM feedback
Human feedback
DPO (wlo SFT)
3.83 (+1.67)
3.77 (+1.55)
DPO (w SFT)
4.84 (42.67)
4.63 (+2.41)
Ablation on the KI coefficient (3) for weak LLM feedback for alignment. The parameter in Equation functions the KL coefficient during DPO training, with higher values indicating more stringent regularization. We employ OPT-125M as the weak LLM to provide feedback and OPT-1.3B the student model, performing alignment using varying 3 values: {0.05, 0.1, 0.2, 0.3. 0.5) on the HH-RLHF dataset. As shown in the left of Figure 7, the gold reward for the model alignment with weak LLM feedback (T²)) is high under relatively small / values such 0.05 and 0. and starts decline as 3 increases. similar trend is observed for the model trained with human feedback.
Ablation on 8 for student model training
weak LLM feedback
feedback
4.81
4.50
4.63
4.55
4.44
4.17
4.08
4.04
Gold reward
0.1
0.5
Figure 7: Ablation the KL Coefficient (9) for alignment with weak LLM feedback and human feedback.
Dataset division with different random seeds. We randomly partition the entire training set into labeled dataset, denoted as D,. and an unlabeled dataset, D,, using a random seed of 22. To ensure that our conclusions insensitive to the data division, also utilized five additional random seeds
21

---

Published as a conference paper at ICLR 2025
to perform the experiments. Same main experiment, we employ OPT- 125M the weak LLM and OPT-1.3B as the student model on the HH-RLHF dataset. Across random runs, the average gold rewards for the Th and FH are 4.81 and 4.67, respectively, with variances of 0.12 and 0.14. These results affirm that our conclusions are statistically stable under different dataset splits.
Understanding the different impact between weak LLM feedback vs, random noise. Our investigation in Section 4 suggests that weak LLM feedback contains more systematic noise (e.g.. errors primarily occur between choices that are subtly different). In this ablation, we contrast performing alignment with feedback from weak LLM by using the feedback containing random noise. Adopting OPT- 1PT- as the weak L.M and OPT-1.3B as the student model, we design three controlled settings for this ablation:
Dmaach
random" Randomly sample 60.6% of the samples from Da, and assign human feedback, i.e.,
(r= ye) > Tw (Z, V+)). This set has the same size as our weakly supervised set weak
Diiismachh Use the remainder 39, 39.4% samples from D.. and assign preference labels opposite the human feedback, i.e., (Tm (I, Mo) < Tw (1, Ur
Prusdom: The union of Dmatch and Dinismatchh
As demonstrated in Table 12, the student model trained with the match set Diaatc outperforms the model trained with Prandom by 0.66 in terms of gold reward. This suggests that removing random noise in preference datasets can markedly enhance aligmment performance. Conversely, the performance of the model trained with Dinismath exhibits gold reward that significantly lower (1.53), illustrating the negative effect of random noise on model alignment. These findings highlight the difference between random noise and the "noise" from weak LLM feedback.
Table 12:
Effect of purifying random noise. Numbers in brackets denote improvement relative to the pre- veeitrained student model without any fine- tuning.
% matching human feedback
Gold reward of mğ
D Prnddom
60.6%
3.91 (+1.69)
Dinach
100%
4.57 (+2.35)
randoo
minsmanch
0%
1.53 (-0.69)
random
Exploring the impact of DPO-trained weak LLMs vs. traditional reward models. To maintain a consistent and unified framework for training both the teacher and student models, we adopt DPO instead of traditional reward modeling for teacher-model training. Furthermore, since DPO inherently formulated as a Bradley-Terry loss, it naturally functions as an implicit reward model. When the weak LLM's role is limited to generating reward signals, it is also feasible to replace with traditional Bradley-Terre reward model. To explore this alternative, we conducted additional experiments comparing DPO-trained weak LLMs with traditional reward models. The results show that both approaches achieve comparable performance, with both surpassing human labelers.
Table 13: Comparison of gold reward between DPO-trained weak L.IMss and traditional reward models.
Feedback Source
Gold Reward
Weak L.LM (with DPO)
4.84
Weak LJL (with reward modeling)
4.86
Human labeler
4.63
22

---

Published as a conference paper at ICLR 2025
D LIMITATIONS AND FUTURE WORK
Limitations. Our research presents an in-depth investigation of alignment with the feedback from weak LLM. We delve deeply into the reasons underlying this phenomenon. However, our study without limitations, which we aim to address in subsequent research. Our focus has predominanty been on empirical experiments, leaving the theoretical underpinnings of our findings less explored. These areas present opportunities for further exploration and development.
Implications for future alignment research. As we look toward the future of Al alignment, several recommendations emerge from our study that could further refine the practice and enhance the reliability of aligmment methodologies. First, integrating hybrid feedback systems that combine human insights with Al-generated feedback could leverage the strengths of both, minimizing the limitations inherent in each approach. Secondly, it is crucial to develop more sophisticated metrics for evaluating alignment quality that go beyond traditional accuracy metrics, capture the nuanced understanding and generative capabilities required real-world applications. Finally. exploring the ethical implications of Al-generated feedback and ensuring that these systems adhere ethical guidelines is vital. By addressing these areas, the field can move towards more effective, efficient, and ethically responsible Al alignment strategies that are capable of supporting the safe integration of Al systems into society.
23


Reviews:
[{"review_id": "DQ640QzSs7", "reviewer": "Reviewer_1dzH", "summary": "This paper concentrates on the critical research topic of large language model (LLM) alignment, especially related to super-alignment. Considering existing mainstream alignment approaches rely on paired human preference data, which highly relies on human or powerful LLMs for annotation, using a weak LLM to provide alignment feedback becomes a promising alternative. This paper aims to bridge that gap of lacking a systematic evaluation and understanding of weak LLM’s capability of generating feedback of alignment. The authors first formalize a workflow to conduct semi-supervised alignment training of a model and conduct evaluation. Based on this uniformed experimental setting, the paper assesses the effectiveness of models of various scales in providing alignment feedback and finds that weaker LLMs can indeed play an effective role in alignment tasks. Additionally, an in-depth analysis is conducted on the quality of feedback provided by weaker LLMs.", "questions": "- Why the weaker LLM to provide alignment feedback is referred to as a task-specific model? It is trained with the HH-RLHF dataset, which is a general domain dataset for LLM alignment.\n- What does the “pre-trained strong model” mentioned in Line 371 specifically refer to? Is is the LLM that only undergone pre-training without fine-tuning or alignment, or is it the trained with positive examples?\n- I have concern about the result that GPT-4 underperforms compared to smaller models. Could you please provide more reasonable explanations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The research topic in this paper is critical and popular. Given the high costs associated with human and large model annotations, exploring the potential of weaker LLMs in providing alignment feedback is a timely and necessary endeavor.\n- This paper is well-written and easy-to-follow, with an introduction to associated alignment background techniques.\n- A dedicated discussion section highlights the differences between this work and the most relevant prior studies.", "weaknesses": "- More comprehensive experiments are required to validate the reliability of the primary conclusions in this paper, including but not limited to the following aspects.\n    + The most relevant baselines with the idea of RLAIF. Beyond that the original RLAIF methods rely on large LLMs to provide feedback, it is crucial to delineate differences between the proposed training pipeline and RLAIF. Additionally, a comparison with RLAIF employing weaker LLMs as the reward model would strengthen the findings.\n    + Human annotation data of different quality levels. Previous studies [1,2] have indicated that the HH-RLHF dataset may contain noise in its human preference labels. Comparing with higher-quality human data, such as a revised version of HH-RLHF, would enhance the robustness of the results.\n    + LLMs with larger sizes.\n    + GPT-4 with more advanced prompts or instructions to provide alignment feedback.\n\n[1] Wang, Binghai, et al. \"Secrets of rlhf in large language models part ii: Reward modeling.\" arXiv preprint arXiv:2401.06080 (2024).\n\n[2] Liu, Yan, et al. \"Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment.\" arXiv preprint arXiv:2409.19024 (2024).\n\n- The issue of overfitting to a specific dataset in smaller models is not discussed. Since small LLMs are prone to overfitting to specific datasets but LLMs alignment target diverse domains, you should consider the generalization of the weaker LLM to provide alignment across domains. If it shows very weak generalization, it may need annotation data in each domain, which also requires a large human cost.\n- More diverse and more reliable evaluation metrics should be considered.\n- The actionable insights derived from the in-depth analysis are sparse. The main finding is that issues persist in the original annotated data, and both weaker LLMs and large models like GPT struggle to handle noisy data with high confidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ckn5WAEdkb", "reviewer": "Reviewer_oS4F", "summary": "This paper introduces an alignment approach using weak large language models (LLMs) that balances cost and efficiency. The authors find that weak LLMs can generate feedback comparable to or even better than human feedback and stronger LLMs and propose the method to employ a semi-supervised framework where a weak LLM, trained on a small labeled dataset, provides preference feedback for a larger unlabeled dataset, reducing reliance on human annotation.", "questions": "1. Is the consistency of weak LLM feedback assessed across multiple prompts or model runs?\n\n2. How do the authors ensure that the gold reward model and GPT-4 are reliable in evaluation? Are other potential evaluation metrics considered?\n\n3. The paper listed many LLM alignment methods in the related work. Did the authors compare their method with these alignment techniques?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a cost-effective alignment framework that reduces computational costs and reliance on extensive human feedback,  using weak LLMs that have significantly fewer parameters than high-capacity models.\n\n2. The study demonstrates that weak LLMs can perform as well as, and in some cases better than, human annotators in generating preference feedback for alignment through extensive experiments. For example, by demonstrating similar alignment success across different model families and sizes, the study provides a strong case for the generalizability of the weak LLM approach, suggesting that high alignment quality can be achieved without the latest or most powerful models.", "weaknesses": "1. The paper lacks a thorough analysis of feedback consistency over repeated evaluations. For example, when weak LLM and human preferences conflict, the study does not adequately explore the stability of weak LLM feedback. A quantitative analysis of consistency across multiple runs or various weak models would be helpful to assess whether the weak LLM’s feedback is repeatable and reliable. \n\n2. The evaluation metrics are only based on 2 models, the gold reward model and GPT-4. If the models are biased, the evaluation given by them would be unreliable and it could lead to skewed assessments that do not accurately reflect the quality of weak LLM alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTjdcIYIjP", "reviewer": "Reviewer_ntey", "summary": "The paper explores the potential of using weak large language models (LLMs) for AI alignment. It proposes a new framework that leverages weak LLMs to provide feedback, thereby reducing reliance on costly human labor and computational resources. Experimental results show that even weak LLMs with as few as 125 million parameters can offer feedback comparable to human feedback. The paper also analyzes the quality of the feedback from weak LLMs and finds that it can surpass human judgment in some cases, although there is uncertainty in fine differentiation. The study offers a new perspective for AI alignment and demonstrates that weak LLM feedback is a promising, scalable and sustainable solution.", "questions": "1.See weakness 2.\n\n2.See weakness 3. Can this weak-to-strong approach be evaluated for more complex tasks? Like a math task or a code task?\n\n3.In Contribution 2, the authors mentioned that using weak LLMs to provide preference labels for alignment can match or even surpass the performance of using full human feedback with the same training data scale. Why is it that training on data labeled by weak LLMs on unlabeled data can outperform the training set labeled by humans? Is it because the human-labeled training set contains some incorrect labels and its quality still needs improvement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1.The paper introduces a framework that leverages weak LLMs for alignment, which offers a promising alternative to existing methods that rely of either extensive human labor or expensive computational resources.\n\n2.The paper conducts comprehensive evaluation of weak LLM feedback across various model scales and families. The result consistently demonstrate the effectiveness of this approach, highlighting its potential for practical application.\n\n3.The paper is well-organized, with clear sections and logical flow. The authors effectively communicate the research objectives, methodology, and findings.\n\n4.The research has significant practical implications for the development and deployment of aligned LLMs. Utilizing weak LLM feedback can reduce costs and improve efficiency, making alignment more scalable and accessible.", "weaknesses": "1.In this paper， the authors propose a framework that utilizes weak LLMs for alignment, striving to strike a balance between RLHF and RLAIF, it appears that the weak-to-strong approach and methodology in the paper are still quite similar to Burns et al. (2024). Burns et al. (2024)’s study focus on reward modeling and binary classification, where the outputs in this paper are either scalar or categorical labels. Your study is closely tied to alignment and involves a more complex output space. However, it seems that the two are consistent in their methodological approach. Therefore, I think that there are still some limitations in terms of novelty. I hope you can explain more about the difference between the two works.\n\n2.The paper points out that even advanced LLMs like GPT-4 may show feedback inconsistency when the quality difference between two responses is subtle. In the case where it is uncertain whether gold RM is stronger than GPT-4, when using gold RM for evaluation, is it also possible to make incorrect judgments about two responses with similar quality, leading to bias in the evaluation？Additionally，When using GPT-4 for evaluation, although the prompt already mentions and ensures that the order of the responses should not affect its judgment, in actual experiments, it is difficult to avoid the model's preference caused by the order. A fairer approach would be to swap the positions of the two responses and conduct the experiment again. I believe that conducting the experiment twice with different positions would result in a more accurate evaluation.\n\n3.While the paper demonstrates the effectiveness of weak LLM feedback across alignment task, it would be beneficial to investigate its applicability to more complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ljKoAnaX5W", "reviewer": "Reviewer_ZoiK", "summary": "This paper investigates the use of a weak supervised LLM to annotate preference data, as a middle ground between labor-intensive, costly human annotations and computationally expensive annotation based on high-capacity LLMs such as GPT-4. More specifically, the weak LLM (with approx. 100M parameters) is fine-tuned with DPO on a set of human labeled preference data and then its associated implicit reward model is applied to an unlabeled set to predict preferences. The resulting completed preference dataset is ultimately used to align a student LLM, whose performance enables measuring the quality of the annotations provided by the weak LLM.\n\nThe proposed weak LLM-based annotator is evaluated through multiple experiments where it is compared against both human annotators and stronger LLM annotators. It also includes experiments with weak LLMs and students models from different model families. Overall, the results show that using weak LLM-based annotations lead to similar or better performance compared to using human annotations or prompted high-capacity LLMs.", "questions": "- In the second paragraph of the introduction (lines 40-50), expliciting the fact that what is studied is a weaker _supervised_ LLM might help the reader understand the middle-ground nature of this approach. Otherwise it may not be fully clear why weaker LLMs are positioned in between human annotators and high-capacity LLM annotators.\n- In Figure 1, it is a bit unintuitive to show A < B in all the annotations, as it seems that answer A is a better (at least more helpful) answer than B in the given example. This example is mentioned later in Table 3, as a case where human annotation contradicts the gold reward model.  But using this (potentially incorrect) human annotation in Figure 1 when describing the approach might be confusing for the reader.\n- The construction of $\\mathcal{D}_{weak}$ in Section 3.1 is done by first learning a generation model through DPO and then use the implicit reward model induced by DPO to annotate the unlabeled dataset $\\mathcal{D}_u$. It seems unnecessary to train a generation model as only the reward model is needed for the annotation. Given this, it is then unclear why DPO was used in this step instead of simply training a traditional reward model by following the standard RLHF protocol (but without the unneeded PPO stage). The argument given lines 139-140 that training with RLHF requires the use of multiple models is only valid for the PPO stage (when there are both the LLM policy and the reward model to handle), but not in the reward modeling stage which only consists of an LLM augmented with a regression head. It would have been interesting to study the impact of the DPO reward model vs the RLHF reward model.\n- In Section 3.2, one of the metric introduced is the gold reward. However it is not entirely clear why a large auxiliary reward model can be considered as \"gold\". Does it just have to be a \"recognized\" reward model, or are there any requirements on the training data used for this reward model?\n- For the GPT-4 win-rate metric, it does not seem necessary to ask for a numeric rating to be assigned to the two compared responses. Would it not be sufficient to just ask GPT-4 to identify the better response out of the two? This seems to be a more standard approach (see, e.g., Prometheus 2: https://arxiv.org/abs/2405.01535).\n- What is the ratio $|\\mathcal{D}_{l}|/|\\mathcal{D}|$ used for the main experiments in Section 3.3? The ablations in Section 3.4 do study the results when this ratio is varied, but I could not find the default ratio that is used in other experiments.\n- The x axis label of Figure 7 in Appendix C seems to be incorrect: it should be $\\beta$ instead of \"student model size\".", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is clear, polished and reads very well.\n- The experiments are comprehensive and convincing.\n- The proposed idea of using a weak supervised LLM as annotator is likely to have a significant impact on future LLM alignment practices.", "weaknesses": "- The choice of using DPO fine-tuning to obtain a weak LLM-based annotator is not fully intuitive and lacks motivations, as only a reward model is needed for the preference annotation -- a model with generation capabilities is unnecessary for this (more details on this point in the \"Questions\" section).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X8fUcGe1qL", "reviewer": "Reviewer_2XYr", "summary": "This work proposes to investigate whether \"weak\" language models could serve as a reward model/preference generator for alignment. Experiments demonstrate that they can to varying extents and match some of the benefits of HF/AIF.", "questions": "please see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ ailgnment with synthetic feedback/preference is an important research question\n+ the experiments are comprehensive", "weaknesses": "- On the high-level framing, why is \"weak LLM as feedback\" positioned as the \"middle ground\" (line 42) between human feedback and AI/LM feedback? You could say that weak LLM feedback is still AI feedback, just with a smaller/weaker model, and not necessarily offers something strikingly new that make it \"middle ground\" in the \"spectrum\".\n\n- By \"weak LLM\" the authors actually mean \"small LLM\": for example, \"while a model like GPT-4 might have trillions of parameters,\na weak LLM might only have hundreds of millions or even fewer\" (lines 44-45). The experiments focus on size too by employing OPT models, which are small compared to today's standards. However, \"small LLM\" for feedback is not necessarily \"weak\": if you take a small model and train it on large quantities of quality human preference data, it might become a moderate reward model. It would be great to disentangle \"weak\" and \"small\" in the narrative and also in the experiments.\n\n- Section 2 is a bit trivial and takes up too much space. These math primers on DPO/PPO were not particularly helpful in later sections too. It would be great to assume readers of an \"alignment\" paper are already familiar with the basics, move them to the appendix, and use the space for analysis/results.\n\n- I see that there is some investigation into the quality of preferences by the \"weak\" OPT model. Maybe something like RewardBench might offer a more rigorous quantitive evaluation.\n\n- I would argue that things like self-rewarding language models [1] did things similar to \"feedback from a weak model\", albeit not as weak as OPT. Starting from a \"weak-ish\" models that are not good in both generating and evaluating, they show it is possible to jointly improve both. I wonder if the uniqueness of this paper's contribution might be weakened a bit by works like this.\n\n- Ultimately, some might say that this work is a fine analysis paper on a niche technical design choice and does not offer something greater: taking RLAIF pipelines, changing the \"AI\" model part to a smaller model, and run some evaluation. Please highlight additional contribution if any.\n\n[1] Yuan, Weizhe, et al. \"Self-Rewarding Language Models.\" ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)