Your Weak LLM is Secretly a Strong Teacher for Alignment

## Abstract

Abstract The burgeoning capabilities of large language models (LLMs) have underscored the need for alignment to ensure these models act in accordance with human values and intentions. Existing alignment frameworks present constraints either in the form of expensive human effort or high computational costs. This paper explores a promising middle ground, where we employ a weak LLM that is significantly less resource-intensive than top-tier models, yet offers more automation than purely human feedback.
We present a systematic study to evaluate and understand weak LLM’s ability to generate feedback for alignment. Our empirical findings demonstrate that weak LLMs can provide feedback that rivals or even exceeds that of fully human-annotated data. Our study indicates a minimized impact of model size on feedback efficacy, shedding light on a scalable and sustainable alignment strategy. To deepen our understanding of alignment under weak LLM feedback, we conduct a series of qualitative and quantitative analyses, offering novel insights into the quality discrepancies between human feedback vs. weak LLM feedback.

## 1 Introduction

As we observe the impressive capabilities of large language models (LLMs) across diverse applications (Brown et al., ; Achiam et al., ; Bubeck et al., ; Team et al., ; Anthropic, ), there emerges a critical need to ensure AI systems are helpful and harmless. AI alignment aims to harmonize AI behaviors with human intentions and values and ensure safe and desirable behavior. A key recipe to achieve alignment involves presenting pairs of responses and collecting binary feedback (*e.g.*, preferred, less preferred) based on the comparative quality of these responses. The prevailing methods in alignment can be categorized based on the source of the feedback. For example, popular framework Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., ; Ziegler et al., ; Ouyang et al., ; Bai et al., ) relies on pure human judgments, which often involves considerable human labor and manual effort. On the other end of the spectrum, framework such as Reinforcement Learning from AI Feedback (RLAIF) (Lee et al., ) harnesses feedback from high-capacity LLMs to annotate preference datasets, often incurring significant computational and financial costs, along with the need for heavy prompt engineering.

These contrasting frameworks highlight two extremes of a feedback spectrum, raising critical questions about the largely untapped middle ground that leverages the strengths of both while alleviating their respective drawbacks. This middle ground entails using significantly smaller LLM that are less resource-intensive than top-tier models, yet offer more automation than purely human feedback. For instance, while a model like GPT-4 might have trillions of parameters, a weak LLM might only have hundreds of millions or even fewer. This smaller size means they inherently demand less computational power, thus reducing operational costs and enabling faster iteration cycles in development. This approach not only aligns with the practical demands of deploying scalable AI solutions but also addresses the need for sustainable development within AI research, making it a prudent choice for tasks where the ultra-high capabilities of state-of-the-art models do not necessarily translate to significant improvements in feedback quality.

Despite the appeal, *the research community still lacks a systematic evaluation and understanding of weak LLMs’ capability to generate feedback for alignment*. Motivated by this gap, we undertake a comprehensive investigation of alignment via weak LLM feedback. Specifically, this paper makes three key contributions:

Contribution 1: A framework for evaluating alignment via weak LLM feedback. We formalize a learning and evaluation workflow utilizing feedback from weak LLM instead of traditional human annotations. Our framework operates on the combination of labeled and unlabeled preference datasets. Different from the existing RLHF framework, we leverage data comprising of unlabeled triplet $(x,y_{1},y_{2})$, where both $y_{1}$ and $y_{2}$ are responses corresponding to prompt $x$ but the preference is unknown. In practice, the unlabeled triplets can be collected in large volumes
without the need for human preference annotations. A weak LLM trained on labeled data will provide preference feedback on the large unlabeled data. Finally, we train a target LLM policy based on the weak LLM’s feedback. The training workflow is in reminisce of semi-supervised learning (Zhu, ); however, it introduces a novel perspective by connecting the classical framework to alignment—a field that has yet to be thoroughly explored, especially in the context of using weak AI feedback.

Figure: Figure 1: A spectrum of feedback for aligning LLMs, ranging from labor-intensive human annotations (e.g., RLHF (Ouyang et al., )) to highly automated, resource-intensive LLM feedback (e.g., RLAIF (Bai et al., ; Lee et al., )). Our work explores the largely untapped middle ground, evaluating weak LLM feedback for alignment.
Refer to caption: x1.png

Contribution 2: A comprehensive evaluation and novel empirical findings (Section 3).
We employ the framework to evaluate the impact of weak LLM feedback on alignment across a variety of model scales and diverse model families. Surprisingly, *our results reveal that using a weak LLM, with size as small as 125M (Zhang et al., ), to provide preference label for alignment can match or even exceed the performance of using full human feedback* (see Figure 2 and Figure 3). Moreover,
we systematically evaluate alignment performance when the feedback is provided by LLM of varying capacities: a weak supervisor (OPT-125M), a moderate supervisor (OPT-1.3B), a strong supervisor (Llama-3-8B), and a very strong supervisor (GPT-4). We found that the performance under weak, moderate, and strong supervisors is nearly comparable, suggesting that the supervisor model’s size has minimal impact on feedback effectiveness. Notably, the weak LLM, OPT-125M, outperforms the more advanced GPT-4, indicating that a task-specific weak LLM can provide more effective feedback than a larger, more powerful LLM that relies solely on prompt engineering.

Contribution 3: An in-depth analysis on the quality of weak LLM’s feedback (Section 4). To deepen our understanding of alignment under weak LLM feedback and reason our observations, we
conduct a series of qualitative and quantitative analyses. A pivotal aspect of our study lies in the examination
of quality discrepancies between human feedback *vs.* weak LLM feedback. Our key observations are threefold: (1) when weak LLM’s chosen response contradicts human feedback, nearly half of these responses exhibit higher quality, suggesting the lack of reliability in human annotation and weak LLM can surpass human judgments; (2) weak LLM’s feedback can be qualitatively similar to human feedback in contexts with clear gap, yet they exhibit uncertainty in judgments when the distinctions in response quality are subtle; and (3) advanced LLMs like GPT-4 also show increased feedback inconsistency in scenarios where response distinctions are minimal, highlighting the challenges faced by strong LLMs in providing feedback.

## 2 Preliminaries on LLM Alignment

We denote $\pi_{\theta}$ as a language model policy parameterized by $\theta$, which takes in an input prompt $x$, and outputs a discrete probability distribution $\pi_{\theta}(\cdot|x)$ over the vocabulary space $\mathcal{V}$.
$\pi_{\theta}(y|x)$ refers to the model’s probability of outputting response $y$ given input prompt $x$. Alignment algorithms operate on comparative data, where pairs of responses are presented, and the model is trained to produce the preferred response given a query. Formally, we define the preference data below.

###### Definition 2.1 ( Preference data ) .

Consider two responses $y_{w},y_{l}$ for an input prompt $x$, we denote $y_{w}\succ y_{l}$ if $y_{w}$ is preferred over $y_{l}$. We call $y_{w}$ the preferred response and $y_{l}$ the non-preferred response. Each triplet $(x,y_{w},y_{l})$ is referred to as a preference. Furthermore, the empirical dataset $\mathcal{D}=\{(x_{i},y_{w,i},y_{l,i})\}_{i=1}^{n}$ consists of $n$ such triplets sampled from a preference distribution.

###### Definition 2.1 ( Preference data ) .

#### Reinforcement Learning from Human Feedback.

RLHF is a widely used paradigm for fine-tuning language models based on human preferences (Christiano et al., ; Ziegler et al., ; Ouyang et al., ; Bai et al., ). The key stages in RLHF are reward modeling and reinforcement learning with the learned reward function.

Reward modeling learns a function mapping, which takes in the prompt $x$ and response $y$ and outputs a scalar value $r(x,y)$ signifying the reward. A preferred response should receive a higher reward, and vice versa. Based on the Bradley–Terry model (Bradley & Terry, ), the reward function is optimized over a dataset of human preferences, with the following objective:

$$ $\mathcal{L}_{R}=-\mathbb{E}_{(x,y_{w},y_{l})\in\mathcal{D}}[\log\sigma(r(x,y_{ w})-r(x,y_{l}))],$ (1) $$

where $\sigma$ denotes the sigmoid function. Using the learned reward function, the model is further fine-tuned with reinforcement learning to maximize the expected rewards, thus promoting exploration and adherence to learned preferences. The optimization objective is formulated as follows:

$$ $\max_{\pi_{\theta}}~{}~{}\mathbb{E}_{\hat{y}\sim\pi_{\theta}(\cdot|x)}[r(x, \hat{y})]-\beta\log\frac{\pi_{\theta}(\hat{y}|x)}{\pi_{\text{ref}}(\hat{y}|x)},$ (2) $$

where $\hat{y}$ represents the response generated by the current policy
$\pi_{\theta}$ for the prompt $x$, $\pi_{\text{ref}}$ indicates the reference policy or the initial policy before running RL optimization, and $\beta$ serves as a hyperparameter to regulate the KL divergence.

Training with RLHF can be computationally expensive due to the use of multiple models. As an alternative, Rafailov et al. () proposed to directly optimize for the policy best satisfying the preferences with a simple objective:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}};\mathcal{D})=-\mathbb{E }_{(x,y_{w},y_{l})\in\mathcal{D}}\left[\log\sigma\left(\beta\left(\log\frac{ \pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\log\frac{\pi_{\theta}(y_{l} |x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right)\right].$ (3) $$

Rafailov et al. () showed that under mild assumptions, the optimal policy under the DPO objective (3) is equivalent to the optimal policy under the RLHF objective (2). This objective facilitates a more direct reflection of human preference judgments within the optimization framework.

## 3 How good is alignment with weak LLM feedback?

In existing alignment approaches described above, models are often trained on fully supervised data where each preference label is hand-annotated by humans.
To systematically evaluate the reliability of using feedback from a weak LLM instead of human feedback, we outline the training workflow in Section 3.1, experimental setup in Section 3.2, and present our main findings in Section 3.3.

### 3.1 Alignment via Weak LLM Feedback

Consider two empirical datasets, $\mathcal{D}_{l}$ and $\mathcal{D}_{u}$, representing a labeled preference dataset and an unlabeled dataset respectively. The labeled dataset $\mathcal{D}_{l}$ comprises of triplets $(x,y_{w},y_{l})$, where human annotators have indicated a known preference $y_{w}\succ y_{l}$. $\mathcal{D}_{u}$, on the other hand, comprises of unlabeled triplet $(x,y_{1},y_{2})$, where both $y_{1}$ and $y_{2}$ are responses corresponding to $x$ but the preference is unknown. A weak LLM, trained on $\mathcal{D}_{l}$, will provide preference feedback on the unlabeled data $\mathcal{D}_{u}$.
This setup offers practical advantages since unlabeled triplets can be collected in large volumes without the need for human preference annotations. For example, one can generate multiple responses by querying varying LLMs with the same prompt. Below, we describe the alignment process using feedback from the weak LLM.

#### Preference feedback from the weak LLM.

We first train a weak language policy $\pi_{w}$ based on the labeled preference dataset $\mathcal{D}_{l}$. We use the subscript $w$ to indicate “weak” in the remainder of the paper. Specifically, the weak LLM is optimized using the DPO loss (*cf.* Equation 3), under which the optimal policy is equivalent to that of RLHF:

$$ $\pi_{w}^{*}=\operatorname{argmin}\mathcal{L}_{\text{DPO}}(\pi_{w};\pi_{w}^{ \text{SFT}};\mathcal{D}_{l}),$ (4) $$

where $\pi_{w}^{*}$ signifies the optimal policy for the weak model. $\pi_{w}^{\text{SFT}}$ is the reference model or the initialization, which is an SFT model fine-tuned on the preferred question-answer pairs $(x,y_{w})$ in $\mathcal{D}_{l}$. Compared to directly using the untuned base model as a reference model, performing SFT enhances the model’s ability to generate desired responses to questions (see Appendix C).

We then generate the weak feedback for unlabeled data $\mathcal{D}_{u}$, leveraging the weak language model $\pi_{w}^{*}$. For each triplet $(x,y_{1},y_{2})\in\mathcal{D}_{u}$, we compute the reward $r_{w}(x,y_{1})$ and $r_{w}(x,y_{2})$ according to DPO’s implicit reward:

$$ $r_{w}(x,y)=\beta\log\frac{\pi_{w}(y|x)}{\pi_{w}^{\text{SFT}}(y|x)}.$ (5) $$

We then assign the preference label $\hat{y}_{w}$ for the response with a higher predicted reward, and $\hat{y}_{l}$ for the response with a lower predicted reward. Mathematically:

$$ $\hat{y}_{w}=\left\{\begin{array}[]{ll}y_{1}&r_{w}(x,y_{1})>r_{w}(x,y_{2})\\ y_{2}&r_{w}(x,y_{1})\leq r_{w}(x,y_{2})\end{array}\right.$ $$

We denote the resulting weakly labeled dataset as $\mathcal{D}_{\text{weak}}=\{(x,\hat{y}_{w},\hat{y}_{l})\}$, where $|\mathcal{D}_{\text{weak}}|=|\mathcal{D}_{u}|$.

#### Alignment with feedback from the weak LLM.

Finally, we train an LLM policy $\pi_{\theta}$ based on weak LLM feedback $\mathcal{D}_{\text{weak}}$. The model is aligned using the following objective:

$$ ${\pi}_{\theta}^{*}=\operatorname{argmin}\mathcal{L}_{\text{DPO}}({\pi}_{\theta };{\pi}_{\theta}^{\text{SFT}};\mathcal{D}_{\text{weak}}),$ (6) $$

where ${\pi}_{\theta}^{*}$ is the optimal policy based on the weak LLM feedback. To obtain the reference model ${\pi}_{\theta}^{\text{SFT}}$, we fine-tune the base model using question-answer pairs $(x,\hat{y}_{w})$ in $\mathcal{D}_{\text{weak}}$. The training workflow outlined above is in reminisce of semi-supervised learning; however, it introduces a novel perspective by connecting the classical framework to alignment—a field that has yet to be thoroughly explored.
*A central unresolved question we address in this paper is how effectively LLMs can be aligned using feedback from weak LLMs as opposed to relying solely on human feedback.* The training workflow thus serves as the tool to explore our core research question. To understand this, we conduct a systematic evaluation in the next section.

### 3.2 Experimental Setup

#### Dataset.

To evaluate the performance, we use the Anthropic HH-RLHF (Helpful and Harmless) dataset (Bai et al., ), which is the most commonly used dataset for alignment. The dataset consists of 112,000 training samples and 12,500 test samples and is publicly available. Each sample includes a prompt and two responses, with one being preferred over the other. The selected responses are annotated based on the opinions of crowd workers, who assess which response is more helpful and harmless. We preprocess the dataset by filtering out samples with token lengths less than 512, which yields 100,000 training samples and 11,000 test samples. We split the training data into two disjoint sets. The first subset is used as labeled data $\mathcal{D}_{l}$, and the remainder is used as the unlabeled data $\mathcal{D}_{u}$ (by disregarding the preference labels). We will vary the size of $\mathcal{D}_{l}$ in our ablation. We also evaluate on the Reddit TL;DR (TL;DR) summarization dataset from Stiennon et al. (), which consists of a Reddit post and several short summaries, judged for quality and informativeness by human evaluators.

#### Models.

For our main experiment, we utilize the OPT family model introduced by Zhang et al. (), which provides a spectrum of model sizes. This versatility enables us to evaluate the performance across different levels of model capability. Specifically, we employ OPT models of varying sizes (1.3B, 2.7B, 6.7B, and 13B) as our LLM policy models, trained with feedback from a weak LLM. Additionally, to validate the efficacy of our experiments, we consider more advanced open-sourced models, including Llama-2-7B (Touvron et al., ), Mistral-7B (Jiang et al., ) and Gemma-7B (Team et al., ). For a comprehensive description of the hyper-parameters employed in our experiments, please refer to Appendix A.

#### Evaluation metrics.

Given the test set $\mathcal{D}_{\text{test}}$, we evaluate the generation performance under two policies: ${\pi}_{\theta}^{*}$ (with weak LLM feedback), and ${\pi}_{h}^{*}$ (with human feedback). ${\pi}_{h}^{*}$ is trained with DPO loss on the same set of triplets in $\mathcal{D}_{u}$, except for using the original preference label provided by human annotators. This allows us to estimate the performance achieved with fully supervised data. For a fair comparison, ${\pi}_{\theta}^{*}$ and ${\pi}_{h}^{*}$ always share the same model capacity and only differ in the source of feedback. We assess the generation performance using the following metrics:

- •
Gold reward:
Previous studies (Gao et al., ; Coste et al., ; Xiong et al., ) commonly utilize gold reward as a metric for assessing the generation quality of language models. Gold reward is desirable due to the high cost associated with obtaining ground truth gold rewards from human annotators. We employ the output of a large auxiliary gold reward model, denoted as $r_{\text{gold}}$, to evaluate the quality of generated responses. For each test input prompt $x$ from $\mathcal{D}_{\text{test}}$, we generate a response $\hat{y}$ according to a given language policy, and then compute its gold reward as $r_{\text{gold}}(x,\hat{y})$. A higher gold reward signifies that the model’s responses better align with desired preferences. Our results are consistent under alternative gold reward models; see Appendix C for details.
- •
GPT-4 win-rate: We employ GPT-4 as a proxy for human evaluation by instructing it to review two responses (from two different policies) to the same prompt. It then assigns a rating on a scale from 1 to 10. A higher win rate indicates that a policy on average produces more favorable answers.

### 3.3 Main Results

Figure: Figure 2: (a) Alignment with feedback from a weak LLM (OPT-125M) can outperform human feedback. (b) Alignment performance on OPT-1.3B model under varying capability of supervisor. See Section 3.3 for details.
Refer to caption: x2.png

#### Alignment with weak LLM feedback can outperform human feedback.

In Figure 2(a), we evaluate the alignment performance using feedback from the weak LLM *vs.* human. Employing OPT-125M as our weak LLM to provide supervision (with the lowest capacity in the model family), we align student models of varying capacities (OPT-1.3B, OPT-2.7B, OPT-6.7B, and OPT-13B). *Interestingly, the alignment performance using weak LLM feedback closely matches or even surpasses that of using human feedback*. The similar alignment performance between $\pi_{\theta}^{*}$ and ${\pi}_{h}^{*}$ has not been observed in previous studies. Moreover, under the same feedback provided by the weak LLM, the alignment performance improves as the student model capacity increases from 1.3B to 13B. Given that the primary objective of alignment is to enhance the model’s ability to generate content that better resonates with human goals, our findings suggest that leveraging weak LLM feedback is a promising route without access to full human supervision. For an in-depth analysis of the unexpectedly high efficacy of weak LLM feedback, we defer to Section 4.

#### Alignment performance under the varying capability of supervisor.

Existing frameworks on learning from AI feedback, such as RLAIF (Lee et al., ), employ more advanced LLMs (*e.g.*, GPT-4) to annotate preference datasets for the alignment of student models. To understand the alignment performance under different supervisor models, we systematically consider a spectrum of supervisor LLMs of varying capabilities: (1) *weak supervisor* based on OPT-125M, (2) *moderate supervisor* based on OPT-1.3B, (3) *strong supervisor* based on Llama-3-8B, and (4) *very strong supervisor* based on GPT-4. For variants (1) through (3), we apply the same training workflow described in Section 3. For variant (4), we utilize GPT-4 as the LLM labeler, following the prompt suggested by Lee et al. (). Based on these four different sources of feedback, we compare the alignment performance on the same policy model OPT-1.3B. As shown in Figure 2 (b),
the alignment performance under the weak, moderate, and strong supervisors is nearly comparable. This suggests that the size of the supervisor model plays a less impactful role in the effectiveness of feedback. Notably, we observe instances where the weak LLM with a small capacity, OPT-125M, can outperform GPT-4 in providing feedback on preferences. Although GPT-4 is a more advanced model, this finding indicates that a task-specific weak LLM can serve as a more effective supervisor than a larger, more powerful LLM that relies solely on prompt engineering.

### 3.4 Additional Ablations

#### Results on different model families.

To further validate our findings, we extend our evaluation to additional model families, including Llama-2-7B (Touvron et al., ), Mistral-7B (Jiang et al., ) and Gemma-7B (Team et al., ). We employ OPT-125M as the weak LLM to provide preference labels. Figure 3(a) displays the average gold rewards of the generated responses, based on policies $\pi_{\theta}^{*}$ (with weak LLM feedback) and ${\pi}_{h}^{*}$ (with human feedback). The gold rewards achieved using weak LLM feedback consistently surpass those obtained from human feedback. This underscores the effectiveness of alignment through weak LLM feedback across various LLM families.

Figure: Figure 3: (a) Results on different model families. (b) GPT-4 evaluation for different models aligned with weak LLM feedback *vs.* human feedback.
Refer to caption: x3.png

Results under GPT-4 evaluation. Beyond gold reward measurement, we assess the win-rate of the model aligned with weak LLM feedback ($\pi_{\theta}^{*}$) against the human feedback (${\pi}_{h}^{*}$), utilizing GPT-4 to judge the helpfulness and harmlessness within dialogue contexts. We randomly select 100 prompts from the test set of HH-RLHF and employed GPT-4 to determine whether responses generated under the policy $\pi_{\theta}^{*}$ were superior to those from ${\pi}_{h}^{*}$. Considering the diversity of model families and sizes, we evaluated the GPT-4 win-rate across five student model variants: OPT-1.3B, OPT-2.7B, OPT-6.7B, Llama-2-7B, and Mistral-7B. The weak LLM feedback all comes from the OPT-125m. As illustrated in Figure 3 (b), the win-rate for policy $\pi_{\theta}^{*}$ (with weak LLM feedback) approaches 50% for all cases, indicating that its performance competitively matches that of using human feedback ${\pi}_{h}^{*}$.

Figure: Figure 4: (a) Results under different data sizes. (b) Results on different tasks (TL;DR).
Refer to caption: x4.png

Results under different data sizes.
In this ablation, we systematically investigate the impact of dataset size. We vary the ratio between the labeled dataset size and the full dataset size: 1/16, 1/8, 1/4, 1/2, with the remaining data serving as the unlabeled subset. We train the weak LLM across different ratios with the same steps.
We utilize OPT-125M as the weak LLM to align the policy model OPT-1.3B. Notably, as shown in Figure 4 (a), even under a small labeled dataset size with a ratio of 1/16, the performance using weak LLM feedback is favorably close to that of using human feedback.

Results on different datasets and tasks.To examine the effectiveness of weak LLM feedback for alignment across broader NLP tasks, we turn our attention to the Reddit TL;DR dataset (Stiennon et al., ), which focuses on summarization—a stark contrast to dialogue tasks, which consists of a Reddit post and several short summaries, judged for quality and informativeness by human evaluators. Similar to the setup for HH-RLHF, we use OPT-125M as the weak LLM and assess the performance of weak LLM feedback v.s. human feedback for different student sizes: OPT-1.3B, OPT-2.7B, OPT-6.7B, and OPT-13B. As depicted in Figure 4 (b), the average gold rewards for outputs sampled under the policies $\pi_{\theta}^{*}$ also outperform ${\pi}_{h}^{*}$ across all student model sizes. This finding reinforces the efficacy of weak LLM feedback for alternative linguistic tasks.

## 4 In-Depth Understanding of Weak LLM’s Feedback

#### When weak LLM feedback matches/mismatches human feedback.

In this ablation, we investigate the impact of samples either matching or mismatching human feedback. Adopting OPT-125M as the weak LLM to provide feedback for the OPT-1.3B model alignment, we design three controlled settings for this ablation:

- •
$\mathcal{D}_{\text{weak}}$: Use all samples, $\{(x,\hat{y}_{w},\hat{y}_{l})\}$, for training. This is the same as our main setting.
- •
$\mathcal{D}_{\text{weak}}^{\text{match}}$: Use samples with weak labels *matching* the human feedback, i.e., $(r_{w}(x,y_{w})>r_{w}(x,y_{l}))$.
- •
$\mathcal{D}_{\text{weak}}^{\text{mismatch}}$: Use samples with labels *mismatching* the human feedback, i.e., $(r_{w}(x,y_{w})<r_{w}(x,y_{l}))$.

**Table 1: Effect of purifying weak LLM feedback. Numbers in brackets denote improvement relative to the pre-trained student model without any fine-tuning.**
|  | % matching | Gold reward of |
| --- | --- | --- |
|  | human feedback | aligned model $\pi_{\theta}^{*}$ |
| $\mathcal{D}_{\text{weak}}$ | 60.6% | 4.84 (+2.61) |
| $\mathcal{D}_{\text{weak}}^{\text{match}}$ | 100% | 4.78 (+2.56) |
| $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ | 0% | 4.01 (+1.79) |

As shown in Table 1,
training with $\mathcal{D}_{\text{weak}}^{\text{match}}$ (samples with matching labels between weak LLM feedback and human feedback) yields gold rewards comparable to those obtained from using the full set $\mathcal{D}_{\text{weak}}$.
Additionally, when examining the alignment performance using $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$,
the gold reward significantly improves relative to the pre-trained strong model without any fine-tuning by 1.79. This indicates that $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ effectively improves alignment, even when the weak LLM provides feedback that are entirely contrary to the human preference. We further confirm that the noise in $\mathcal{D}_{\text{weak}}$ appears to behave differently from random noise (see Appendix C). The phenomenon is intriguing and prompts our next analysis.

**Table 2: Summary statistics of average gold reward on HH-RLHF dataset. “Chosen” and “Rejected” indicate the human preference for original dataset $\mathcal{D}_{\text{human}}$ and the weak LLM preference for the $\mathcal{D}_{\text{weak}}$, $\mathcal{D}_{\text{weak}}^{\text{match}}$ and $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$.**
|  |  | Chosen | Rejected | $\Delta$ |
| --- | --- | --- | --- | --- |
| $\mathcal{D}_{\text{human}}$ |  | 5.77 | 4.23 | 1.52 |
| $\mathcal{D}_{\text{weak}}$ |  | 5.63 | 4.39 | 1.23 |
| $\mathcal{D}_{\text{weak}}^{\text{match}}$ |  | 5.93 | 3.66 | 2.27 |
| $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ |  | 5.17 | 5.53 | -0.36 |

#### Dive into the quality of weak LLM vs. human feedback.

In Table 3, we provide a qualitative example from $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ where the chosen response by weak LLM mismatches that of the human feedback. More qualitative examples are in Appendix B. Despite contradicting the human preference, the chosen response by the weak LLM appears to be of higher quality. We further measure the gold reward as a proxy for response quality, which is indeed higher for the response chosen by the weak LLM, i.e., $r_{\text{gold}}(x,\hat{y}_{w})>r_{\text{gold}}(x,\hat{y}_{l})$. Across all the samples in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$, we notice that 44.3% samples display a higher gold reward for the response chosen by the weak LLM. *This indicates that the preferences from weak LLM in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ are not entirely erroneous, but in fact better than human feedback in nearly half of the cases*(^1^11This is further evidenced by our GPT-4 evaluation, where the response chosen by weak LLM achieved a 46% win rate over the rejected response.). Considering the feedback from weak LLM and human shares $\mathcal{D}_{\text{weak}}^{\text{match}}$ portion and only differs in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$, the close performance achieved between the two models can be attributed to the fact that human feedback is similarly imperfect.

Building on the observation that weak LLM can sometimes surpass human judgment, we delve into the characteristics of datasets produced under such supervision. Table 2 shows that the gold rewards for chosen responses in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ are on average comparable to those for rejected responses, while the gold rewards for chosen responses in $\mathcal{D}_{\text{weak}}^{\text{match}}$ are significantly higher than the rejected responses.
*This suggests that the weak LLM primarily errs between choices that are subtly different, while it remains reliable in distinguishing between options with clear quality distinctions*. Overall, the discrepancy ($\Delta$) within $\mathcal{D}_{\text{weak}}$ is similar to that in $\mathcal{D}_{\text{human}}$, indicating that the quality of weak labels is not severely compromised.

Table: Table 3: Qualitative example of the chosen and rejected responses in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$.

Table: Table 4: A qualitative example in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ with low preference consistency in GPT-4 eval.

**Table 5: The GPT-4 preference consistency.**
| Dataset | consistency |
| --- | --- |
| $\mathcal{D}_{\text{weak}}^{\text{match}}$ | 0.84 |
| $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ | 0.66 |

#### More qualitative analysis.

To further investigate the quality gap in responses from the dataset where weak LLM feedback and human feedback diverge ($\mathcal{D}_{\text{weak}}^{\text{mismatch}}$), we conducted a qualitative assessment using GPT-4. Our goal is to determine whether GPT-4’s preferences for chosen and rejected responses in $\mathcal{D}_{\text{weak}}^{\text{match}}$ and $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ remain consistent. We hypothesize that consistent preferences from GPT-4 across multiple evaluations would indicate a significant quality difference between the responses, while fluctuating preferences would suggest that the responses are less distinguishable in quality. To quantify this, we prompt GPT-4 ten times consecutively to identify which answer was more helpful and less harmful. We randomly sample 100 examples from $\mathcal{D}_{\text{weak}}^{\text{match}}$ and $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$, and measure preference consistency by calculating the fraction of majority votes for each question, which ranges from 0.5 to 1. For instance, Table 4 presents a qualitative example from $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$, where the weak LLM and humans have different preferences. We observe that GPT-4 indeed exhibits mixed preferences across the ten evaluations, indicating a low preference consistency (0.5) and thus the indistinguishable nature between the two responses. Indeed, as illustrated in Table 5, the average consistency for samples in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$ is lower than that of $\mathcal{D}_{\text{weak}}^{\text{match}}$. This suggests that advanced LLMs such as GPT-4 experience significantly higher inconsistency when the quality distinctions between two responses are minimal, highlighting the challenges faced by both weak and ultra-large LLMs in providing feedback.

## 5 Discussion

#### Differences w.r.t. Burns et al. ( 2024 ) .

Previous work by Burns et al. () explored weak-to-strong generalization, wherein a less capable model’s weak supervision signals are used to guide a stronger, larger model. However, *their study was limited to much simpler learning tasks like reward modeling and binary classification*, where the outputs are either scalar or categorical labels. They did not investigate the more challenging task of language generation, which is closely tied to alignment and involves a significantly more complex output space. Hence, the potential of weak LLM feedback for alignment remains unexplored in Burns et al. (), which is the novel focus of our study.

Contrary to Burns et al. (), which employed discriminative metrics such as accuracy, our study evaluates the generative performance essential for AI alignment. Although Burns et al. () reported a noticeable performance gap between models trained with weak LLM feedback versus those with human feedback, our findings challenge these conclusions. Considering that human preferences are noisy and unreliable, accuracy on the preference test set is a questionable metric. Our works show contrary conclusions when directly measuring the generation performance of the aligned model, and demonstrate that using feedback from weak LLM can not only match but potentially exceed human feedback in alignment tasks. Our work suggests a promising direction for future research in leveraging weak AI signals for alignment.

## 6 Related work

#### Large language model alignment.

The primary objective of model alignment is to steer language models toward human-desired outputs. Numerous studies have leveraged human feedback to refine language models in accordance with human preferences (Christiano et al., ; Ziegler et al., ; Ouyang et al., ; Stiennon et al., ; Kreutzer et al., ).
However, gathering high-quality human preference data can be costly, prompting researchers to explore AI-generated feedback for alignment purposes (Bai et al., ; Lee et al., ; Ding et al., ; Gilardi et al., ; Guo et al., ).
Given the computational inefficiency of RLHF, recent shifts in focus favor closed-form losses that directly utilize offline preferences, like Direct Preference Optimization (Rafailov et al., ) and related methodologies (Xiong et al., ; Ethayarajh et al., ; Liu et al., ). Nonetheless, both RLHF and offline preference-based methods presuppose access to high-quality human, raising doubts about their scalability. Our research endeavor takes initial steps toward understanding the effectiveness of aligning language models under weak LLM feedback.
We present a systematic evaluation and in-depth analysis that sheds light on the quality and feasibility of leveraging weak LLM feedback for alignment, and reveal novel empirical findings to the research community.

#### Large language model as a judge.

The use of LLM-as-a-Judge prompting to evaluate language models has become a common practice (Dubois et al., ; Li et al., ; Bai et al., ; Saha et al., ). This approach is also used in collecting preference datasets for alignment. For example, Bai et al. () employs an LLM to assess responses, refine them, and then use the resulting data to train a reward model known as “RL from AI Feedback” (RLAIF). Similarly, Lee et al. () demonstrated that employing a strong LLM for LLM-as-a-Judge prompting to create preference datasets yields performance nearly on par with traditional RLHF. Yuan et al. () explored the concept of LLM-as-a-Judge prompting to enable models to generate their own rewards during training. While many studies (Li et al., ; Kim et al., ; Chen et al., ; Ye et al., ) focus on leveraging advanced LLMs as judges for preference dataset collection, we investigate the effectiveness of feedback from weak LLMs with significantly smaller capacity. We reveal new insight that a task-specific weak LLM
can provide more effective preference feedback than an ultra-large LLM (*e.g.*, GPT-4) that relies solely on prompt engineering.

#### Weak-to-strong generalization.

Weak-to-strong generalization refers to the scenario where a weaker teacher model supervises a stronger student model. Unlike traditional teacher-student frameworks such as semi-supervised learning (Laine & Aila, ; Tarvainen & Valpola, ), domain adaptation (French et al., ; Chen et al., ), and knowledge distillation (Hinton et al., ; Beyer et al., )—which generally involve a stronger teacher guiding a weak student—other studies have demonstrated the utility of employing comparably weaker or even lesser-capable teachers to enhance student model performance (Xie et al., ; Higuchi et al., ; Burns et al., ; Freund & Schapire, ; Furlanello et al., ; Green Larsen & Ritzert, ). Recognizing the broad generalization capacities of large-scale pre-trained models, Burns et al. () introduced the concept of weak-to-strong generalization for large language model, focusing on simpler discriminative tasks such as reward modeling rather than generative tasks such as alignment. A detailed discussion on the differences with Burns et al. () is in Section 5. Different from previous works that explore the effectiveness of weak LLM under the standard supervised fine-tuning (Liu & Alahi, ; Li et al., ; Ji et al., ; Hase et al., ; Sun et al., ), we aim to delve deeper into the role of weak LLM could play for alignment. These tasks pose unique challenges due to their inherently diverse output and the complexity of performance evaluation but hold promise for advancing superalignment in generative models (Puthumanaillam et al., ). Instead of adopting model interpolation for alignment without any additional training (Zheng et al., ), we explore the three-stage framework for alignment with feedback from weak LLM.

## Acknowledgement

Research is supported in part by the AFOSR Young Investigator Program under award number FA9550-23-1-0184, National Science Foundation (NSF) Award No. IIS-2237037 & IIS-2331669, and the UL Research Institutes through the Center for Advancing Safety of Machine Intelligence. The authors would like to thank Shawm Im for the valuable feedback on the work.

## Appendix A Experimental Detail

### A.1 Training Details

Software and hardware. Our experiments are conducted on servers equipped with NVIDIA A100 GPUs, with 80 GB of VRAM. The operating system used is Ubuntu 22.04.2 LTS, supported by NVIDIA CUDA Toolkit version 12.1 and cuDNN version 8.9. All experimental implementations are carried out in Python version 3.11.4, utilizing the PyTorch framework version 1.12.1.

#### Training hyper-parameters.

Based on TRL (^2^22[https://github.com/huggingface/trl](https://github.com/huggingface/trl)), we deploy the training of both teacher and student models with the same hyperparameters as shown in Table 7 and Table 8.

#### Reddit TL;DR dataset preprocessing details.

For the Reddit TL;DR dataset, we preprocess the data by filtering out samples with fewer than 512 tokens. This results in 92,000 training samples and 8,000 test samples. We divide the training data into two disjoint sets of 46,000 samples each. The first set serves as the labeled dataset, denoted as $\mathcal{D}_{l}$, while the second set, referred to as the unlabeled dataset, $\mathcal{D}_{u}$, disregards the preference labels.

### A.2 Evaluation Details

Table: Table 6: Sample prompt for the GPT-4 evaluation. Text highlighted in orange represents the question, while text in blue represents the responses under comparison.

GPT-4 evaluation details. Table 6 presents the prompts and responses in our GPT-4 evaluation, adopted from (Khanov et al., ). Each GPT-4 request comprises both a system and a user prompt. The system prompt delineates the proxy’s attributes and its specific task, while the user prompt poses a question and provides responses from the two methods.

Gold reward model. For the evaluation of the HH-RLHF, we leverage the state-of-the-art gold reward model Ray2333/reward-model-Mistral-7B-instruct-Unified-Feedback from the RewardBench leaderboard (Lambert et al., ), which fine-tunes Mistral-7B-Instruct-v0.2 on the llm-blender/Unified-Feedback dataset. We also verify our results consistently hold under different choices of gold reward models in Appendix C. For Reddit TL;DR, we take the state-of-the-art gold reward model OpenAssistant/reward-model-deberta-v3-large-v2 on this dataset from the RewardBench leaderboard (Lambert et al., ) for evaluation.

Hyper-parameters for model generation. To evaluate the response of the model, we adopted the temperature as 0.7 and the max tokens as 256.

**Table 7: Summary of training hyperparameters for supervised fine-tuning and direct preference optimization for OPT-family models for HH-RLHF.**
|  | Parameters | Value |
| --- | --- | --- |
| Supervised fine-tuning | Number of epochs | 1 |
| Learning rate | $1\times 10^{-5}$ |  |
| Batch size | 32 |  |
| Gradient accumulation steps | 1 |  |
| Maximum sequence length | 512 |  |
| DeepSpeed Zero stage | 2 |  |
| LoRA rank | 0 |  |
| Direct preference optimization | Number of epochs | 1 |
| Learning rate | $5\times 10^{-5}$ |  |
| $\beta$ | 0.1 |  |
| Batch size | 16 |  |
| Gradient accumulation steps | 1 |  |
| Maximum sequence length | 512 |  |
| DeepSpeed Zero stage | 2 |  |
| LoRA rank | 8 |  |

**Table 8: Summary of training hyperparameters for supervised fine-tuning and direct preference optimization for OPT-family models on Reddit TL;DR.**
|  | Parameters | Value |
| --- | --- | --- |
| Supervised fine-tuning | Number of epochs | 1 |
| Learning rate | $1\times 10^{-5}$ |  |
| Batch size | 32 |  |
| Gradient accumulation steps | 1 |  |
| Maximum sequence length | 512 |  |
| DeepSpeed Zero stage | 2 |  |
| LoRA rank | 0 |  |
| Direct preference optimization | Number of epochs | 1 |
| Learning rate | $5\times 10^{-6}$ |  |
| $\beta$ | 0.5 |  |
| Batch size | 16 |  |
| Gradient accumulation steps | 1 |  |
| Maximum sequence length | 512 |  |
| DeepSpeed Zero stage | 2 |  |
| LoRA rank | 8 |  |

## Appendix B Additional Qualitative Examples

In Table 9, we provide additional qualitative examples from $\mathcal{D}_{\text{weak}}$ where the chosen response by weak LLM mismatches that of
the human annotators. Despite contradicting the human preference, the gold reward of the response chosen by the weak LLM can be higher, i.e., $r_{\text{gold}}(x,\hat{y}_{w})>r_{\text{gold}}(x,\hat{y}_{l})$. This indicates that the preferences of the weak LLM can align with the gold reward model, even if they differ from human labels.

Table: Table 9: Comparative examples of the chosen and rejected responses in $\mathcal{D}_{\text{weak}}^{\text{mismatch}}$.

## Appendix C Additional Experiments

#### Evaluation with more gold reward models.

To verify the reliability of our conclusions, we employ different gold reward models to evaluate the quality of model output. Specifically, we leverage another two competitive gold reward models weqweasdas/RM-Mistral-7B and OpenAssistant/reward-model-deberta-v3-large-v2 from the RewardBench leaderboard (Lambert et al., ). As shown in Figure 5, using these two alternative gold reward models, we can still observe that alignment with weak LLM feedback can outperform human feedback. This confirms the reliability of our findings, and its insensitivity to the choice of gold reward model.

Figure: Figure 5: (a) Evaluation with the gold reward model reward-model-deberta-v3-large-v2. (b) Evaluation with the gold reward model RM-Mistral-7B.
Refer to caption: x5.png

#### Correlation of generation quality.

In Figure 6, we investigate whether the generation quality correlates between models using weak LLM feedback *vs.* human feedback. In particular, for each test input prompt, we measure the sentence-level similarity between the model’s generation $\hat{y}$ (using either $\pi_{\theta}^{*}$ or ${\pi}_{h}^{*}$) and the chosen response $y_{w}$ by human annotators from HH-RLHF. Thibault Sellam () proposed a cosine similarity based on the BLUERT embedding.
These similarities are denoted as $\text{sim}_{\pi_{\theta}^{*}}(\hat{y},y_{w})$ for the model aligned with weak LLM feedback and $\text{sim}_{{\pi}_{h}^{*}}(\hat{y},y_{w})$ for the model aligned with human feedback, represented on the $x$-axis and $y$-xis respectively. We observe a moderate correlation between the two across all test samples from HH-RLHF.

Additionally, we examine the
correlation between the model $\pi_{\theta}^{*}$ aligned with weak LLM feedback and the weak LLM itself $\pi_{w}^{*}$ on the right of Figure 6. This analysis helps reveal whether aligning a model under weak feedback could risk imitating the errors inherent in a weak LLM. We observe that the correlation between $\pi_{\theta}^{*}$ and $\pi_{w}^{*}$ (with $R^{2}=0.4888$) is relatively weaker compared to that between $\pi_{\theta}^{*}$ and ${\pi}_{h}^{*}$ (with $R^{2}=0.5789$). These results suggest that the model $\pi_{\theta}^{*}$ not only aligns more closely with $\pi_{h}^{*}$, but also effectively extrapolates beyond imitating the weaker teacher.

Figure: Figure 6: Left: Agreement between models using weak LLM feedback ($\pi_{\theta}^{*}$) *vs.* human feedback ($\pi_{h}^{*}$). Right: Agreement between the model ($\pi_{\theta}^{*}$) aligned with weak LLM feedback and the weak LLM itself ($\pi_{w}^{*}$).
Refer to caption: x6.png

#### Ablation on the impact of SFT.

Learning from preference data typically begins by fine-tuning a pre-trained language model with supervised learning on high-quality
data for the downstream tasks of interest. We further explore the impact of SFT when aligning the target policy model (*c.f.* Equation 6). We utilize the OPT-125M as the weak LLM to provide feedback and the OPT-1.3B as the student model, conducting our experiments on the HH-RLHF dataset. For the model $\pi_{\theta}^{*}$ aligned with weak LLM feedback and the model $\pi_{h}^{*}$ aligned with human feedback, we compare the following two settings: (1) DPO (w/o SFT): Directly using the pre-trained model as the reference model or initialization. (2) DPO (w SFT): Use fine-tuned model $\pi_{\theta}^{\text{SFT}}$ as a reference model for DPO training. Compared to directly using the untuned base model as a reference model, performing SFT enhances
the model’s ability to generate desired responses to questions.

**Table 10: Ablation on the impact of supervised fine-tuning (SFT) as initialization for DPO. We report the average gold reward. Numbers in brackets denote improvement relative to the pre-trained model (OPT-1.3B) without any fine-tuning.**
| Method | Weak LLM feedback | Human feedback |
| --- | --- | --- |
| DPO (w/o SFT) | 3.83 (+1.61) | 3.77 (+1.55) |
| DPO (w SFT) | 4.84 (+2.61) | 4.63 (+2.41) |

#### Ablation on the KL coefficient ( β 𝛽 \beta italic_β ) for weak LLM feedback for alignment.

The $\beta$ parameter in Equation 3 functions as the KL coefficient during DPO training, with higher values indicating more stringent regularization. We employ OPT-125M as the weak LLM to provide feedback and OPT-1.3B as the student model, performing alignment using varying $\beta$ values: $\{0.05,0.1,0.2,0.3,0.5\}$ on the HH-RLHF dataset. As shown in the left of Figure 7, the gold reward for the model alignment with weak LLM feedback ($\pi_{\theta}^{*}$) is high under relatively small $\beta$ values such as 0.05 and 0.1, and starts to decline as $\beta$ increases. A similar trend is observed for the model trained with human feedback.

Figure: Figure 7: Ablation on the KL Coefficient ($\beta$) for alignment with weak LLM feedback and human feedback.
Refer to caption: x8.png

#### Dataset division with different random seeds.

We randomly partition the entire training set into a labeled dataset, denoted as $\mathcal{D}_{l}$, and an unlabeled dataset, $\mathcal{D}_{u}$, using a random seed of 22. To ensure that our conclusions are insensitive to the data division, we also utilized five additional random seeds to perform the experiments. Same as our main experiment, we employ OPT-125M as the weak LLM and OPT-1.3B as the student model on the HH-RLHF dataset. Across 5 random runs, the average gold rewards for the $\pi_{\theta}^{*}$ and $\pi_{h}^{*}$ are 4.81 and 4.67, respectively, with variances of 0.12 and 0.14. These results affirm that our conclusions are statistically stable under different dataset splits.

#### Understanding the different impact between weak LLM feedback vs. random noise.

Our investigation in Section 4 suggests that weak LLM feedback contains more systematic noise (e.g., errors primarily occur between choices that are subtly different). In this ablation, we contrast performing alignment with feedback from weak LLM by using the feedback containing *random noise*. Adopting OPT-125M as the weak LLM and OPT-1.3B as the student model, we design three controlled settings for this ablation:

- •
$\mathcal{D}_{\text{random}}^{\text{match}}$: Randomly sample 60.6% of the samples from $\mathcal{D}_{u}$, and assign human feedback, i.e., $(r_{w}(x,y_{w})>r_{w}(x,y_{l}))$. This set has the same size as our weakly supervised set $\mathcal{D}_{\text{weak}}^{\text{match}}$.
- •
$\mathcal{D}_{\text{random}}^{\text{mismatch}}$: Use the remainder 39.4% samples from $\mathcal{D}_{u}$, and assign preference labels *opposite to the human feedback*, i.e., $(r_{w}(x,y_{w})<r_{w}(x,y_{l}))$.
- •
$\mathcal{D}_{\text{random}}$: The union of $\mathcal{D}_{\text{random}}^{\text{match}}$ and $\mathcal{D}_{\text{random}}^{\text{mismatch}}$.

As demonstrated in Table 11, the student model trained with the match set $\mathcal{D}_{\text{random}}^{\text{match}}$ outperforms the model trained with $\mathcal{D}_{\text{random}}$ by 0.66 in terms of gold reward. This suggests that removing random noise in preference datasets can markedly enhance alignment performance. Conversely, the performance of the model trained with $\mathcal{D}_{\text{random}}^{\text{mismatch}}$ exhibits a gold reward that is significantly lower (1.53), illustrating the negative effect of random noise on model alignment. These findings highlight the difference between random noise and the “noise” from weak LLM feedback.

**Table 11: Effect of purifying random noise. Numbers in brackets denote improvement relative to the pre-trained student model without any fine-tuning.**
|  | % matching human feedback | Gold reward of $\pi_{\theta}^{*}$ |
| --- | --- | --- |
| $\mathcal{D}_{\text{random}}$ | 60.6% | 3.91 (+1.69) |
| $\mathcal{D}_{\text{random}}^{\text{match}}$ | 100% | 4.57 (+2.35) |
| $\mathcal{D}_{\text{random}}^{\text{mismatch}}$ | 0% | 1.53 (-0.69) |

#### Results on additional weak LLM feedback.

To validate our conclusion that weak LLM provides feedback comparable to human feedback, we extend our analysis beyond the specific case of OPT-125M. We incorporated GPT-Neo-125M (Gao et al., ) and Pythia-160M (Biderman et al., ) as additional weak LLM to provide feedback. We use OPT-1.3B and Mistral-7B as student models to be aligned. Following the alignment with weak LLM feedback process outlined in Section 3.1, we assess the generalizability of our conclusion. As shown in Table 12, models aligned with feedback from weak LLMs GPT-Neo-125M and Pythia-160M achieved performance comparable to the model aligned with human feedback. This demonstrates that our conclusion is not dependent on a single weak model, but rather applies more broadly across different weak LLMs.

**Table 12: Results on using different weak LLM supervisors: OPT-125M, GPT-Neo-125M and Pythia-160M.**
|  | Human feedback | OPT-125M | GPT-Neo-125M | Pythia-160M |
| --- | --- | --- | --- | --- |
| OPT-1.3B | 4.63 | 4.84 | 4.55 | 4.87 |
| Mistral-7B | 7.19 | 7.90 | 7.27 | 8.01 |

## Appendix D Limitations and future work

#### Limitations.

Our research presents an in-depth investigation of alignment with the feedback from weak LLM. We delve deeply into the reasons underlying this phenomenon. However, our study is not without limitations, which we aim to address in subsequent research. Our focus has predominantly been on empirical experiments, leaving the theoretical underpinnings of our findings less explored. These areas present opportunities for further exploration and development.

#### Implications for future alignment research.

As we look toward the future of AI alignment, several recommendations emerge from our study that could further refine the practice and enhance the reliability of alignment methodologies. First, integrating hybrid feedback systems that combine human insights with AI-generated feedback could leverage the strengths of both, minimizing the limitations inherent in each approach. Secondly, it is crucial to develop more sophisticated metrics for evaluating alignment quality that go beyond traditional accuracy or performance metrics, to capture the nuanced understanding and generative capabilities required in real-world applications. Finally, exploring the ethical implications of AI-generated feedback and ensuring that these systems adhere to ethical guidelines is vital. By addressing these areas, the field can move towards more effective, efficient, and ethically responsible AI alignment strategies that are capable of supporting the safe integration of AI systems into society.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Why the weaker LLM to provide alignment feedback is referred to as a task-specific model? It is trained with the HH-RLHF dataset, which is a general domain dataset for LLM alignment.
- What does the “pre-trained strong model” mentioned in Line 371 specifically refer to? Is is the LLM that only undergone pre-training without fine-tuning or alignment, or is it the trained with positive examples?
- I have concern about the result that GPT-4 underperforms compared to smaller models. Could you please provide more reasonable explanations?
soundness: 3
strengths: - The research topic in this paper is critical and popular. Given the high costs associated with human and large model annotations, exploring the potential of weaker LLMs in providing alignment feedback is a timely and necessary endeavor.
- This paper is well-written and easy-to-follow, with an introduction to associated alignment background techniques.
- A dedicated discussion section highlights the differences between this work and the most relevant prior studies.
summary: This paper concentrates on the critical research topic of large language model (LLM) alignment, especially related to super-alignment. Considering existing mainstream alignment approaches rely on paired human preference data, which highly relies on human or powerful LLMs for annotation, using a weak LLM to provide alignment feedback becomes a promising alternative. This paper aims to bridge that gap of lacking a systematic evaluation and understanding of weak LLM’s capability of generating feedback of alignment. The authors first formalize a workflow to conduct semi-supervised alignment training of a model and conduct evaluation. Based on this uniformed experimental setting, the paper assesses the effectiveness of models of various scales in providing alignment feedback and finds that weaker LLMs can indeed play an effective role in alignment tasks. Additionally, an in-depth analysis is conducted on the quality of feedback provided by weaker LLMs.
weaknesses: - More comprehensive experiments are required to validate the reliability of the primary conclusions in this paper, including but not limited to the following aspects.
    + The most relevant baselines with the idea of RLAIF. Beyond that the original RLAIF methods rely on large LLMs to provide feedback, it is crucial to delineate differences between the proposed training pipeline and RLAIF. Additionally, a comparison with RLAIF employing weaker LLMs as the reward model would strengthen the findings.
    + Human annotation data of different quality levels. Previous studies [1,2] have indicated that the HH-RLHF dataset may contain noise in its human preference labels. Comparing with higher-quality human data, such as a revised version of HH-RLHF, would enhance the robustness of the results.
    + LLMs with larger sizes.
    + GPT-4 with more advanced prompts or instructions to provide alignment feedback.

[1] Wang, Binghai, et al. "Secrets of rlhf in large language models part ii: Reward modeling." arXiv preprint arXiv:2401.06080 (2024).

[2] Liu, Yan, et al. "Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment." arXiv preprint arXiv:2409.19024 (2024).

- The issue of overfitting to a specific dataset in smaller models is not discussed. Since small LLMs are prone to overfitting to specific datasets but LLMs alignment target diverse domains, you should consider the generalization of the weaker LLM to provide alignment across domains. If it shows very weak generalization, it may need annotation data in each domain, which also requires a large human cost.
- More diverse and more reliable evaluation metrics should be considered.
- The actionable insights derived from the in-depth analysis are sparse. The main finding is that issues persist in the original annotated data, and both weaker LLMs and large models like GPT struggle to handle noisy data with high confidence.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the consistency of weak LLM feedback assessed across multiple prompts or model runs?

2. How do the authors ensure that the gold reward model and GPT-4 are reliable in evaluation? Are other potential evaluation metrics considered?

3. The paper listed many LLM alignment methods in the related work. Did the authors compare their method with these alignment techniques?
soundness: 4
strengths: 1. This paper proposes a cost-effective alignment framework that reduces computational costs and reliance on extensive human feedback,  using weak LLMs that have significantly fewer parameters than high-capacity models.

2. The study demonstrates that weak LLMs can perform as well as, and in some cases better than, human annotators in generating preference feedback for alignment through extensive experiments. For example, by demonstrating similar alignment success across different model families and sizes, the study provides a strong case for the generalizability of the weak LLM approach, suggesting that high alignment quality can be achieved without the latest or most powerful models.
summary: This paper introduces an alignment approach using weak large language models (LLMs) that balances cost and efficiency. The authors find that weak LLMs can generate feedback comparable to or even better than human feedback and stronger LLMs and propose the method to employ a semi-supervised framework where a weak LLM, trained on a small labeled dataset, provides preference feedback for a larger unlabeled dataset, reducing reliance on human annotation.
weaknesses: 1. The paper lacks a thorough analysis of feedback consistency over repeated evaluations. For example, when weak LLM and human preferences conflict, the study does not adequately explore the stability of weak LLM feedback. A quantitative analysis of consistency across multiple runs or various weak models would be helpful to assess whether the weak LLM’s feedback is repeatable and reliable. 

2. The evaluation metrics are only based on 2 models, the gold reward model and GPT-4. If the models are biased, the evaluation given by them would be unreliable and it could lead to skewed assessments that do not accurately reflect the quality of weak LLM alignment.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1.See weakness 2.

2.See weakness 3. Can this weak-to-strong approach be evaluated for more complex tasks? Like a math task or a code task?

3.In Contribution 2, the authors mentioned that using weak LLMs to provide preference labels for alignment can match or even surpass the performance of using full human feedback with the same training data scale. Why is it that training on data labeled by weak LLMs on unlabeled data can outperform the training set labeled by humans? Is it because the human-labeled training set contains some incorrect labels and its quality still needs improvement?
soundness: 3
strengths: 1.The paper introduces a framework that leverages weak LLMs for alignment, which offers a promising alternative to existing methods that rely of either extensive human labor or expensive computational resources.

2.The paper conducts comprehensive evaluation of weak LLM feedback across various model scales and families. The result consistently demonstrate the effectiveness of this approach, highlighting its potential for practical application.

3.The paper is well-organized, with clear sections and logical flow. The authors effectively communicate the research objectives, methodology, and findings.

4.The research has significant practical implications for the development and deployment of aligned LLMs. Utilizing weak LLM feedback can reduce costs and improve efficiency, making alignment more scalable and accessible.
summary: The paper explores the potential of using weak large language models (LLMs) for AI alignment. It proposes a new framework that leverages weak LLMs to provide feedback, thereby reducing reliance on costly human labor and computational resources. Experimental results show that even weak LLMs with as few as 125 million parameters can offer feedback comparable to human feedback. The paper also analyzes the quality of the feedback from weak LLMs and finds that it can surpass human judgment in some cases, although there is uncertainty in fine differentiation. The study offers a new perspective for AI alignment and demonstrates that weak LLM feedback is a promising, scalable and sustainable solution.
weaknesses: 1.In this paper， the authors propose a framework that utilizes weak LLMs for alignment, striving to strike a balance between RLHF and RLAIF, it appears that the weak-to-strong approach and methodology in the paper are still quite similar to Burns et al. (2024). Burns et al. (2024)’s study focus on reward modeling and binary classification, where the outputs in this paper are either scalar or categorical labels. Your study is closely tied to alignment and involves a more complex output space. However, it seems that the two are consistent in their methodological approach. Therefore, I think that there are still some limitations in terms of novelty. I hope you can explain more about the difference between the two works.

2.The paper points out that even advanced LLMs like GPT-4 may show feedback inconsistency when the quality difference between two responses is subtle. In the case where it is uncertain whether gold RM is stronger than GPT-4, when using gold RM for evaluation, is it also possible to make incorrect judgments about two responses with similar quality, leading to bias in the evaluation？Additionally，When using GPT-4 for evaluation, although the prompt already mentions and ensures that the order of the responses should not affect its judgment, in actual experiments, it is difficult to avoid the model's preference caused by the order. A fairer approach would be to swap the positions of the two responses and conduct the experiment again. I believe that conducting the experiment twice with different positions would result in a more accurate evaluation.

3.While the paper demonstrates the effectiveness of weak LLM feedback across alignment task, it would be beneficial to investigate its applicability to more complex tasks.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - In the second paragraph of the introduction (lines 40-50), expliciting the fact that what is studied is a weaker _supervised_ LLM might help the reader understand the middle-ground nature of this approach. Otherwise it may not be fully clear why weaker LLMs are positioned in between human annotators and high-capacity LLM annotators.
- In Figure 1, it is a bit unintuitive to show A < B in all the annotations, as it seems that answer A is a better (at least more helpful) answer than B in the given example. This example is mentioned later in Table 3, as a case where human annotation contradicts the gold reward model.  But using this (potentially incorrect) human annotation in Figure 1 when describing the approach might be confusing for the reader.
- The construction of $\mathcal{D}_{weak}$ in Section 3.1 is done by first learning a generation model through DPO and then use the implicit reward model induced by DPO to annotate the unlabeled dataset $\mathcal{D}_u$. It seems unnecessary to train a generation model as only the reward model is needed for the annotation. Given this, it is then unclear why DPO was used in this step instead of simply training a traditional reward model by following the standard RLHF protocol (but without the unneeded PPO stage). The argument given lines 139-140 that training with RLHF requires the use of multiple models is only valid for the PPO stage (when there are both the LLM policy and the reward model to handle), but not in the reward modeling stage which only consists of an LLM augmented with a regression head. It would have been interesting to study the impact of the DPO reward model vs the RLHF reward model.
- In Section 3.2, one of the metric introduced is the gold reward. However it is not entirely clear why a large auxiliary reward model can be considered as "gold". Does it just have to be a "recognized" reward model, or are there any requirements on the training data used for this reward model?
- For the GPT-4 win-rate metric, it does not seem necessary to ask for a numeric rating to be assigned to the two compared responses. Would it not be sufficient to just ask GPT-4 to identify the better response out of the two? This seems to be a more standard approach (see, e.g., Prometheus 2: https://arxiv.org/abs/2405.01535).
- What is the ratio $|\mathcal{D}_{l}|/|\mathcal{D}|$ used for the main experiments in Section 3.3? The ablations in Section 3.4 do study the results when this ratio is varied, but I could not find the default ratio that is used in other experiments.
- The x axis label of Figure 7 in Appendix C seems to be incorrect: it should be $\beta$ instead of "student model size".
soundness: 4
strengths: - The paper is clear, polished and reads very well.
- The experiments are comprehensive and convincing.
- The proposed idea of using a weak supervised LLM as annotator is likely to have a significant impact on future LLM alignment practices.
summary: This paper investigates the use of a weak supervised LLM to annotate preference data, as a middle ground between labor-intensive, costly human annotations and computationally expensive annotation based on high-capacity LLMs such as GPT-4. More specifically, the weak LLM (with approx. 100M parameters) is fine-tuned with DPO on a set of human labeled preference data and then its associated implicit reward model is applied to an unlabeled set to predict preferences. The resulting completed preference dataset is ultimately used to align a student LLM, whose performance enables measuring the quality of the annotations provided by the weak LLM.

The proposed weak LLM-based annotator is evaluated through multiple experiments where it is compared against both human annotators and stronger LLM annotators. It also includes experiments with weak LLMs and students models from different model families. Overall, the results show that using weak LLM-based annotations lead to similar or better performance compared to using human annotations or prompted high-capacity LLMs.
weaknesses: - The choice of using DPO fine-tuning to obtain a weak LLM-based annotator is not fully intuitive and lacks motivations, as only a reward model is needed for the preference annotation -- a model with generation capabilities is unnecessary for this (more details on this point in the "Questions" section).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: please see above
soundness: 3
strengths: + ailgnment with synthetic feedback/preference is an important research question
+ the experiments are comprehensive
summary: This work proposes to investigate whether "weak" language models could serve as a reward model/preference generator for alignment. Experiments demonstrate that they can to varying extents and match some of the benefits of HF/AIF.
weaknesses: - On the high-level framing, why is "weak LLM as feedback" positioned as the "middle ground" (line 42) between human feedback and AI/LM feedback? You could say that weak LLM feedback is still AI feedback, just with a smaller/weaker model, and not necessarily offers something strikingly new that make it "middle ground" in the "spectrum".

- By "weak LLM" the authors actually mean "small LLM": for example, "while a model like GPT-4 might have trillions of parameters,
a weak LLM might only have hundreds of millions or even fewer" (lines 44-45). The experiments focus on size too by employing OPT models, which are small compared to today's standards. However, "small LLM" for feedback is not necessarily "weak": if you take a small model and train it on large quantities of quality human preference data, it might become a moderate reward model. It would be great to disentangle "weak" and "small" in the narrative and also in the experiments.

- Section 2 is a bit trivial and takes up too much space. These math primers on DPO/PPO were not particularly helpful in later sections too. It would be great to assume readers of an "alignment" paper are already familiar with the basics, move them to the appendix, and use the space for analysis/results.

- I see that there is some investigation into the quality of preferences by the "weak" OPT model. Maybe something like RewardBench might offer a more rigorous quantitive evaluation.

- I would argue that things like self-rewarding language models [1] did things similar to "feedback from a weak model", albeit not as weak as OPT. Starting from a "weak-ish" models that are not good in both generating and evaluating, they show it is possible to jointly improve both. I wonder if the uniqueness of this paper's contribution might be weakened a bit by works like this.

- Ultimately, some might say that this work is a fine analysis paper on a niche technical design choice and does not offer something greater: taking RLAIF pipelines, changing the "AI" model part to a smaller model, and run some evaluation. Please highlight additional contribution if any.

[1] Yuan, Weizhe, et al. "Self-Rewarding Language Models." ICML 2024.

Meta-review: This work focuses on LLM alignment and proposes a method that employs a semi-supervised framework. In this framework, a weak LLM, trained on a small labeled dataset, provides preference feedback for a larger unlabeled dataset. Experiments show its effectiveness in alignment tasks and validate the quality of feedback provided by weak LLMs.

Decision: Accept (Poster)