Bayesian WeakS-to-Strong from Text Classification to Generation

## Abstract

Abstract Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them weakly. Weak-to-Strong mimics such a scenario where weak model supervision attempts to harness the full capabilities of a much stronger model.
This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS-to-Strong generalization. Furthermore, we extend the application of WeakS-to-Strong from text classification tasks to text generation tasks where more advanced strategies are investigated for supervision.
Moreover, direct preference optimization is applied to advance the student model’s preference learning, beyond the basic learning framework of teacher forcing. Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superalignment. 1 1 1 Code will be available upon acceptance.

## 1 Introduction

With the increase in computing power and the amount of training data available, the capabilities of large language models (LLMs) have been continuously brought closer to humans in many aspects. Despite their impressive performance, the preferences and values of pre-trained LLMs do not always align with humans, and dedicated approaches are needed to tackle the problem. Based on large-scale instruction datasets, supervised finetuning (SFT) encourages LLMs to follow human instructions more strictly and respond more safely (Wei et al., ). Reinforcement learning (RL) is commonly applied to such alignment. By collecting model output values and the corresponding human feedback, the model can be finetuned by RL to avoid generating undesirable outputs (Ziegler et al., ; Bai et al., ; Ouyang et al., ; Nakano et al., ; Askell et al., ).

Since no current model has yet surpassed human intelligence, alignment methods, such as SFT and RL from human feedback (RLHF), remain effective. However, it is worthwhile considering future scenarios where artificial intelligence (AI) might surpass human intelligence in all aspects. Would the current alignment methods still be effective for such super AI models? How could humans supervise the super AI? To simulate this future scenario, an analogy situation is designed that downgrades both sides: using a weak model to simulate humans and a strong model to simulate future super AI (Burns et al., ), which is termed as superalignment. It has been demonstrated that adding a simple auxiliary loss can achieve effective Weak-to-Strong generalization, even if the weak model’s supervision contains many errors, which offers hope of achieving superalignment. Nonetheless, this is just the beginning of exploring along the path of Weak-to-Strong.

This paper extends the discussion on Weak-to-Strong in two directions.
First, given the inherent capability gap between the weak model and the strong model, we propose using an ensemble of multiple weak models to improve the quality of weak supervision, which is called WeakS-to-Strong. This also accounts for the scenario where human opinions might diverge in tasks without a commonly accepted standard. Several approaches have been studied to effectively leverage the diversity of different weak models, and we adapt a Bayesian approach referred to as evidential deep learning (EDL) (Sensoy et al., ) to better estimate broader human preferences by learning a prior distribution over the weak labels produced by the weak models.
Furthermore, the Weak-to-Strong task was primarily studied for text classification tasks (Burns et al., ).
This paper extends the scope to text generation and compares three different approaches, Naive Multi-Weak, Joint Decoding and Bayesian Multi-Weak, as shown in Figure 1. The proposed Bayesian WeakS-to-Strong approach is demonstrated effective for both classification and generation.
To better align with human preferences, a variant of direct preference optimization (DPO) (Rafailov et al., ) called conservative DPO (cDPO) (Eric, ) is used to finetune the strong model further on RL principles.

Our main contributions are summarized as follows.

- •
We proposed Bayesian WeakS-to-Strong which largely improves the quality of weak supervision and recovers the performance of the strong model.
- •
We propose to generalize both Weak-to-Strong and WeakS-to-Strong from text classification to generation tasks, extending their scope from content regulation to content generation.
- •
When applied to text generation, a token-level probability estimation is proposed to achieve soft labels for strong model training. We also propose the modified DPO algorithm under the Bayesian WeakS-to-Strong framework to further improve text generation performance.

## 2 Related Work

AI Alignment. Aligning LLMs with human preferences has been a long-standing goal. Instruction tuning uses extensive datasets to improve LLMs’ adherence to human instructions (Wei et al., ). RL allows LLMs to learn what types of responses humans prefer or dislike, with proximal policy optimization (PPO) being an effective RL method first applied to LLMs and becoming part of the standard RLHF process (Ziegler et al., ; Bai et al., ; Ouyang et al., ; Nakano et al., ; Askell et al., ). However, PPO training can be unstable, leading to the development of DPO (Rafailov et al., ). Given the high cost of obtaining human preference data, researchers are now exploring the use of LLMs to simulate human preferences, provide feedback, and finetune models (Lee et al., ; Bai et al., ; Gulcehre et al., ).

Variability in Human Opinions. In the process of aligning AI with human preferences, it is important to consider the inconsistency of human preferences (Liu et al., ), which often leads to multi-label problems. Previously, many approaches used simple methods like voting, aggregation, and averaging to handle multi-labels (Davani et al., ; Munos et al., ; Paun & Simpson, ; Prabhakaran et al., ). However, these methods do not effectively capture the preference differences of individual annotators included in the multiple labels. To better estimate the diversity of human preferences, Bayesian principles have been introduced. Deep learning models can be used to predict prior distributions, which are considered to produce the multiple available labels to estimate a broader range of human preferences (Sensoy et al., ; Wu et al., ; ).

Weak-to-Strong. The goal of Weak-to-Strong is to use a weak model to better supervise a strong model.
OpenAI demonstrated that adding auxiliary confidence loss from the strong model itself can significantly improve the Weak-to-Strong performance (Burns et al., ).
Following OpenAI’s work, several studies emerged to introduce multiple weak models, used either in series or parallel, to improve the quality of supervision provided by the weak models (Liu & Alahi, ; Sang et al., ).
Early model ensemble methods like Adaboost (Freund & Schapire, ) and Bootstrap aggregating (Leo, ) were explored in these works.
Furthermore, confidence scores are incorporated to help the strong model assess the supervision quality provided by the weak models (Guo et al., ) and the weak model can be directly used to modify the output of the strong model (Ji et al., ).

## 3 WeakS-to-Strong Methodology

Figure: Figure 1: An overview diagram of the three ensemble approaches: (a) Naive Multi-Weak: directly learn all weak labels produced by weak models, (b) Joint Decoding: weak models collaboratively determine one single target, (c) Bayesian Multi-Weak: learn a prior distribution over weak labels.
Refer to caption: x1.png

### 3.1 Preliminary: Weak-to-Strong

The Weak-to-Strong pipeline (Burns et al., ) involves three steps: (i) create a weak supervisor by finetuning a small pre-trained model on ground-truth labels; (ii) train a strong student model $\bm{f_{\Lambda}}$ with weak supervision by finetuning a pre-trained LLM using “weak labels” generated by weak supervisors, where $\bm{\Lambda}$ is the parameters of the strong model; (iii) finetune the large pre-trained model directly using ground-truth labels which serve as the ceiling.

To leverage the superior generalization capabilities and prior knowledge of the strong model, a loss function with auxiliary confidence loss is proposed (Burns et al., ):

$$ $\mathcal{L}=(1-\gamma)\cdot\mathcal{L}_{\text{CE}}(\bm{f_{\Lambda}}(\bm{x}), \bm{y}_{w})+\gamma\cdot\mathcal{L}_{\text{CE}}(\bm{f_{\Lambda}}(\bm{x}),\bm{ \hat{f}_{\Lambda}}(\bm{x}))$ (1) $$

where $\bm{y}_{w}$ represents the weak label from weak model, $\bm{\hat{f}_{\Lambda}}(\bm{x})$ refers to the predicted class of strong model given input $\bm{x}$, and $\mathcal{L}_{\text{CE}}(\cdot,\cdot)$ denotes the cross-entropy loss. The second term is an (optional) auxiliary self-training loss designed to increase the confidence of the strong model in itself. The weight of the second loss $\gamma$ linearly grows up from 0 to a pre-defined hyper-parameter $\gamma_{\text{max}}$, which gradually reduces the weight on the weak labels and increases the weight on self-training when the number of training steps increases.

### 3.2 Extending Weak-to-Strong with multiple weak models

Although it has been shown that the Weak-to-Strong approach can recover part of the strong model’s performance (Burns et al., ), the errors in weak labels limit the performance of Weak-to-Strong generalization.
In response to this problem, we propose to leverage the complementarity of the error patterns of multiple weak models using an ensemble strategy, which is referred to as WeakS-to-Strong.

A naive approach to implementing an ensemble of multiple weak models is to calculate the loss for each weak label respectively and then average these losses. An improvement of this approach is to take a weighted sum instead of a simple average:

$$ $\mathcal{L}_{\text{Naive}}=\sum_{i=1}^{N}\lambda_{i}\mathcal{L}_{\text{CE}}( \bm{f_{\Lambda}}(\bm{x}),\bm{y}_{w}^{(i)})),$ (2) $$

where $N$ is the number of weak models, $\bm{y}_{w}^{(i)}$ is the $i$th weak label produced by the $i$th weak model, and $\lambda_{i}$ is a pre-defined weight of the loss regarding the the $i$th weak model.
This approach is referred to as a Naive Multi-Weak system in the rest of the paper (as illustrated in Figure 1(a)), which is treated as one of the baselines.

### 3.3 Bayesian WeakS-to-Strong

For superalignment, multiple weak models are used to mimic the subjective preferences of multiple humans, which can be considered as observations drawn from an underlying distribution of the opinions of all humans.
The naive approach described in Section 3.2 solely relies on these observations. The number of observations (human annotations or weak labels) is often very limited due to the considerable cost of hiring a new human annotator or training a new weak model.
Such a limited number of observations may not result in a good approximation of the true human opinion distribution. Having biased preferences or values is particularly unacceptable in the safety domain and can cause a failure of superalignment.
Therefore, we propose a Bayesian WeakS-to-Strong approach based on EDL (Sensoy et al., ) to estimate the human opinion distribution based on the weak labels.
Figure 1(c) illustrates the framework where three weak models are involved.

Consider a weak label from the $i\text{th}$ weak model $\bm{y}_{w}^{(i)}$, which is a one-hot vector with $y_{w}^{(i,k)}$ being one if it belongs to class $k$ and zero otherwise. $\bm{y}_{w}^{(i)}$ is sampled from a categorical distribution of weak labels $\operatorname{Cat}(\bm{\pi})$, where each component ${\pi}_{k}$ corresponds to the probability assignment over the possible classes $\bm{y}_{w}^{(i)}\sim\mathrm{P}(\bm{y}|\bm{\pi})=\operatorname{Cat}(\bm{\pi}).$ EDL places a Dirichlet prior over the categorical distribution representing the probability of each categorical probability assignment, hence modelling second-order probability $\bm{\pi}\sim\mathrm{p}(\bm{\pi}|\bm{\alpha})$, where $\bm{\alpha}$ is the hyperparameter of the Dirichlet prior. The strong model $\bm{f_{\Lambda}}$ is trained to predict $\bm{\alpha}$ for each input by minimizing the negative log-likelihood of sampling $\bm{y}_{w}^{(i)}$ given the predicted Dirichlet prior:

$$ $\mathcal{L}_{\text{NLL}}^{(i)}=-\log\int\mathrm{P}(\bm{y}_{w}^{(i)}|\bm{\pi}) \mathrm{p}(\bm{\pi}|\bm{\alpha})\mathrm{d}\bm{\pi}=\sum_{k=1}^{K}y_{w}^{(i,k)} (\log(\alpha_{0})-\log(\alpha_{k})),$ (3) $$

where $K$ is the number of classes, $\alpha_{0}=\sum_{k=1}^{K}\alpha_{k}$ is the Dirichlet strength, and $y_{w}^{(i,k)}$ is the $k^{th}$ value of label $\bm{y}_{w}^{(i)}$.
Following Sensoy et al. (), a regularization term $\mathcal{L}_{\text{REG}}^{(i)}$ (see Appendix G for details) is added to calibrate uncertainty estimation, resulting in the EDL loss $\mathcal{L}_{\text{EDL}}^{(i)}=\mathcal{L}_{\text{NLL}}^{(i)}+\lambda_{\text{
EDL}}\mathcal{L}_{\text{REG}}^{(i)}$ where $\lambda_{\text{EDL}}$ is the coefficient.

Apart from the class predicted by the weak models, the confidence of weak models is also incorporated for better distribution estimation. Let $(p_{1}^{(i)},\dots,p_{K}^{(i)})$ be the probability assignment predicted by the $i^{th}$ weak model, the EDL loss for each class is calculated based on the predicted probability assignment for each weak model and then combined in the same way as in Eqn. (2). That is,

$$ $\mathcal{L}_{\text{EDL}}(\bm{f_{\Lambda}}(\bm{x}),\{\bm{y}_{w}^{(i)}\}_{i=1}^{ N})=\sum_{i=1}^{N}\lambda_{i}\sum_{k=1}^{K}p_{k}\mathcal{L}_{\text{EDL}}^{(i)} (\bm{f_{\Lambda}}(\bm{x}),\hat{\bm{y}}_{w}^{(i)}))$ (4) $$

where $\hat{\bm{y}}_{w}^{(i)}$ is the predicted class, and $\lambda_{i}$s are hyperparameters set to the same values as used in the Naive Multi-Weak approach. As a result, the auxiliary confidence loss described in Eqn. (1) is adapted for Bayesian WeakS-to-Strong as follows:

$$ $\mathcal{L}=(1-\gamma)\cdot\mathcal{L}_{\text{EDL}}(\bm{f_{\Lambda}}(\bm{x}), \{\bm{y}_{w}^{(i)}\}_{i=1}^{N})+\gamma\cdot\mathcal{L}_{\text{EDL}}(\bm{f_{ \Lambda}}(\bm{x}),\bm{\hat{f}_{\Lambda}}(\bm{x})).$ (5) $$

In the term of $\mathcal{L}_{\text{EDL}}(\bm{f_{\Lambda}}(\bm{x}),\bm{\hat{f}_{\Lambda}}(\bm{x
}))$, the class index predicted by the strong model $\bm{\hat{f}_{\Lambda}}(\bm{x})$ is used as the target. That is to say, the predictions of the strong student model are applied as part of the distribution estimation along with the weak label.

## 4 WeakS-to-Strong for Sequence Generation

### 4.1 Probability estimation for weak sequence labels

Figure: Figure 2: The process of transforming per-token confidence scores from the sequence tokenized by the weak model to the sequence tokenized by the strong. The word “hello” is used as an example. Stage 1: The words and word scores are obtained from the weak model wordpieces and their scores. Stage 2: The words are tokenized by the strong model tokenizer, and the tokenized sequences are fed into the strong model to obtain the strong model predicted probability (denoted as confidence) for each token $s_{i}$. This strong model confidence is then used to split word scores into target wordpiece probabilities $P(s_{i})$ while keeping the probability of the word unchanged. Stage 3: The obtained target probability is transformed into the label. Probabilities of other categories are calculated by scaling the strong output distribution using $P(s_{i})$.
Refer to caption: x2.png

To enable the strong model to directly generate trustworthy content rather than only being trained to understand whether the content is trustworthy or not, we propose to extend the scope of Weak-to-Strong from text classification to text generation.
The key challenge of directly applying the Weak-to-Strong loss to the sequence generation task is the token-level soft labelling for the target sequence. As the tokenizers are different between weak and strong models, it is infeasible to obtain a one-to-one mapping from weak model output distributions to each token in the target sequence.

To obtain the soft label $\bm{y}_{w}$ for the strong model using weak model output probabilities and bridge the gap caused by different tokenizers, we use words as an intermediary, following the equation

$$ $\mathrm{P}(W)=\mathrm{P}(w_{2}|w_{1})\mathrm{P}(w_{1})=\mathrm{P}(s_{2}|s_{1}) \mathrm{P}(s_{1}),$ (6) $$

where we use a word containing two wordpieces from both weak and strong tokenizers as an example, and $w_{1},w_{2}$ and $s_{1},s_{2}$ are both token strings that can form word $W$, which are generated by the tokenizers of the weak and strong models respectively.
Figure 2 shows an example of the process in three stages.
In stage 1, the per-token output probabilities of weak models are obtained when generating output sequences. The probabilities of wordpieces in a word are then multiplied together to obtain the score of word $W$, following Eqn. (6).

In stage 2, the word probability is used to assign probabilities to tokens from the strong model tokenizer. In the process of training the strong model via teacher-forcing, the model gives a probability to each token by applying softmax to the output logits. This probability can be seen as the model’s confidence in predicting the target token, where we assume that the weak model and strong model have similar confidence for wordpiece tokens in similar positions. This allows us to to obtain the actual assignment of scores to each target token $s_{i}$, instead of assigning equal probabilities to all tokens involved.
Taking an example of splitting a word $W$ into two target wordpieces $s_{1}$ and $s_{2}$, the decomposition can be approximated by

$$ $\mathrm{P}(s_{1})=\frac{e^{C_{s}(s_{1})}}{e^{C_{s}(s_{1})}+e^{C_{s}(s_{2})}} \mathrm{P}(W),\qquad\mathrm{P}(s_{2})=\frac{e^{C_{s}(s_{2})}}{e^{C_{s}(s_{1})} +e^{C_{s}(s_{2})}}\mathrm{P}(W),$ (7) $$

where $C_{s}(s_{i})$ is strong model confidence at the step predicting wordpiece $s_{i}$ that is the maximum probability in the strong model output distribution at that step in practice.
In this way, lower target probabilities are allocated to tokens with lower strong model confidence, while higher probabilities are allocated to tokens with higher strong model confidence.

After obtaining the probability of the target token of the strong model, the probabilities of other categories can be obtained by scaling strong prediction probabilities, which can be treated as the soft label, as shown in Figure 2.
Then the obtained soft labels can be handled using methods similar to those used in classification (as described in Section 3). Notably, during the computation of EDL loss, the sparsity caused by high dimensional spaces results in a large KL (Kullback–Leibler) penalty term. To solve this problem, a coefficient is added to the KL penalty to balance it with the magnitude of the negative log-likelihood term. Additionally, clamping is applied to restrict all values within an appropriate range, preventing potentially extremely large outliers on any particular token.

### 4.2 DPO for sequence generation optimization

Different from classification tasks, sequence generation tasks often benefit from sequence-level objectives that directly optimize the entire sequence jointly rather than the individual tokens separately. To further improve the strong model for sequence generation, direct preference optimization (DPO) (Rafailov et al., ) is investigated for WeakS-to-Strong after supervised finetuning, where we propose to use weak models to provide the preference for the strong model generation.

After the strong model is pretrained by supervised finetuning, it generates $M$ output sequences based on a given output. Then for each sequence, $N$ scores are computed by generating it using $N$ weak models separately (via teacher forcing) and aggregating the output (log-)probabilities. A weighted sum
of the $N$ scores is performed as the final score assigned to each sequence. The sequence with the highest final score is viewed as the preferred sequence in DPO training, as shown below

$$ $\bm{y}_{c}=\operatorname*{arg\,max}_{m,m=1,2,\dots,M}\mathrm{P}(\bm{y}_{s}(m)) =\operatorname*{arg\,max}_{m,m=1,2,\dots,M}\sum_{i=1}^{N}\lambda_{i}\mathrm{P} (\bm{y}_{s}(m)|\bm{\theta}_{i}),$ (8) $$

where $\bm{y}_{s}(m)$ is the $m$th output sequence generated by the strong model, $\mathrm{P}(\bm{y}_{s}(m)|\bm{\theta}_{i})$ is computed using the $i$th weak model with model parameters $\bm{\theta}_{i}$, and $\lambda_{i}$ is the weight assigned to the $i$th weak model. The dispreferred sequence can be computed similarly by

$$ $\bm{y}_{r}=\operatorname*{arg\,min}_{m,m=1,2,\dots,M}\mathrm{P}(\bm{y}_{s}(m)) =\operatorname*{arg\,min}_{m,m=1,2,\dots,M}\sum_{i=1}^{N}\lambda_{i}\mathrm{P} (\bm{y}_{s}(m)|\bm{\theta}_{i}).$ (9) $$

Considering potential errors by weak models, a variant of DPO, conservative DPO (cDPO) (Eric, ) with a more conservative target distribution is applied in our work. The loss of cDPO is

$$ $\mathcal{L}_{\text{DPO}}^{\epsilon}=(1-\epsilon)\mathcal{L}_{\text{DPO}}(\bm{ \Lambda},\bm{y}_{c},\bm{y}_{r})+\epsilon\mathcal{L}_{\text{DPO}}(\bm{\Lambda}, \bm{y}_{r},\bm{y}_{c}),$ (10) $$

where $\epsilon$ is a small constant probability that labels are flipped to make DPO more conservative, and $\mathcal{L}_{\text{DPO}}$ is the standard DPO loss in (Rafailov et al., ), which can be written as

$$ $\mathcal{L}_{\text{DPO}}(\bm{\Lambda},\bm{y}_{c},\bm{y}_{r})=-\log\sigma\Big{( }\beta\log\frac{\bm{f_{\Lambda}}(\bm{y}_{c})}{\bm{f}_{\text{ref}}(\bm{y}_{c})} -\beta\log\frac{\bm{f_{\Lambda}}(\bm{y}_{r})}{\bm{f}_{\text{ref}}(\bm{y}_{r})} \Big{)}.$ (11) $$

## 5 Experimental Setup

### 5.1 Datasets

Classification Task. The setup of the classification task follows Burns et al. ().
The SciQ dataset (Welbl et al., ) is used, which contains 13,679 crowdsourced science exam questions about Physics, Chemistry and Biology, among others. The questions are in multiple-choice format with 4 answer options each. In our experiment, 5k data samples were extracted for training weak models and another 5k samples were reserved for generating weak labels to train the strong model. The standard test set which contains 1k data samples was used for the test. The data is restructured into a balanced binary classification task, i.e., given a question and an answer, the model is required to determine whether the answer is correct.

Slot filling. The performance of WeakS-to-Strong on the reliability of generated content was evaluated on the slot-filling task, which is a crucial spoken language understanding task aiming at filling in the correct value for predefined slots (e.g. restaurant and hotel names).
SLURP dataset (Bastianelli et al., ) was used which contains 16.5k sentences and 72k audio recordings of single-turn user interactions with a home assistant, annotated with scenarios, actions and entities.
Only the reference transcriptions of the speech were used for training.
Following Sun et al. (; ), we designed the prompt with slot keys and descriptions in the same way.
In our setup, 2k utterances from the train split were extracted for training the weak models, and another 2k utterances were reserved for generating weak labels and training the strong model. We report the performance of both weak and strong models on the standard SLURP test set.

### 5.2 Models and baselines

Models. For classification, the Qwen-7B (Bai et al., )(^2^22[https://huggingface.co/Qwen/Qwen-7B](https://huggingface.co/Qwen/Qwen-7B)) model was applied as the strong student model. Five models were used as weak teachers: GPT2-Large (Radford et al., )(^3^33[https://huggingface.co/openai-community/gpt2-large](https://huggingface.co/openai-community/gpt2-large)), OPT-1.3B (Zhang et al., )(^4^44[https://huggingface.co/facebook/opt-1.3b](https://huggingface.co/facebook/opt-1.3b)), Pythia-1.4B (Biderman et al., )(^5^55[https://huggingface.co/EleutherAI/pythia-1.4b](https://huggingface.co/EleutherAI/pythia-1.4b)), BLOOM-1B1 (Le Scao et al., )(^6^66[https://huggingface.co/bigscience/bloom-1b1](https://huggingface.co/bigscience/bloom-1b1)), and TinyLlama_v1.1 (Zhang et al., )(^7^77[https://huggingface.co/TinyLlama/TinyLlama_v1.1](https://huggingface.co/TinyLlama/TinyLlama_v1.1)).
The last linear layer which maps the embeddings to tokens is replaced with a linear classification head with two outputs to adapt language models to the classification setting.

For slot filling, Llama-2-7B (Touvron et al., )(^8^88[https://huggingface.co/meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)) was used as the strong model which yielded better performance than Qwen-7B in this task. The same set of weak models was used as the classification task. As before, both weak and strong models are finetuned with all model parameters.

Experiments with different numbers of weak models were conducted. One set involved three weak models (GPT2-Large, OPT-1.3B and Pythia-1.4B) in a WeakS-to-Strong experiment, which is referred to as WeakS-to-Strong-3. Another set included all five weak models, called WeakS-to-Strong-5.

Baselines. The proposed Bayesian WeakS-To-Strong method is compared to the following two baselines.

Naive Multi-Weak. The Naive Multi-Weak approach introduced in Section 3.2 servers as the baseline for both classification and generation tasks.
For the classification task, the loss can be computed following Eqn. (2). For generation task, Eqn. (2) is modified as follows:

$$ $\mathcal{L}_{\text{Naive}}^{\text{Gen}}=\sum_{i=1}^{N}\lambda_{i}\frac{1}{T_{i }}\sum_{j=1}^{T_{i}}\mathcal{L}_{\text{CE}}(\bm{f_{\Lambda}}(\bm{x},\bm{y}_{w} ^{(i,1)},\dots,\bm{y}_{w}^{(i,j-1)}),\bm{y}_{w}^{(i,j)})),$ (12) $$

where $\{\bm{y}_{w}^{(i,1)},\bm{y}_{w}^{(i,2)},\dots,\bm{y}_{w}^{(i,T)}\}$ is the sequence generated by $i$th weak model with length $T_{i}$. In contrast to the classification task, where the strong model $\bm{f_{\Lambda}}$ takes only $\bm{x}$ as input, in the generation task, $\bm{x}$ is paired with the sequence generated by each weak model (which is the weak target) and then fed into the strong model to obtain predictions. Teacher-forcing is used during training.

Joint Decoding. Joint Decoding is used as an additional baseline for text generation tasks.
In contrast to the Naive Multi-Weak scheme where each weak model provides a weak target sequence, Joint Decoding employs multiple weak models to collaboratively determine one single target, as illustrated in Figure 1(b).
Specifically, we perform Joint Decoding in a re-ranking fashion. For each weak model, the top $M$ sequences are generated by beam search in decoding. The sequences from the $N$ weak models are gathered to form a list of $M\times N$ sequences. Then each sequence is scored in the same way in Section 4.2. The sequence with the highest final score is used as the target sequence for strong model training. Unless otherwise mentioned, $M=5$ was used in the experiments. Each weak-to-strong experiment was run three times with different random seeds. The average and standard deviation of the results were reported. More implementation details can be found in AppendixC.

### 5.3 Evaluation Metrics

The classification task is evaluated by accuracy, and the SLU-F1 (Bastianelli et al., ) is used for slot filling, which combines both word-level and character-level F1 scores to give partial credit to non-exact match predictions.
Performance gap recovered (PGR) (Burns et al., ) is used to measure the performance gap recovered with weak supervision, which is defined as follows:

$$ $\text{PGR}:=\frac{\mathcal{P}-\mathcal{P}_{w}}{\mathcal{P}_{s}-\mathcal{P}_{w}}$ (13) $$

where $\mathcal{P}$ is Weak-to-Strong performance, $\mathcal{P}_{s}$ strong performance and $\mathcal{P}_{w}$ weak performance.

## 6 Results

### 6.1 Text classification

**Table 1: Performance of single model on text classification task. Trained by ground-truth labels.**
|  | Pre-trained model | # Param | Accuracy |
| --- | --- | --- | --- |
| Strong Model (ceiling) | Qwen-7B | 7.7B | 0.898 |
| Weak Model | GPT2-Large | 0.8B | 0.717 |
| OPT-1.3B | 1.3B | 0.699 |  |
| Pythia-1.4B | 1.4B | 0.685 |  |
| BLOOM-1B1 | 1.1B | 0.729 |  |
| TinyLlama_v1.1 | 1.1B | 0.731 |  |

**Table 2: Weak(S)-to-Strong performance (train the strong model on the weak label) on text classification task for with (w/) and without (w/o) auxiliary loss. $\gamma$ in Eqn. (1) and Eqn. (5) is set to 0 if auxiliary loss is not used. Experiments with 3 weak models and 5 weak models were conducted. Each experiment was run with three different random seeds. The best results are shown in bold.**
|  |  | w/o aux loss | w/ aux loss |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | Accuracy | PGR | Accuracy | PGR |
| Weak-to-Strong | GPT2-Large | 0.808 $\pm$ 0.007 | 0.503 $\pm$ 0.034 | 0.828 $\pm$ 0.006 | 0.614 $\pm$ 0.029 |
| OPT-1.3B | 0.807 $\pm$ 0.005 | 0.541 $\pm$ 0.026 | 0.841 $\pm$ 0.012 | 0.714 $\pm$ 0.058 |  |
| Pythia-1.4B | 0.775 $\pm$ 0.009 | 0.421 $\pm$ 0.042 | 0.793 $\pm$ 0.009 | 0.507 $\pm$ 0.042 |  |
| BLOOM-1B1 | 0.823 $\pm$ 0.015 | 0.556 $\pm$ 0.087 | 0.843 $\pm$ 0.008 | 0.677 $\pm$ 0.049 |  |
| TinyLlama_v1.1 | 0.832 $\pm$ 0.004 | 0.603 $\pm$ 0.024 | 0.838 $\pm$ 0.005 | 0.643 $\pm$ 0.030 |  |
| WeakS-to-Strong-3 | Naive Multi-Weak | 0.816 $\pm$ 0.002 | 0.586 $\pm$ 0.008 | 0.831 $\pm$ 0.013 | 0.661 $\pm$ 0.064 |
| Bayesian Mutli-Weak | 0.819 $\pm$ 0.006 | 0.600 $\pm$ 0.033 | 0.850 $\pm$ 0.006 | 0.756 $\pm$ 0.028 |  |
| WeakS-to-Strong-5 | Naive Multi-Weak | 0.832 $\pm$ 0.005 | 0.641 $\pm$ 0.025 | 0.853 $\pm$ 0.006 | 0.754 $\pm$ 0.032 |
| Bayesian Mutli-Weak | 0.831 $\pm$ 0.008 | 0.627 $\pm$ 0.027 | 0.866 $\pm$ 0.006 | 0.828 $\pm$ 0.038 |  |

The proposed Bayesian WeakS-to-Strong approach was first evaluated on a classification task. Table 1 shows the respective performance of the strong model and the weak models trained using ground-truth labels, with the former being the ceiling of the Weak(S)-to-Strong approaches. The strong model has about 7 times the number of parameters as the weak models, which also leads to about 28% relative improvement in the classification accuracy. Results of Weak(S)-to-Strong approaches are shown in Table 2. $\gamma$ in Eqn. (1) and Eqn. (5) were set to 0 if auxiliary loss was not used.
It can be seen that auxiliary loss was effective for both. Comparing single Weak-to-Strong results, different weak models show significant differences in performance. For example, Pythia-1.4B recovered only 50% of the strong performance, while OPT-1.3B recovered around 70%.

The Naive Multi-Weak approach and Bayesian Multi-Weak using EDL were applied to the classification task for WeakS-to-Strong. Experiments with three weak models (GPT2-Large, OPT-1.3B and Pythia-1.4B) and all five weak models were conducted.
With three weak models, the Naive Multi-Weak method got an average PGR of 0.661, which is lower than the single OPT-1.3B model, and Bayesian Multi-Weak boosted PGR to 0.756. This indicates that a naive ensemble approach doesn’t necessarily outperform the best single model, especially when there is a certain weak model which does not perform well. The Bayesian approach can increase the fault tolerance with the usage of prior in this case, as it learns patterns from the entire dataset.
With five weak models compared to three, the Bayesian Multi-Weak approach further increased average PGR to 0.828, which is 11% higher than the best single model and 7% higher than the naive baseline, and consistently better with all seeds. The results show the effectiveness of the Bayesian approach for distribution estimation.

### 6.2 Text generation

**Table 3: Performance of single model on text generation task. Trained by ground-truth labels.**
|  | Pre-trained model | # Param | SLU-F1 |
| --- | --- | --- | --- |
| Strong Model (ceiling) | Llama-2-7B | 6.7B | 0.748 |
| Weak Model | GPT2-Large | 0.8B | 0.660 |
| OPT-1.3B | 1.3B | 0.665 |  |
| Pythia-1.4B | 1.4B | 0.680 |  |
| BLOOM-1B1 | 1.1B | 0.651 |  |
| TinyLlama_v1.1 | 1.1B | 0.676 |  |

**Table 4: Weak(S)-To-Strong performance on text generation task, training strong model on weak labels, for with (w/) and without (w/o) auxiliary loss. In cases without auxiliary loss, $\gamma$ in Eqn. (1) and Eqn. (5) is set to 0. Experiments were conducted using 3 and 5 weak labels, each run with three different random seeds. The best results are shown in bold.**
|  |  | w/o aux loss | w/ aux loss |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | SLU-F1 | PGR | SLU-F1 | PGR |
| Weak-to-Strong | GPT2-Large | 0.687 $\pm$ 0.011 | 0.303 $\pm$ 0.125 | 0.673 $\pm$ 0.012 | 0.150 $\pm$ 0.139 |
| OPT-1.3B | 0.660 $\pm$ 0.059 | -0.066 $\pm$ 0.715 | 0.696 $\pm$ 0.009 | 0.367 $\pm$ 0.103 |  |
| Pythia-1.4B | 0.702 $\pm$ 0.007 | 0.320 $\pm$ 0.095 | 0.691 $\pm$ 0.022 | 0.173 $\pm$ 0.322 |  |
| BLOOM-1B1 | 0.690 $\pm$ 0.021 | 0.399 $\pm$ 0.220 | 0.684 $\pm$ 0.021 | 0.337 $\pm$ 0.220 |  |
| TinyLlama_v1.1 | 0.667 $\pm$ 0.006 | 0.000 $\pm$ 0.084 | 0.658 $\pm$ 0.016 | -0.250 $\pm$ 0.224 |  |
| WeakS-to-Strong-3 | Naive Multi-Weak | 0.711 $\pm$ 0.008 | 0.531 $\pm$ 0.101 | 0.694 $\pm$ 0.013 | 0.318 $\pm$ 0.169 |
| Joint Decoding | 0.703 $\pm$ 0.003 | 0.434 $\pm$ 0.032 | 0.704 $\pm$ 0.015 | 0.442 $\pm$ 0.191 |  |
| Bayesian Mutli-Weak | 0.712 $\pm$ 0.014 | 0.549 $\pm$ 0.180 | 0.714 $\pm$ 0.014 | 0.574 $\pm$ 0.176 |  |
| WeakS-to-Strong-5 | Naive Multi-Weak | 0.716 $\pm$ 0.010 | 0.606 $\pm$ 0.123 | 0.694 $\pm$ 0.012 | 0.328 $\pm$ 0.144 |
| Joint Decoding | 0.671 $\pm$ 0.010 | 0.037 $\pm$ 0.120 | 0.675 $\pm$ 0.005 | 0.091 $\pm$ 0.066 |  |
| Bayesian Mutli-Weak | 0.718 $\pm$ 0.014 | 0.627 $\pm$ 0.173 | 0.721 $\pm$ 0.013 | 0.668 $\pm$ 0.166 |  |

For the text generation task using slot filling, the performance of the student strong model and teacher weak model finetuned on ground-truth labels is presented in Table 3. The strong ceiling performance, which is obtained by training the strong model with ground-truth labels, is 0.748, and the highest weak performance is 0.680.

The Weak-to-Strong and WeakS-to-Strong performances are reported in Table 4. For a single weak model, the Weak-to-Strong model performance didn’t necessarily surpass the original weak performance (e.g. TinyLlama_v1.1), and the highest PGR is 0.399. With five weak models, our proposed Bayesian Multi-Weak approach achieved an average PGR of 0.668, which is 26% better than a single weak model, 6% better than the naive baseline, and consistently better across three seeds. Comparing WeakS-to-Strong performance with and without auxiliary loss, adding auxiliary loss is not effective for naive approaches, but improves the performance of Bayesian approaches. It indicates that the proposed Bayesian method, where the predictions of a strong model are applied as part of the distribution estimation along with weak label as in Eqn. (5), more effectively integrates strong model predictions into the training process, showing the effectiveness of our Bayesian approach for the model ensemble. The ablation study can be found in Appendix D.
A noticeable decline has been observed for Joint Decoding when the number of weak models increases from three to five. This may result from the poor performance of the TinyLlama weak model which affects the quality of the generated weak target. Recall that in Joint Decoding, multiple weak models collaboratively determine one single target. In contrast, Naive Multi-Weak and Bayesian Multi-Weak methods still benefit from the increased number of weak models, which indicates that they are more robust against the quality of a single weak model.
Additional experiments on Joint Decoding can be found in Appendix F for more analysis and insights.

**Table 5: Results before and after DPO. Based on the Bayesian Multi-Weak model using five weak models (the WeakS-to-Strong-5 setting) with auxiliary loss. Average results on three seeds are reported. Note that the initial SFT models are different for different seeds.**
|  | SLU-F1 | PGR |
| --- | --- | --- |
| Before cDPO | 0.721 $\pm$ 0.013 | 0.668 $\pm$ 0.166 |
| After cDPO | 0.724 $\pm$ 0.015 | 0.705 $\pm$ 0.187 |

Based on the strong model supervised by five weak models on the Bayesian Multi-Weak approach with auxiliary loss, a cDPO training is conducted, which further train the model in student-forcing form.
Three separate cDPO experiments were conducted using different initial models obtained in SFT stage with three different seeds. The results are shown in Table 5. After cDPO, three models showed consistent performance improvements. The average PGR reached 0.705, 6% relatively better than that before DPO, with a maximum PGR of 0.813.

### 6.3 Complementarity of weak models

An experiment about the complementarity of different weak models was conducted. For classification models, the agreement is assessed by calculating the accuracy of each model’s predictions on test set, treating outputs from other models as references. For generation models, the Levenshtein distance is calculated between different outputs from two models for a same input, which is obtained using the minimum edit number required to change one sequence to another. The average Levenshtein distance across all samples in the test set is used to measure the agreement between two models.

The results are shown in Figure 3. The agreement among different weak models is among 0.75 for classification models, while for generation task the maximum agreement between different models is 0.56. This suggests that the consistency among different weak models is low, thus they can complement each other well as the faults made by different weak models are not the same. Moreover, comparing with Naive Multi-Weak approach, our proposed Bayesian method estimates a distribution based on weak labels, learning patterns from the entire dataset, thereby increasing the tolerance for fault in weak models, as shown in results in Section 6.1 and 6.2.

Figure: (a) Agreement of 5 models on classification.
Refer to caption: x3.png

## 7 Conclusion

This paper extends the Weak-to-Strong framework to WeakS-to-Strong by leveraging an ensemble of weak models to capture the variability in human opinions. We propose a Bayesian inference method, Bayesian WeakS-to-Strong, to more accurately estimate the weak label distribution based on the outputs of multiple weak models. Additionally, while the original Weak-to-Strong method was limited to text classification tasks, this paper expands its applicability to text generation, enabling both the assessment of content trustworthiness and the generation of trustworthy content. Finally, DPO is utilized to enhance the student model’s preference for learning, going beyond the traditional teacher-forcing approach. Our results demonstrate the effectiveness of Bayesian WeakS-to-Strong for both classification and generation tasks, highlighting its potential for superalignment.

## Appendix A Limitations

The proposed Bayesian WeakS-to-Strong method was tested on two different types of tasks: text classification and generative slot filling. We believe the proposed method is general, and further experiments on other applications are reserved for future work. Due to computational resource limitations, experiments on only three and five weak models were conducted in this paper, mimicking the situation where human annotations are costly and time-consuming to obtain. We believe that the capabilities of the strong model can be recovered to a greater extent when more weak models are involved.

## Appendix B Broader Impact

As an approach that enhances the original Weak-To-Strong method, our paper will have the following positive broader impact:

- •
By ensuring strong LLMs behave in ways that are predictable and consistent with societal values, the proposed WeakS-To-Strong further increase public trust in AI technologies.
- •
The use of weak model ensemble for strong model training helps to reduce risks of ethical violations such as gender or racial biases.
- •
Multiple weak models can be more easily updated or replaced to adapt to changing social norms and values. This flexibility allows LLMs to remain relevant and responsive to societal changes, ensuring that they continue to serve the public good over time.

This paper does not give rise to any additional potential biases beyond the ones directly inherited from the pre-trained LLM checkpoints. We encourage the practitioner to carefully select weak models such that the biases present in individual weak models do not accumulate or amplify when combined.

## Appendix C Implementation Details

All models were trained on NVIDIA A800 GPUs using the bfloat16 data type. For the classification tasks, the Adam optimizer was used with a cosine learning rate scheduler and no warm-up period. The batch size was set to 32, with a mini-batch size of 1. The weak models were finetuned on the ground-truth labels with an initial learning rate of $5\times 10^{-5}$, while the strong models were trained with a starting learning rate of $1\times 10^{-5}$ (both on the weak labels and ground-truth labels). The Weak(S)-to-Strong training was run for two epochs.

For generation tasks, the AdamW optimizer was used with a linear learning rate scheduler, also with no warm-up. The initial learning rates were set at $4\times 10^{-5}$ for GPT2-Large and Pythia-1.4B, and $8\times 10^{-5}$ for OPT-1.3B, with a batch size of 8 (mini-batch size of 4). These models were trained for 15 epochs. The checkpoints with the lowest validation loss were selected to ensure the quality of weak labels produced by the weak models. The strong model was trained with a batch size of 2 (mini-batch size of 1) and an initial learning rate of $1\times 10^{-5}$, evaluated at the end of two epochs. In the training for strong ceiling performance, the hyperparameters were adjusted based on the validation set. For Weak-to-Strong training, in which case the ground truth is not accessible, we aligned the hyperparameter settings with those used in strong ceiling training.

For DPO, the initial learning rate was set to $5\times 10^{-7}$ for two epochs, with the cDPO’s hyperparameter $\beta$ set to 2.0 and label smoothing $\epsilon$ as 0.1. Other settings remain the same as the generation tasks.

## Appendix D Ablation Study on Probability Estimation for Weak Sequences

The importance of probability estimation for weak sequences was explored in this section. Two key steps were calculating target wordpiece probability using the word as a gap bridge, and estimating probabilities of other categories by scaling the strong output (see Section 4.1 for details). The results are shown in Table 6. The average PGR of using a one-hot label (Line 1) is negative, showing that WeakS-to-Strong performance doesn’t surpass the weak model without probability estimation. Estimating probability for the target wordpiece significantly improved performance compared to using one-hot labels. This indicates the necessity of introducing target probabilities during training, which allows the strong student model to learn the weak model’s confidence for its generated labels, considering it may be incorrect. Based on this, adding probabilities of other categories, the overall precision of target label estimation improves, resulting in about 20% PGR gain. This experiment highlights the importance of precise probability estimation for weak sequences.

**Table 6: Ablation study on probability estimation for weak sequence. Tested whether to calculate the probability for target wordpiece, and the estimation for probabilities for other categories.**
| Target wordpiece | Other category | SLU-F1 | PGR |
| --- | --- | --- | --- |
|  |  | 0.656 $\pm$ 0.016 | -0.145 $\pm$ 0.204 |
| ✓ |  | 0.704 $\pm$ 0.025 | 0.453 $\pm$ 0.305 |
| ✓ | ✓ | 0.721 $\pm$ 0.013 | 0.668 $\pm$ 0.166 |

## Appendix E Weights of Weak Models

The selection for weights of weak models was explored in this section, where experiments on average, dynamic and fixed weights were conducted. For average, the weight $\lambda_{i}$ in Eqn. (2) and Eqn. (4) is set to the same value for each weak model. That is to say, the losses are calculated for each weak model respectively and then averaged as the final loss. For fixed weight, the weights are set to a fixed value based on the performance of different weak models. For dynamic weight, for a certain sample, the confidence in the utterance of a weak model is treated as a weight for that weak model, considering the importance of different weak models could vary on different samples. The results are shown in Table 7. Compared to the average, fixed weight improves the average PGR by about 8% because it assigns weights based on the performance of different weak models, rather than just averaging. Dynamic weight shows a noticeable decline in performance compared to fixed weight. It may be because our proposed per-token target probability already provided the student model with fine-grained information about the reliability of the label, making dynamic weight entirely redundant.

**Table 7: Different weighting strategies for weak models: experiments using average, dynamic and fixed weights.**
|  | SLU-F1 | PGR |
| --- | --- | --- |
| Average | 0.715 $\pm$ 0.019 | 0.589 $\pm$ 0.237 |
| Dynamic weight | 0.687 $\pm$ 0.013 | 0.241 $\pm$ 0.163 |
| Fixed weight | 0.721 $\pm$ 0.013 | 0.668 $\pm$ 0.166 |

## Appendix F Performance of Joint Decoding

### F.1 The Impact of the Quality of Weak Models

This section provides additional experiments where different weak model combinations are used: (i) three weak models (GPT2-Large, OPT-1.3B, Pythia-1.4B), (ii) four models (adding BLOOM-1B1); and (iii) all five models (further adding TinyLlama_v1.1). For TinyLlama_v1.1, two setups were investigated: (i) only using it to generate weak target sequence but not for scoring (scoring done by other four models); (ii) using it for both generation and scoring. The results are listed in Table 8. Compared to the three-weak-model system, introducing the $4$th model shows a 10% increase in PGR. However, incorporating TinyLlama as the $5$th weak model undermines the performance (last row in Table 8), even nearly fails to surpass weak model performance. A possible reason is that TinyLlama performs poorly on the text generation task. The impact of the TinyLlama can be reduced by excluding it from scoring. As shown in the second to the last row of Table 8, without TinyLlama scoring, the results with five weak models are close to those with three.
This indicates that TinyLlama’s poor scoring ability prevents it from selecting a good target sequence among the candidates and the poor quality from a certain weak model can largely impact the overall results for Joint Decoding methods. In contrast, as discussed in Section 6.2, the proposed Bayesian WeakS-to-Strong method is more robust against the quality of a single weak model.

### F.2 The Impact of Beam Size

As introduced in Section 5.2, in Joint Decoding, each weak model generates $M$ output sequences by beam search in decoding. This section investigates Joint Decoding with different beam sizes $M$, as shown in Table 9. For WeakS-to-Strong-3, results with different $M$ yields similar results. However, for WeakS-to-Strong-5, with TinyLlama included in the weak model set, the result with $M=10$ is significantly worse than that with $M=3$ and $M=5$. The experiments on beam size further demonstrate that the poor results of the WeakS-to-Strong-5 are due to its weak ability to select the target sequence, as $M=10$ introduces more distractions than $M=5$.

**Table 8: Joint Decoding performance with different weak models. Experiments with three weak models (GPT2-Large, OPT-1.3B, Pythia-1.4B), four weak models (adding BLOOM-1B1), and all five weak models are conducted.**
| 3 Weak models | BLOOM-1B1 | TinyLlama_v1.1 | SLU-F1 | PGR |  |
| --- | --- | --- | --- | --- | --- |
| Generating | Scoring |  |  |  |  |
| ✓ |  |  |  | 0.703 $\pm$ 0.003 | 0.434 $\pm$ 0.032 |
| ✓ | ✓ |  |  | 0.711 $\pm$ 0.008 | 0.549 $\pm$ 0.100 |
| ✓ | ✓ | ✓ |  | 0.706 $\pm$ 0.009 | 0.452 $\pm$ 0.086 |
| ✓ | ✓ | ✓ | ✓ | 0.671 $\pm$ 0.010 | 0.037 $\pm$ 0.120 |

**Table 9: Joint Decoding performance with different beam sizes in beam search when weak models generate labels. Both WeakS-to-Strong-3 and WeakS-to-Strong-5 are explored.**
|  | WeakS-to-Strong-3 | WeakS-to-Strong-5 |  |  |
| --- | --- | --- | --- | --- |
|  | SLU-F1 | PGR | SLU-F1 | PGR |
| M=3 | 0.706 $\pm$ 0.012 | 0.471 $\pm$ 0.155 | 0.669 $\pm$ 0.023 | 0.022 $\pm$ 0.281 |
| M=5 | 0.703 $\pm$ 0.003 | 0.434 $\pm$ 0.032 | 0.671 $\pm$ 0.010 | 0.037 $\pm$ 0.120 |
| M=10 | 0.703 $\pm$ 0.004 | 0.431 $\pm$ 0.045 | 0.650 $\pm$ 0.004 | -0.224 $\pm$ 0.050 |

## Appendix G Regularising Term of EDL

As introduced in Section 3.3, the negative log-likelihood of a sample $\bm{y}$ with a predicted Dirichlet prior with hyperparameter $\bm{\alpha}$ is:

$$ $\mathcal{L}_{\text{NLL}}=-\log\int\mathrm{P}(\bm{y}_{w}|\bm{\pi})\mathrm{p}( \bm{\pi}|\bm{\alpha})\mathrm{d}\bm{\pi}=\sum_{k=1}^{K}y_{w}^{(k)}(\log(\alpha_ {0})-\log(\alpha_{k}))$ $$

When a sample is not correctly classified, it is expected the total evidence shrinks to zero for the sample. Taking this into consideration, Sensoy et al. () added a regularization term to penalise the misleading evidence. The loss with this regularising term reads

$$ $\mathcal{L}_{\text{EDL}}=\mathcal{L}_{\text{NLL}}+\lambda_{t}\mathcal{L}_{ \text{KL}}(\text{Dir}(\bm{\pi}|\bm{\tilde{\alpha}})||\text{Dir}(\bm{\pi}|\bm{1 }))$ $$

where the KL term refers to the $\mathcal{L}_{\text{REG}}$ in Section 3.3, $\text{Dir}(\bm{\pi}|\bm{1})$ denotes a Dirichlet distribution with zero total evidence, $\bm{\tilde{\alpha}}=\bm{y}+(1-\bm{y})\odot\bm{\alpha}$ is the Dirichlet parameter after removal of the non-misleading evidence from predicted $\bm{\alpha}$, and $\lambda_{t}$ is the annealing coefficient. By adding a KL-divergence between the Dirichlet distribution with misleading evidence and zero total evidence, the total evidence is enforced to shrink to zero for the simple which is not correctly classified. The annealing coefficient increases by training step, enabling the model to explore the parameter space.

## Appendix H Prompt Used in Experiment

The prompt used for slot filling task is shown as below, in which the input reference transcription refers to the reference transcription as the input. For example, with input “remind me about my business meeting at 3 and 45 pm”, the expected output is ‘{“time”: “3 and 45 pm”}’

[Uncaptioned image]: x5.png

## Appendix I Licenses for Existing Assets

The licenses for each asset used is listed below:

| Model/Dataset | License |
| --- | --- |
| GPT2-Large | Modified MIT License |
| OPT-1.3B | MIT license |
| Pythia-1.4B | Apache 2.0 |
| BLOOM-1B1 | RAIL License v1.0 |
| TinyLlama_v1.1 | Apache 2.0 |
| Qwen-7B | Tongyi Qianwen LICENSE AGREEMENT |
| Llama2-7B | Custom commercial license |
| SciQ | CC BY-NC 3.0 DEED |
| SLURP | CC BY 4.0 |

We provide the following links to special licenses below:

- •
Modified MIT License for GPT2-Large: [https://github.com/openai/gpt-2/blob/master/LICENSE](https://github.com/openai/gpt-2/blob/master/LICENSE)
- •
RAIL License v1.0 for BLOOM-1B1: [https://huggingface.co/spaces/bigscience/license](https://huggingface.co/spaces/bigscience/license)
- •
Tongyi Qianwen LICENSE AGREEMENT: [https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT](https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20LICENSE%20AGREEMENT)
- •
Custom commercial license for Llama-2: [https://ai.meta.com/resources/models-and-libraries/llama-downloads](https://ai.meta.com/resources/models-and-libraries/llama-downloads)

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: **(Q1) How exactly were the the $\lambda$’s set?** Related to (W3) above, the appendix does not say exactly what the final $\lambda$'s are for the "Fixed weight" approach in Appendix E and how "the performance of different weak models" was used.

**(Q2) How is PGR calculated when you have multiple weak models?** The definition provided in Sec 5.3 is clear when there is a single weak model but there are perhaps multiple ways to define in the WeakS-to-Strong setting (e.g. max v.s. average over the weak models in). 

**(Q3) Miscellaneous clarifications**
- In Eq. 4, what is $\hat{y}_w^{(i)}$? Should it not just be the label $k$; in general it seems something in the inner sum should depend on  $k$?
- In Eq. 7, my understanding was that we wanted $P(s_1)P(s_2) = P(W)$ but here it seems like their sum will be equal to P(W) instead of their product.
soundness: 3
strengths: **(S1) Originality and Significance** Studying the WeakS-to-Strong setting is well-motivated and using the proposed Bayesian method here is novel. Furthermore, the expansion to from binary verification tasks to generation is notable since the latter more realistically reflects LLM use cases. 

**(S2) Quality** The paper’s proposed method results in notable improvements on the studied tasks, outperforming the individual weak model teachers in isolation and the naive baseline that simply does a weighted average over the standard CE-based losses. I also appreciated the extra ablations the authors ran to verify the importance of certain aspects of their design (e.g., the aux loss, probability estimation for weak sequences and the $\lambda$ weights across tasks). 

**(S3) Clarity** Overall, I found the writing to be clear, doing a good job of contextualizing methods/results in terms of prior work.
summary: This paper extends work in Weak-to-Strong generalization in two notable ways: (1) considering the setting in which there is an ensemble of multiple weak teachers; (2) expanding from more basic text classification to generation-based tasks. For (1), they show that using a Bayesian approach (based on EDL) is effective for learning from the ensemble of weak teachers, improving upon any of the weak teachers individually as well as simpler baselines that weight typical cross-entropy losses across teachers. For (2), they overcome the key issue that weak and strong models may have different tokenizers and also try applying a modified DPO stage on top, reporting a positive effect.
weaknesses: **(W1) Significance of some results.** I found that certain claims in the manuscript were not as strongly supported by the empirical results. For instance, it appears that the improvements obtained from the DPO-based stage weren't too big (i.e., within standard deviations across runs). Another case was the claim in L461 about the auxiliary loss improving Bayesian methods. That being said, this doesn't affect my opinion about what I consider to be the main result, i.e., the superiority of Bayesian approach. 

**(W2) Other baselines and tasks.** There are some other baselines that I think would be good to consider, namely:
- Methods from data programming / programmatic weak supervision (e.g Snorkel, FlyingSquid) to aggregate the labels generated by different weak models (i.e, treating them as Labeling Functions). In any case, it would be nice to discuss this line of work given that they also operate in a setting where the goal is aggregate multiple weak label sources. 
- In the related work, the authors mentioned that some previous works (Liu & Asahi, 2024; Sang et al., 2024) tried traditional ensembling methods. Perhaps these would also be useful baselines?

In terms of evaluations, it would strengthen the paper if the authors could show similar results on more tasks per category (i.e., classification, generation). Currently the results are limited to one task for each. 

**(W3) Clarification on key assumptions.**  Some further discussion would be helpful on the following
- From cross-referencing Appendix E with the main result tables, it appears that the best results rely on the "Fixed Weight" approach which is based on how good each of the weak models are individually. The assumption here (which I think should be made more explicit), seems to be that one is actually able to accurately estimate the weak model performances. I wonder how applicable this assumption is for real superhuman tasks where presumably getting ground-truth labels is difficult. Perhaps the authors can comment here? 
- Given one stated motivation for the study of WeakS-to-Strong was to consider the variation in human opinions, how might the methods/results generalize to when supervision actually comes from humans rather than weak models? In particular, it seems that in the current experiments, having confidence estimates associated with the weak model labels is important, but this may not readily exist for human-generated labels?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: **Suggest**: As I understand, the regularization loss is critical. I suggest moving the corresponding discussion to the main paper. I also suggest mentioning and discussing the exact values of the weights in the loss to the main paper, as it can provide some insight into how the method works.
soundness: 3
strengths: **Originality**: The paper is the first to successfully extend weak-to-strong to the weaks-to-strong setting. The paper also contains several useful ideas for performing sequence-level training using soft-labels when the student and the teacher have different tokenizers.

**Clarity**: The paper is written well and is easy to follow.

**Quality**: The experiments are well structured to answer the central questions posed by the paper, i.e., is weaks-to-strong better than weak-to-strong, and is the proposed method of performing weaks-to-strong training better than naive baselines? 

**Significance**: Based on the quality of the experiments and the topic addressed, I believe that a broad community will find the paper useful.
summary: The paper extends the weak-to-strong training to multi-weak-to-strong or weaks-to-strong. In a weak-to-strong setting, a smaller model provides a training signal for a larger model. This paper proposes to use multiple small models to provide the training signal. To squize out the best from the ensemble of small models, the paper proposes a Bayesian strategy based on evidential deep learning. The paper also performs sequence generation training, which was not done previously in the weak-to-strong setting. The results demonstrate that weaks-to-strong provides produces better results compared to weak-to-strong in both classification as well as generation.
weaknesses: 1. Some of the design choices are not well justified. For example, for the sequence-level training using soft labels, it is not clear why the strategy proposed in section 4.1 should work or is optimal. 
2. The datasets selected for the classification and generation tasks seem very specific. Is there a particular reason for selecting only these? The impact of the paper could be much better if model general datasets were included, at least for the generation task.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - In Equation 3, are $\alpha_k$s are trainable parameters?
- Is the Equation 3 the loss for each sample?
soundness: 2
strengths: - The proposal is simple but works.
summary: The paper introduces a Bayesian-based method inspired by the weak-to-strong approach for the classification task of language models. The author also extends the algorithm for the generation task by some simple transformations. To improve the performance of the generation task further, the author utilized DPO to finetune the model at the tail of the framework.
weaknesses: - The motivation for using Dirichlet distribution to model the prior is ambiguous.
- The writing for the methodology description is unclear and hard to follow
- The transformations in Equation 3 should be clarified.
- What is the definition of $p_k$ in Equation 4?
- In Section 4.1, the probability of the target token of the strong model changing after each update of network parameters because $C_s(s_1)$ and $C_s(s_2)$ shift. This means that the targets vary along the training, likely creating instability or leading to divergence. The time consumed for recomputing these terms is also one of my concerns.
- All baselines that are naive and very simple algorithms are weak for both classification and generation tasks. It is suggested that stronger baselines be included for both tasks.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. I didn't get the idea behind "The description about complementary of weak models"(Section 6.3). I can tell that classification enjoy better agreement than generation(as generally expected), but I didn't get "This suggests that the consistency among different weak models is low, thus they can complement each other well as the faults made by different weak models are not the same"
soundness: 2
strengths: 1. They propose a Bayesian WeakS-to-Strong framework that improves weak supervison.(Although I tend to say this improvement is not well-proved). 
2. Extention of Weak-to-Strong framework from text classification to generation tasks.
3. A method for estimating token-level probabilities is introduced to create soft labels for training robust models.
summary: This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS- to-Strong generalization.

Furthermore, they broaden the use of the WeakS-to-Strong approach from its original focus on text classification to encompass text generation tasks(this point causes confusion for me, see in the weakness below), exploring more sophisticated supervision strategies. Additionally, direct preference optimization is employed to enhance the student model's ability to learn preferences, moving beyond the traditional teacher-forcing training framework(I think this contribution is not clear and verified by comprehensive experiments). 

I am not an expert in this domain, but I read this paper twice to get the paper's idea. I am open to other reviewers' opinion.
weaknesses: I have the following concerns regarding to this paper: 

1. The experiments is not comprehensive: the experiment they conduct are mainly on the ~1B weak models and the strong model are ~7B. I think from the experiment results conducted now, the claim on WeakS-to-Strong in the paper is slightly overclaimed. (**The major weakness in my opinion**)
2. The improvement shown in the experiment is not observable: for instance, for accuracy: 0.843(baseline) vs 0.850(WeakS-to-Strong-3). The improvement is negligible.

Meta-review: This paper seeks to improve weak-to-strong generalization. This phenomenon, which has been known for quite a long time, has recently become popularized due to its theorized application to superalignment. In weak-to-strong generalization, a weak model produces outputs that a strong model is trained on. In certain settings, this strong model can perform better than the weak model did, despite only being trained on the weak model’s outputs. 

The authors note that one challenge is the choice of the weak model. They propose replacing a single model with a simple Bayesian ensembling approach and show how to use this for both predictive and generative tasks. The latter setting is challenging, since it requires ensembling output sequences from multiple generators. They also show some nice extensions for alignment. 

The reviewers generally liked the paper, with one caveat being the strength of the empirical results. While I share some of the sentiment, the paper has a bunch of good ideas and sufficient evidence of their value, so I voted for acceptance.

Decision: Accept (Poster)