Bayesian WeakS-to-Strong from Text Classification to Generation
Authors: Ziyun Cui
Ziyang Zhang
Guangzhi Sun
Wen Wu
Chao Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: weaks-to-strong
bayesian
generation
per token
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them weakly. Weak-to-Strong mimics such a scenario where weak model supervision attempts to harness the full capabilities of a much stronger model. This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS-to-Strong generalization. Furthermore, we extend the application of WeakS-to-Strong from text classification tasks to text generation tasks where more advanced strategies are investigated for supervision. Moreover, direct preference optimization is applied to advance the student model's preference learning, beyond the basic learning framework of teacher forcing. Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superalignment.

Published as a conference paper at ICLR 2025
BAYESIAN WEAKS-TO-STRONG
FROM TEXT CLASSIFICATION TO GENERATION
Ziyun Cui C Ziyangg Zhang' ; Guangzhi Sun?, Wen Wu Chao Zhang "Department of Electronic Engineering, Tsinghua University, Beijing, China Shhnggaai Artificial Intelligence Laboratory, Shanghai, China
Department of Engineering, University of Cambridge, Cambridge, UK cui mui-zy24 1ey24mmaill 1s. tsiinggue edu. cn, ziyang leraangz240mmm 220mmaiss.tssnnhheeee tsinghua. edu, cn, gs5348 @am. ac. uk, wuwenêpji jaappplllbooooo www.aa............ cn, en, cz2770tsinge edu. cn
ABSTRACT
Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them weakly. Weak-to-Stronp mimics such scenario where weak model supervision attempts to harness the full capabilities of a much stronger model. This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using Bayesian approach to guide the WeakS- to-Strong generalization. Furthermore, we extend the application of WeakS -to- Strong from text classification tasks to text generation tasks where more advanced strategies are investigated for supervision. Moreover, direct preference optimization is applied to advance the student model's preference learning, beyond the basic learning framework of teacher forcing. Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superaligmment.
INTRODUCTION
With the increase in computing power and the amount of training data available, the capabilities of large language models (LLMs) have been continuously brought closer to humans many aspects. Despite their impressive performance, the preferences and values of pre-trained LLMs do not always align with humans, and dedicated approaches are needed to tackle the problem. Based on large- instruction datasets, supervised finetuning (SFT) encourages LLMs follow human instructions more strictly and respond more safely (Wei et al. 2022). Reinforcement learning (RL) is commonly applied to such alignment. By collecting model output values and the corresponding human feedback, the model can be finetuned by RL to avoid generating undesirable outputs (Ziegler et al., 2019 Bai
al. 2022a Ouyang et al., 2022 Nakano et al. 2021 Askell et al., 2021
Since no current model has yet surpassed human intelligence, alignment methods, such as SFT and RL from human feedback (RLHF). remain effective. However, it is worthwhile considering future scenarios where artificial intelligence (Al) might surpass human intelligence in all aspects. Would the current alignment methods still effective for such super Al models? How could humans supervise the super AI? To simulate this future scenario, analogy situation is designed that downgrades both sides: using a weak model to simulate humans and a strong model to simulate future super AI (Burns al. 2023). which is termed superalignment. It has been demonstrated that adding simple auxiliary loss can achieve effective Weak-to- Yeek-tt-SStrong Strong generalization, even if the weak model's supervision contains many errors, which offers hope of achieving superalignment, Nonetheless, this just the beginning of exploring along the path of Weak-to- to-Ston.......
'Equal contribution.
1 Corresponning author.
Supportedd by Shanghai Artificial Intelligence Laboratory.
Code is available in httpttppp... //gi thub. com www.///wwwwwo///...ooooooo.o http://gt..././///////////////// anws25

---

Published as a conference paper at ICLR 2025
This paper extends the discussion Weak-to-Shee torkoon in two directions. First, given the inherent capability gap between the weak model and the strong model, we propose using an ensemble of multiple weak models to improve the quality of weak supervision, which is called WeakS-to- Strong. This also accounts for the scenario where human opinions might diverge in tasks without commonly accepted standard. Several approaches have been studied effectively leverage the diversity of different weak models, and we adapt Bayesian approach referred to as evidential deep learning (EDL) (Sensoy al. 2018) better estimate broader human preferences by learning a prior distribution over the weak labels produced by the weak models. Furthermore, the Weak-to-Stron task was primarily studied for text classification tasks (Burns et al.) 2023). This paper extends the scope to text generation and compares three different approaches, Naive Multi-Weakk Joint Decoding and Bayesian Multi-Weak, as shown in Figure []] The proposed Bayesian WeakS-to-Strong approach demonstrated effective for both classification and generation. To better align with human preferences, a variant of direct preference optimization (DPO) (Rafailov et al., 2023) called conservative DPO (cDPO) (Eric 2023 is used to finetune the strong model further on RL principles.
Our main contributions are summarized as follows.
We proposed Bayesian WeakS-to-Strond which largely improves the quality of weak super- vision and recovers the performance of the strong model.
We propose to generalize both Viaa-ttoSSrrong and WeakS Sooootong from text classification to generation tasks, extending their scope from content regulation content generation.
When applied to text generation, a token-level probability estimation is proposed to achieve soft labels for strong model training. We also propose the modified DPO algorithm under the Bayesian WeakS-to -t-Sttng framework to further improve text generation performance.
RELATED WORK
AI Alignment. Aligning LLMs with human preferences has been a long-standing goal. Instruction tuning uses extensive datasets to improve LLMs' adherence to human instructions (Wei et al., (022). RL allows LLMs to learn what types of responses humans prefer or dislike, with proximal policy optimization (PPO) being effective RL method first applied to LLMS and becoming part of the
standard RLHF process (Ziegler et al.. 2019 Bai al., 2022a Ouyang al. 2022, Nakano et al.
2021 Askell et al.] 2021) However, PPO training can be unstable, leading to the development of DPO Rafailov et al. 2023). Given the high cost of obtaining human preference data, researchers
now exploring the use of LLMs simulate human preferences, provide feedback, and finetune models (Lee et al. 2023 Bai et al. 2022b Gulcehre et al. 2023).
Variability in Human Opinions. In the process of aligning Al with human preferences, it is important to consider the inconsistency of human preferences (Liu et al., (2023), which often leads to multi-label problems. Previously, many approaches used simple methods like voting, aggregation,
and averaging to handle multi-labels (Davani et al., 2022 Munos et al., 2023 Paun & Simpson 2021 Prabhakaran et al.) 2021). However, these methods do not effectively capture the preference differences of individual annotators included in the multiple labels. To better estimate the diversity of human preferences, Bayesian principles have been introduced. Deep learning models can be used to predict prior distributions, which are considered to produce the multiple available labels to estimate a broader range of human preferences (Sensoy et al. 2018 Wu al. 2022 2023).
Weak-to-Strong, The goal of Weak-to-Strone is to use weak model to better supervise strong model. OpenAl demonstrated that adding auxiliary confidence loss from the strong model itself can significantly improve the Weak-to-Strong performance (Burns et al. 2023). Following OpenAl's work, several studies emerged to introduce multiple weak models, used either in series or parallel, to improve the quality of supervision provided by the weak models (Liu & Alahi 2024 [ang et al. (024). Early model ensemble methods like Adaboost (Freund & Schapire 1995) and Bootstrap aggregating (Leo] 1996) were explored in these works. Furthermore, confidence scores are incorporated to help the strong model assess the supervision quality provided by the weak models (Guo et al. (024) and the weak model can be directly used modify the output of the strong model (Ji al. 2024)..
2

---

Published as a conference paper at ICLR 2025
Q: How to keep healthy?
Balanced diet
Balanced diet
diet exereise
Balanced diet
Regular exercise
Regular exercise
00
OO
M
Eat lot
C9
Eat lot
(a) Naive Multi-Weak
(b) Joint Decoding
(c) Bayesian Multi-Weak
Weak
An overview diagram of the three ensemble approaches: (a) Naive Multi-Weak: directly learn all weak labels produced by weak models, (b) Joint Decoding: weak models collaboratively determine one single target, (c) Bayesian Multi-Weak: learn a prior distribution over weak labels.
Figure 1:
3 WEAKS-TO-STRONS METHODOL.OGY
3.1 PRELIMINARY: WEAK-TO-STRON
The Weak-to-Strong pipeline (Burns et 2023) involves three steps: (i) create weak supervisor
by finetuning a small pre-trained model on ground-truth labels; (ii) train a strong student model SA with weak supervision by finetuning pre-trained LLM using "weak labels" generated by weak supervisors, where A is the parameters of the strong model; (iii) finetune the large pre-trained model directly using ground-truth labels which serve as the ceiling.
To leverage the superior generalization capabilities and prior knowledge of the strong model, a loss function with auxiliary confidence loss proposed (Burns al. 2023):
L= C= (1- - 7) (fu (x),, yd) + (CE(fA(x)) (x))
(I)
where Yhe represents the weak label from weak model, (a:) refers to the predicted class of strong model given input æ. and -) denotes the cross-entropy loss. The second term is an (optional) auxiliary self-training loss designed to increase the confidence of the strong model in itself. The weight of the second loss 7 linearly grows up from 0 to a pre-defined hyper-parameter 7/max which gradually reduces the weight the weak labels and increases the weight self-traing self-training when the number training steps increases.
3.2 EXTENDING WEAK-TO STRONG WITH MULTIPLE WEAK MODELS
Although it has been shown that the Weak-to-Strone approach can recover part of the strong model's performance (Burns et al. 2023), the errors in weak labels limit the performance of Seakkto--ttron. generalization. response to this problem, we propose leverage the complementarity the error patterns of multiple weak models using an ensemble strategy, which is referred to as WeakS-to-Stronp
naive approach to implementing an ensemble of multiple weak models to calculate the loss for each weak label respectively and then average these losses. An improvement of this approach is to take a weighted sum instead of a simple average:
N
L Σ (f i=1
where is the number of weak models, you is the éth weak label produced by the ith weak model, and X; is pre-defined weight of the loss regarding the the ith weak model. This approach is referred to as a Naive Multi Weak system in the rest the paper (as illustrated in Figure T(a),, which is treated of the baselines.
3

---

Published as a conference paper at ICLR 2025
3.3 BAYESIAN WEAKS-TO-STRONE
For superalignment, multiple weak models are used to mimic the subjective preferences of multiple humans, which can be considered observations drawn from underlying distribution of the opin- ions of all humans. The naive approach described in Section 3.2 solely relies on these observations. The number of observations (human annotations or weak labels) is often very limited due to the considerable cost of hiring a new human annotator or training new weak model. Such limited number of observations may not result in a good approximation of the true human opinion distribution. Having biased preferences or values is particularly unacceptable in the safety domain and can cause a failure of superalignment. Therefore, we propose a Bayesian WeakS-to-Strong approach based on EDL (Sensoy et al. 2018) to estimate the human opinion distribution based on the weak labels. Figure c) illustrates the framework where three weak models are involved.
For a given input consider a weak label from the ith weak model yoe which is one-hot vector with y/o-4) being one if it belongs class k and zero otherwise. Yo is sampled from a categorical distribution of weak labels Cat(T), where each component TT corresponds to the (5) P(y = Cat(Ir). EDL places Dirichlet probability assignment over the possible classes you
prior over the categorical distribution representing the probability of each categorical probability assignment, hence modelling second-order probability π ~ -ppp(((] [ where a is the hyperparameter of the Dirichlet prior. The strong model fA is trained to predict ar for each input by minimizing the negative log-likelihood of sampling you given the predicted Dirichlet prior:
2 - log / P(U ())(T
(1 (log(as) - log(ax));
where K is the number of classes, og- = >x Or is the Dirichlet strength, and yhe
(2.R) is the t value of label you When sample is not correctly classified, it is expected that the prior should approach non-informative prior for this sample. Following Sensoy et al. (2018), a regularization term c(u (KL-divergence between the misleading prediction and non non-intormative distribution, see Appendix H for details) is added to penalize incorrect predictions and calibrate uncertainty estimation, resulting in the final EDL loss C ckil + >EDI.CRi where ÅEDI. is the coefficient.
Apart from the class predicted by the weak models, the confidence of weak models is also incorporated
for better distribution estimation. Let (pi ..... (6) be the probability assignment predicted by the 5th weak model, the EDL loss for each class is calculated based on the predicted probability assignment for each weak model and then combined in the same way as in Eqn. (2). That is,
N
K
(Ja(),
Le
LED (SA (Z), x)
where gl ((6,k) is the predicted result, i.e., one-hot vector for class k and A;s are hyperparameters set to the same values used in the Naive Multi-Weak approach. As a result, the auxiliary confidence loss described in Eqn. T is adapted for Bayesian WeakS-to-Stren as follows:
L= (1-7) LEDI (æ),
+ LEDI (Sa (x), fx)).
In the term of LEDI(fA (z), ff (z)). the class index predicted by the strong model fx (x) is used as the target. That is to say, the predictions of the strong student model are applied as part of the distribution estimation along with the weak label.
WEAKS-TO-STRES FOR SEQUENCE GENERATION
4.1 PROBABILITY ESTIMATION FOR WEAK SEQUENCE LABELS
To enable the strong model to directly generate trustworthy content rather than only being trained to understand whether the content is trustworthy or not, we propose to extend the scope of Weak- to-Strong from text classification to text generation. The key challenge of directly applying the
4

---

Published as a conference paper at ICLR 2025
Stage 1
Stage
Stage
strong prob
Weak Wordpiece
Word (..., bello,
wordpicce: il..... wordpiece prob: (.., 0.45, 0.S...
word prob: C... 0.36,
Weak Tokenizer
Strong Tokenizer
Calculate target prob
hel wordpizes
wot
P(W) = ((Wglw
Strong Wordpieve
Target Prob
target wordpione taken prob: 0.4
wordpiece: C.., hel, lo, ..) wordpiace prolh for Pu Pa-
target wondpieces C... hel, lo,
torger probo 0.9.
label
Word
word: (-... helloh ...
Strong Model
word prob: 0.36,...
Strong Confidence
10
hccl
W/00
Figure 2: The process of transforming per-token confidence scores from the sequence tokenized by the weak model to the sequence tokenized by the strong. The word "hello" is used as example. Stage I: The words and word scores are obtained from the weak model wordpieces and their scores. Stage 2: The words are tokenized by the strong model tokenizer, and the tokenized sequences are fed into the strong model to obtain the strong model predicted probability (denoted as confidence) for each token s,. This strong model confidence is then used to split word scores into target wordpiece probabilities P(s;) while keeping the probability of the word unchanged. Stage 3: The obtained target probability is transformed into the label. Probabilities of other categories are calculated by scaling the strong output distribution using P(s.).
Weak-to-Stue to-Strong loss to the sequence generation task is the token-level soft labelling for the target sequence. As the tokenizers are different between weak and strong models, is infeasible to obtain one-to-one mapping from weak model output distributions to each token in the target sequence.
To obtain the soft label yor for the strong model using weak model output probabilities and bridge the gap caused by different tokenizers, we use words as an intermediary, following the equation
P(W) P(wy|us) P(w1) ((s))s)))((s)
(6)
where we use a word containing two wordpieces from both weak and strong tokenizers as an example, and W1, We and $1, $2 both token strings that form word W, which generated by the tokenizers of the weak and strong models respectively. Figure 2 shows an example of the process in three stages. In stage the per-token output probabilities of weak models are obtained when generating output sequences. The probabilities of wordpieces in a word are then multiplied together obtain the score of word W, following Eqn. (6).
In stage 2. the word probability is used to assign probabilities to tokens from the strong model tokenizer. In the process of training the strong model via teacher-forcing, the model gives probability to each token by applying softmax to the output logits. This probability can be seen as the model's confidence predicting the target token, where we assume that the weak model and strong model have similar confidence for wordpiece tokens in similar positions. This allows us to to obtain the actual assignment of scores to each target token 8,, instead of assigning equal probabilities to all tokens involved. Taking example of splitting word W into two target wordpicces $1 and s2, the decomposition can be approximated by
C.(s))
Ca(s2)
log P(81) c=C,(a1) + C.() log P(W),
log P(x2) c-C,(s)) + C_() log P(W), (7)
where (s:) is strong model confidence at the step predicting wordpiece 8; that is the maximum probability in the strong model output distribution at that step in practice. In this way, lower target probabilities are allocated to tokens with lower strong model confidence, while higher probabilities allocated to tokens with higher strong model confidence.
After obtaining the probability the target token of the strong model, the probabilities of other categories be obtained by scaling strong prediction probabilities, which can be treated as the soft label, as shown in Figure Then the obtained soft labels can be handled using methods similar to those used in classification (as described in Section 3). Notably, during the computation of EDL loss,
5

---

Published as a conference paper at ICLR 2025
the sparsity caused by high dimensional spaces results in large KL Becllbacc--eibless penalty term. To solve this problem, a coefficient is added to the KL penalty to balance it with the magnitude of the negative log-lik likelihoodd term. Additionally, clamping is applied restrict all values within an appropriate range, preventing potentially extremely large outliers on any particular token.
4.2 DPO FOR SEQUENCE GENERATION OPTIMIZATION
Different from classification tasks, sequence generation tasks often benefit from sequence-level objectives that directly optimize the entire sequence jointly rather than the individual tokens sepa- rately. To further improve the strong model for sequence generation, direct preference optimization (DPO) RRfailov al. 2023) investigated for WeakS-to-Strong after supervised finetuning, where we propose to use weak models to provide the preference for the strong model generation.
After the strong model is pretrained by supervised finetuning, it generates M output sequences based on given output. Then for each sequence, N scores are computed by generating it using N weak models separately (via teacher forcing) and aggregating the output (log- prog---pbobbbiiicss... sum of the N scores is performed the final score assigned to each sequence. The sequence with the highest final score viewed as the preferred sequence in DPO training, as shown below
weighted
y/ arg max
(my) arg m1 1,
(P(y)(m))0)).
(8)
M. m,m 1,2
where ye (m) the mith output sequence generated by the strong model, (m)) 0.) is computed using the ith weak model with model parameters 0,, and A; is the weight assigned to the ith weak model. The dispreferred sequence can be computed similarly by y arg min P P(ye (m)).
Considering potential errors by weak models, variant of DPO, conservative DPO (cDPO) (Eric 2023 with a more conservative target distribution applied in our work. The loss cDPO is
LO = (1 - CDpo ye: U-) + CDro (A, Yr. ye);
where e is a small constant probability that labels are flipped to make DPO more conservative, and Looo is the standard DPO loss in (Rafailov et al. 2023), which can be written
Looo (A, Ye M-) log (Blog fy(Ye) log fA(y)) S1e(
(10)
[re((y)) )
EXPERIMENTAL SETUP
5.1 DATASETS
Classification Task. The setup of the classification task follows Burns et al. (2023) The SciQ dataset (Welbl et al.] (017)) is used, which contains 13,679 crowdsourced science exam questions about Physics, Chemistry and Biology, among others. The questions are in multiple-choice format with 4 answer options each. In our experiment, 5k data samples were extracted for training weak models and another 5k samples were reserved for generating weak labels train the strong model. The standard test set which contains Ik data samples was used for the test. The data is restructured into a balanced binary classification task, i.e., given a question and an answer, the model is required to determine whether the answer is correct.
Slot filling. The performance of WeakS-to- to-onn on the reliability of generated content was evaluated on the slot-filling task, which is a crucial spoken language understanding task aiming at filling in the correct value for predefined slots (e.g. restaurant and hotel names). SLURP dataset (Bastianelli et al. 2020) was used which contains 16.5k sentences and 72k audio recordings of single-turn user interactions with a home assistant, annotated with scenarios, actions and entities. Only the reference transcriptions of the speech were used for training. Following Sun et al. (2023a b), we designed the prompt with slot keys and descriptions in the same way. In our setup, 2k utterances from the train split were extracted for training the weak models, and another 2k utterances were reserved for generating weak labels and training the strong model. We report the performance both weak and strong models on the standard SLURP test set.

---

Published as a conference paper at ICLR 2025
5.2 MODELS AND BASELINES
Models. For classification, the Qwen-7B (Bai et al. dent model. Five models were used as weak teachers: GPT2-Large (Radford et al., 2019). OPT- 1.3B (Zhang et al. 2022). Pythia-Ls Mithia-.44B/ (Biderman et al. 2023), BLOOM-IBI (Le Scao et al. 2022). and TinyLlama vI.1 (Zhang et al., 2024). The last linear layer which maps the embeddings to tokens is replaced with linear classification head with two outputs to adapt language models to the classification setting,
2023) model was applied as the strong stu-
For slot filling, Llama-2-7B (Touvron et al. 2023) was used the strong model which yielded better performance than Qwen-7B in this task. The same set of weak models was used as the classification task. As before, both weak and strong models are finetuned with all model parameters.
Experiments with different numbers of weak models were conducted. One set involved three weak models Large, 1.3B and Pythia-1.4D3 in a WeakS- WeakS-to-Strong experiment, which is referred to as WeakS-to-Strong-S Another set included all five weak models, called StackStto-Strong55.
Baselines. The proposed Bayesian WeakS-to-Strong method is compared to the following three baselines.
Naive Multi-Weak. The Naive Multi-Weak approach introduced in Section 3.2 servers as the baseline for both classification and generation tasks. For the classification task, the loss can be computed following Eqn. (2). For generation task, Eqn. is modified as follows:
GGn Naive
LCE(fA ..... D 11--1) ((
(11)
where {ym "/o
You (8.2)) } is the sequence generated by ith weak model with length T,. In contrast to the classification task, where the strong model fA takes only input, in the generation task, x is paired with the sequence generated by each weak model (which is the weak target) and then fed into the strong model to obtain predictions. feacher-forcing is used during training.
(6.2)
Sqyin. FlyingSquid (Fu et al. 2020) is a method for weak supervision, estimating the ac- curacies and correlations among multiple noisy label functions (different weak labels in our case) without ground-truth data. Latent variable probabilistic graphical models are used to model these dependencies, with weak labels as observed variables and unobserved ground-truth labels hidden variables. Since FlyingSquid is designed for binary classification, this baseline is only used for classification. Through this method, we get label model with multiple weak labels and obtain the probability for the positive category, which is then used as a soft label in strong model training.
Joint Decoding. Joint Decoding is used as an additional baseline for text generation tasks, which is specifically designed for multiple weak model generations. In contrast to the Naive Multi- Weak scheme where each weak model provides a weak target sequence, Joint Decoding employs multiple weak models to collaboratively determine one single target, as illustrated in Figure [b). Specifically, we perform Joint Decoding in a re-ranking fashion. For each weak model, the top M sequences generated by beam search in decoding. The sequences from the N weak models are gathered to form a list of M x N sequences. Then each sequence is scored in the same way in Section 4.2 The sequence with the highest final score is used as the target sequence for strong model training. Unless otherwise mentioned, M = was used in the experiments.
Each weak- teak-to-sstrn to-strong experiment was run three times with different random seeds. The average and standard deviation were reported. More implementation details can be found in Appendix C
5.3 EVALUATION METRICS
The classification task evaluated by accuracy, and the SLU-FI (Bastianelli et al. 2020 is used for slot filling, which combines both word-level and character-level Fl scores to give partial credit to non- exact match predictions. Performance gap recovered (PGR) (Burns et al. 2023 is used to measure the performance gap recovered with weak supervision, which is defined as (P (P, Pw) where P is Weak-to- Weak-to-Strong performance, Pa, strong performance and P.,. weak performance. For multiple weak model cases, the average of PGRs for each weak model is treated the final result.
7

---

Published as a conference paper at ICLR 2025
Table 1: Performance of single model on text classification task. Trained by ground-truth labels.
Pre-trained model
# Param
Accuracy
Strong Model (ceiling)
Qwen-7B
7.7B
0.898
GPT2-Large
0.8B
0.717
OPT-1.3B
1.3B
0.699
Weak Model
Pythia- 1.4B
1.4B
0.685
BLOOM-IBI
1.1B
0.729
TinyLlama_vi
1.1B
0.731
Table Wea performance (train the strong model on the weak label) on text classifi- cation task for with (w/) and without (w/o) auxiliary loss. in Eqn. ① and Eqn. (5) is set to 0 if auxiliary loss is not used. Experiments with weak models and 5 weak models were conducted. Each experiment run with three different random seeds. The best results are shown in bold.
w/o aux loss
w/ aux loss
Accuracy
PGR
Accuracy
PGR
GPT2-Large
0.808 + 0.007 0.503 + 0.034 0.828 0.006 0.614 + 0.029
OPT-1.3B
0.807 + 0.005 0.541 + 0.026
0.841
0.012 0.714 0.058
Wcak-to-Strong
Pythia-
0.775 + 0.009 0.421 + 0.042
0.793
0.009
0.507 + 0.042
BLOOM-IB1
0.823 + 0.015 0.556 + 0.087
0.843
0.008 0.677 + 0.049
TinyLlama v1.1
0.832 + 0.004 0.603 = 0.024
0.838
0.005
0.643 = 0.030
Naive Multi Weak
0.816 1 0.002 0.586 1 0.008
0.831
0.013
0,661 + 0.064
WeakS-to-Strong-3
FlyingSquid
0.809 1 0.005 0.549 1 0.026
0.825
0.003
0.631 1 0.013
Bayesian Mutli- Weak
0.819 1 0.006 0.600 1 0.033
0.850
0.006
0.756 1 0.028
Naive Multi- Weak
0.832 = 0.005
0,641 = 0.025
0.853
0.006
0.754 = 0.032
WeakS-to- SeakS-to-Stronn-5
FlyingSquid
0.832 = 0.004
0.643 = 0.023
0.855
0.007
0.768 0.035
Baycsian Mutli Weak
0,831 = 0.008
0.627 = 0.027
0.866
0.006
0.828 = 0.038
RESULTS
6.1
TEXT CLASSIFICATION
The proposed Bayesian WeakS-to-Siee -Strnng approach was first evaluated on a classification task. Table T shows the respective performance of the strong model and the weak models trained using ground-truth labels, with the former being the ceiling of the Weak(S)-to-Stron about 7 times the number of parameters the weak models, which also leads to about 28% relative improvement in the classification accuracy. Results of Weak(S)) tt-Sttooongg approaches are shown if auxiliary loss not used. It be that auxiliary loss was effective for both. Comparing single Weak-to-Strong results, different weak models show significant differences in performance. For example, Pythia-1.4B recovered only 50% of the strong performance, while OPT-1.3B recovered around 70%.
approaches. The strong model has
in Table 2 7 in Eqn. (1) and Eqn. (5) were
The Naive Multi-Wean Weak baseline, FlyingSquid approach and Bayesian Multi-Weak using EDL were applied to the classification task for WeakS-to-Strong Experiments with three weak models (GPT2- Large, OPT-1.3B and Pythia- 1.4B) and all five weak models were conducted. With three weak models, the Naive Multi-Weak method got an average PGR of 0.661, slightly outperforming FlyingSquid method but still lower than the single OPT-1.3B model, and Bayesian Multi-Weak boosted PGR to 0.756. This indicates that naive ensemble approach doesn't necessarily outperform the best single model, especially when there a certain weak model which does not perform well. The Bayesian approach can increase the fault tolerance with the usage of prior in this case, it learns patterns from the entire dataset. With five weak models compared to three, the Bayesian Multi Weak approach further increased average PGR 0.828, which 16% relatively higher than the best single model and 8% relatively higher than baselines, and consistently better with all seeds. The results show the effectiveness of the Bayesian approach for distribution estimation.
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance of single model on text generation task. Trained by ground truth labels.
Pre-trained model
# Param
SLU-FI
Strong Model (ceiling)
Llama-2-7B
6.7B
0.748
GPT2-
0.8B
0,660
OPT-1.3B
1.3B
0.665
Weak Model
Pythia-1.4B
1.4B
0.680
BLOOM-IBI
1.1B
0.651
TimyLlama_v.11
1.1B
0.676
Table 4: - performance on text generation task, training strong model on weak labels, for with (w/) and without (w/o) auxiliary loss. In cases without auxiliary loss, 7 in Equ. (T) and Eqn. { is set to O. Experiments were conducted using and weak labels, each run with three different random seeds. The best results are shown in bold.
w/o aux loss
w/ aux loss
SLU-FI
PGR
SLU-F1
PGR
GPrg-Large
0.687 = 0.011
0.303 0.12
0.673
0.012
0.150 0.139
OPT-1.3B
0.660 = 0.059
-0.066 0.715
0.696
0.009
0.367 0.103
Weak-to-Strong
Pythia- 1.4B
0.702 = 0.007
0.3209
0.173 =
0.322
BLOOM-IBI
0.690 = 0.021
0.399
0.220
0.684 + 0.021
0.337 + 0.220
TinyLJama.vI.1
0.667 = 0.006
0.000 +: 0.084
0.658 + 0.016
-0.250 + 0.224
Naive Multi-Weal
0.711 + 0.008
0.531
+ 0.101
0.694 + 0.013
0.318 + 0.169
YeakS-to-Strong.
Joint Decoding
0.703 = 0.003
0,434 + 0.032
0.704 + 0.015
0.442 + 0.191
Bayesian Mutli Weak
0.712 = 0.014
0.549 0.180
0.714 0.014
0.574 + 0.176
Naive Multi-Weal
0.711 = 0.010
0.606 + 0.123
0.694 1: 0.012
0.328 1 0.144
WeakS-to-Strongs
Joint Decoding
0.671 = 0.010
0.037 0.120
0.675
0.005
0.091 - 0.066
Bayesian Mutli Wcak
0.718 = 0.014
0.627 0.173
0.721 0.013
0.668 0.166
6.2 TEXT GENERATION
For the text generation task on slot filling, the performance of the student strong model and teacher weak model finetuned on ground-truth labels is presented in Table 3 The strong ceiling performance is 0.748, and the highest weak performance is 0.680.
The (eakk(S)to-SStoog performances are reported in Table For a single weak model, the Weak- to-Strong model performance didn't necessarily surpass the original weak performance (e.g. TinyL- 1a.a_v1.1), and the highest PGR is 0.399. With five weak models, our proposed Bayesian Multi-Weak approach achieved an average PGR of 0.668, which is 26% better than single weak model, 6% better than the naive baseline, and consistently better across three seeds. Comparing WeakS to-Strong per- formance with and without auxiliary loss, adding auxiliary loss is not effective for naive approaches, but improves the performance of Bayesian approaches. It indicates that the proposed Bayesian method, where the predictions of strong model are applied as part of the distribution estimation as in Eqn. (5). more effectively integrates strong model predictions into the training process, showing the effectiveness of our Bayesian approach for the model ensemble. The ablation study can be found in Appendix E A noticeable decline has been observed for Joint Decoding when the number of weak models increases from three five. This may result from the poor performance of the TinyLlama weak model which affects the quality of the generated weak target. Recall that in Joint Decoding, multiple weak models collaboratively determine one single target. In contrast, Naive Multi- Weak and Bayesian Multi-Weak methods still benefit from the increased number of weak models, which indicates that they are more robust against the quality of a single weak model. Additional experiments on Joint Decoding can be found in Appendix G for more analysis and insights.
Based on the strong model supervised by five weak models on the Bayesian Multi- Weeti--Wekka approach with auxiliary loss, cDPO training is conducted, which further train the model in student-fore foreing form. Three separate cDPO experiments were conducted using different initial models obtained in SFT stage with three different seeds. The results are shown in Table 5 After cDPO, three models

---

Published as a conference paper at ICLR 2025
Table Results before and after DPO. Based the Bayesian Multi-Weak model using five weak models (the -oatk--t--Stonn-55 setting) with auxiliary loss. Average results on three seeds are reported. Note that for different seeds, the initial SFT models are different.
seed-0
seed-1
seed-2
Average
SLU-FI
PGR
SLU-F1
PGR
SLU-FI
PGR
SLU-FI
PGR
Before cDPO
0.706
0.477
0.730
0.776
0.728
0.751
0.721 0.013
0.668 + 0.166
After cDPO
0.707
0.490
0.733
0.813
0.733
0.813
0.724 0.015
0.705 + 0.187
showed consistent performance improvements. The average PGR reached 0.705, 6% relatively better than that before DPO, with a maximum PGR of 0.813.
6.3 COMPLEMENTARITY OF WEAK MODELS
An experiment about the complementarity of different weak models was conducted. For classification models, the agreement is assessed by calculating the accuracy of each model's 's predictions on test set, treating outputs from other models as references. For generation models, the Levenshtein distance is calculated between different outputs from two models for a same input, which is obtained using the minimum edit number required to change one sequence to another. The average Levenshtein distance across all samples in the test set is used to measure the agreement between two models.
The results are shown in Figure 3 The agreement among different weak models among 0.75 for classification models, while for generation task the maximum agreement between different models 0.56. This suggests that the consistency among different weak models low, thus they can complement each other well the faults made by different weak models not the same. Morcover, comparing with Naive Multi-Weas approach, our proposed Bayesian method estimates a distribution based weak labels, learning patterns from the entire dataset, thereby increasing the tolerance for fault in weak models, as shown in results in Section 6. and 6.2
1.30
GPT2-Large 1.00
0.75
0.74
0.76
0.78
GPT2-Large
0.9
0.95
OFT
0.75
OPT-1.38.
0.36
1.00
0.42
0.34
0.40
0.8
-0.90
0.7
Pythla-1.4B.
Pyttia-1.48.
0.53
1.00
0.49
-0.55
BLDOM-181
0.78
0.75
0.74
1.00
0.78
BL0OM-181
Tinyllama v1.1
7.
0.76
0.76
0.78
1.00
Tinyt prytllamaavv v1.1
0.51
0.40
8.56
0.51
Lagee
OPT 38
Pythia 48
BL004-181
Tiryllmna v1.
GPTz-targe
OpT 38
Pymta 148
BLOOM 181
Tirelama v1.1
(a) Agreement of 5 models on classification.
(b) Agreement of 5 models on generation.
Figure 3: Agreement of weak models. The similarity between classification models was assessed by calculating the accuracy of each model's 's predictions against the others on the test set. For generation models, the agreement is obtained through the Levenshtein distance.
7 CONCLUSION
This paper extends the Seak-to-Stronp framework to WeakS-to-Strong by leveraging an ensemble of weak models to capture the variability in human opinions. We propose Bayesian inference method, Bayesian WeakS-to-Strong. to more accurately estimate the weak label distribution based on the outputs of multiple weak models. Additionally, while the original Weak-to-Strong method was limited to text classification tasks, this paper expands its applicability to text generation, enabling both the assessment of content trustworthiness and the generation of trustworthy content. Finally, DPO is utilized to enhance the student model's preference for learning, going beyond the traditional teacher-forcing approach. Our results demonstrate the effectiveness of Bayesian WeakS-tr tossS----tronngg for both classification and generation tasks, highlighting its potential for superalignment.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant laboratory for alignment. arXiv preprint 122.i::212.0086666 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprini arXiv:2309. 16609, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 050662, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional Al: Harmlessness from AI feedback. arXiv preprint wwxiv:2212.080777 2022b.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A spoken language understanding resource package. In Proc. EMNLP, 2020.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Proc. ICML, 2023.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXii:2312.099390) 2023.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92-110, 2022.
Mitchell Eric. A note on DPO with noisy preferences & relationship to IPO. https://
ericmi techthhhll.... ai /t.aai/ceae c/do. fd 2023.
Yoav Freund and Robert E Schapire. A desicion-theoreti, generalization of on-line learning and an application to boosting. In European conference on computational learning theory, pp. 23-37, 1995.
Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In Proc. ICML, 2020.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu. al. Reinforced self-training (ReST) for language modeling. arXiv preprint (tt::2308.89999 2023.
Jianyuan Guo, Hanting Chen, Chengcheng Wang. Kai Han, Chang Xu, and Yunhe Wang. Vision superalignment: Weak-to-strong generalization for vision foundation models. arXiv preprint arXiv:2402.03749, 2024.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proc. EMNLP, 2019.
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint -xXX::2402.024166 2024.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176B- parameter open-access multilingual language model. arXiv preprint arXiv:22/ 1.05100000000. 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with AI feedback. arXiv preprint arXiv:2309.00267. 2023.

---

Published as a conference paper at ICLR 2025
Breiman Leo. Bagging predictors. Machine Learning, 24:123-140 1996.
Yajiao Liu, Xin Jiang, Yichun Yin, Yasheng Wang, Fei Mi, Qun Liu, Xiang Wan, and Benyou Wang. One cannot stand for everyone! leveraging multiple user simulators to train task-oriented dialogue systems. In Proc. ACL, 2023.
Yucjiang Liu and Alexandre Alahi. Co-supervised learning: Improving weak-to weak-to-strong strong generalization with hierarchical mixture of experts. arXiv preprint arXiv:2402.15503 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo. Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint drXiv:2312.00886 2023.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. WebGPT: Browser-assisted gnuestion-nswwere with human feedback. arXiv preprint arXiv:2112.09332, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. In Proc. NeurlPS, 2022.
Silviu Paun and Edwin Simpson. Aggregating and learning from multiple annotators. In Proc. ACL, 2021.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Díaz. On releasing annotator-level labels and information in datasets. In Proc. ACL, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1:9, 2019.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proc. NeurlPS, 2023.
Jitao Sang, Yuhang Wang. Jing Zhang, Yanxu Zhu, Chao Kong, Junhong Ye, Shuyu Wei, and Jinlin Xiao. Improving weak-to- seak-to--ttoong generalization with scalable oversight and ensemble learning. arXiv preprint 24Xiv22422.00667 2024.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Proc. NeurIPS, 2018
Guangzhi Sun, Shutong Feng, Dongcheng Jiang, Chao Zhang, Milica Gašić, and Philip C Woodland. Speech- based slot filling using large language models. arXiv preprint arX14.2311.07418. 2023a.
Guangzhi Sun, Chao Zhang, Ivan Vulić, Pawel Budzianowski, and Philip C Woodland. Knowledge- aware audio-grounded generative slot filling for limited annotated data. arXiv preprint arXiv:2307.01764, 2023b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.009288 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proc. ICLR, 2022.
Johannes Welbi, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proc. W-NUT, 2017.
Wen Wu, Chao Zhang, Xixin Wu, and Philip C Woodland. Estimating the uncertainty in emotion class labels with utterfice-ppecific Dirichlet priors. IEEE Transactions Affective Computing, 2022.
Wen Wu, Chao Zhang, and Philip Woodland. Estimating the uncertainty in emotion attributes using deep evidential regression. In Proc. ACI., 2023.
12

---

Published as a conference paper at ICLR 2025
Peiyuan Zhang, Guangtao Zeng. Tianduo Wang. and Wei Lu. TinyLlama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 01066, 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08599, 2019.
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
The proposed Bayesian WeakS-to-Strong method was tested on two different types of tasks: text classification and generative slot filling. We believe the proposed method is general, and further experiments on other applications are reserved for future work. Due to computational resource limitations, experiments on only three and five weak models were conducted in this paper, mimicking the situation where human annotations are costly and time-consumin to obtain. We believe that the capabilities of the strong model can be recovered to a greater extent when more weak models are involved.
B BROADER IMPACT
As an approach that enhances the original Weak-To To-Strong method, our paper will have the following positive broader impact:
By ensuring strong LLMs behave in ways that are predictable and consistent with societal values, the proposed WeakS- To-Strong further increase public trust in Al technologies.
The use of weak model ensemble for strong model training helps to reduce risks of ethical violations such gender racial biases.
Multiple weak models can be more easily updated replaced to adapt to changing social norms and values. This flexibility allows LLMs to remain relevant and responsive to societal changes, ensuring that they continue to serve the public good over time.
This paper does not give rise to any additional potential biases beyond the ones directly inherited from the pre-trained LLM checkpoints. We encourage the practitioner to carefully select weak models such that the biases present in individual weak models do not accumulate or amplify when combined.
C IMPLEMENTATION DETAILS
All models were trained on NVIDIA A800 GPUs using the 6 data type. For the classification tasks, the Adam optimizer was used with a cosine learning rate scheduler and no warm-up period. The batch size was set to 32, with mini-batch size of 1. The weak models were finetuned on the ground-truth labels with an initial learning rate of 5 x 10 while the strong models were trained with a starting learning rate of 10 (both the weak labels and ground-trutt trruttttthhht. labels). The Weak(S) -SokkSS-tttttongg training was run for two epochs. The weights for different weak models (A) was to average values simplify the classification experiments (refer to Appendix F for comparison between average and fixed weights).
For generation tasks, the AdamW optimizer no warm-up. The initial learning rates were set at 4 x 10-5 for GPT2-Large and Pythia-1.4B. and 8 x 10-5 for O.T-,,3B, with batch size of 8 (mini-batch size of 4). These models were trained for 15 epochs. The checkpoints with the lowest validation loss were selected to ensure the quality of weak labels produced by the weak models. The strong model was trained with a batch size of (mini-batch size of I) and initial learning rate of 1 x 10-5, evaluated at the end of two epochs. In the training for strong ceiling performance, the hyperparameters were adjusted based on the validation set. For Weak-to- torkk-ng- training, in which case the ground truth is not accessible, we aligned the hyperparameter settings with those used in strong ceiling training. The weights À in Equ. 4 were set to (0.1, 0.3, 0.2, 0.3, 0.1) for GPT2-Large, OPT-1.3B, Pythia-1. 1.4B, BLOOM-ID IB1 and Tiny Llama v1.1 respectively.
used with linear learning rate scheduler, also with
For DPO, the initial learning rate was set to 5 x 10 for two epochs, with the cDPO's hyperparameter B set to 2.0 and label smoothing as 0.1. Other settings remain the same as the generation tasks.
D EXPERIMENTS ON AN ADDITIONAL DATASET
To verify the generalization of our method, additional experiments were conducted on another dataset, CosmosQA (Huang al. 2019), for the classification task. CosmosQA is large-scale dataset of
14

---

Published as a conference paper at ICLR 2025
problems that require commonsense based reading comprehension. Like the preprocess the SciQ dataset, 5k data samples were extracted for training weak models, another 5k samples for strong models, and 1k samples for testing. This dataset was also reformatted into binary classification format (i.e., determining correctness). Experiments with three weak models were conducted. The weak model performance and strong performance are listed in Table 6 and Weak(S))-o-Stiee performance in Table 7
Table Performance of single model on text classification task on CosmosQA dataset. Trained by ground-truth labels.
Pre-trained model
# Param
Accuracy
Strong Model (ceiling)
Qwen-7B
7.7B
0.847
GPrg-Large
0.8B
0.642
Weak Model
OPT-1.3B
1.3B
0.642
Pythia-1.4)
1.4B
0.654
Table 7:
performance on text classification task for with and without auxiliary loss on CosmosQA dataset. Experiments with weak models were conducted. Each experiment was run with three different random seeds. The best results are shown in bold.
w/o aux loss
w/ aux loss
Accuracy
PGR
Accuracy
PGR
GPT2-Large
0.652 + 0.007 0.077 + 0.032 0.651 0.007 0.073 + 0.032
Weak-to-Strong
OPT-1.3B
0.687 + 0.005 0.218 + 0.022
0.704
0.009 0.304 + 0.044
Pythia-I.4P
0.685 + 0.010 0.162 + 0.052
0.731
0.002
0.399 0.009
Naive Multi-Weak
0.691 + 0.009 0.232 + 0.043
0.698
0.007
0.267 + 0.034
SoakS-to-Stronge
FlyingSquid
0.699 + 0.004
0.272 + 0.020
0.706
0.004
0.303 0.030
Bayesian Mutli Weak
0.694 1 0.007 0.247 + 0.033 0.760
0.005 0.571 + 0.025
Table 6 shows that, on the CosmosQA dataset, both the weak and strong models perform worse compared that on the SciQ dataset, with the weak model's performance dropping more. In the (S-ooS))to experiments, neither multi-weak baseline outperformed the best single model (Pythia-1.44 with auxiliary loss). while our Bayesian approach significantly exceeded it, showing the effectiveness of our method, especially when weak model performance varies.
E ABLATION STUDY ON PROBABILITY ESTIMATION FOR WEAK SEQUENCES
The importance of probability estimation for weak sequences was explored in this section. Two key steps were calculating target wordpiece probability using the word as a gap bridge, and estimating probabilities of other categories by scaling the strong output (see Section 4.1 for details). The results are shown in Table g The average PGR using a one-hot label (Line 1) is negative, showing that WeakS-to-Stronn performance doesn't 't surpass the weak model without probability estimation. Estimating probability for the target wordpiece significantly improved performance compared to using one-hot labels. This indicates the necessity of introducing target probabilities during training, which allows the strong student model to Iearn the weak model's confidence for its generated labels, considering may be incorrect. Based this, adding probabilities of other categories, the overall precision of target label estimation improves, resulting in about 20% PGR gain. This experiment highlights the importance of precise probability estimation for weak sequences.
F WEIGHTS OF WEAK MODELS
The selection for weights of weak models was explored in this section, where experiments average, dynamic and fixed weights were conducted. For average, the weight X; in Eqn. 2 and Eqn. (4) is set to the same value for each weak model. That is to say, the losses calculated for each weak model respectively and then averaged as the final loss. For fixed weight, the weights are set to a fixed value based the performance of different weak models (refer to C for specific value). For
15

---

Published as a conference paper at ICLR 2025
Table 8: Ablation study on probability estimation for weak sequence. Tested whether calculate the probability for target wordpiece, and the estimation for probabilities for other categories.
Target wordpiece
Other category
SLU-FI
PGR
0.656 + 0.016
-0.145 + 0.204
0.704 + 0.025
0.453 + 0.305
V
0.721 = 0.013
0.668 = 0.166
dynamic weight, for a certain sample, the confidence in the utterance of weak model is treated as a weight for that weak model, considering the importance of different weak models could vary on different samples. The results are shown in Table 9 Compared the average, fixed weight improves the average PGR by about 8% because it assigns weights based on the performance of different weak models, rather than just averaging. Dynamic weight shows noticeable decline in performance compared to fixed weight. It may be because our proposed per-token target probability already provided the student model with tnn-grainne information about the reliability of the label, making dynamic weight entirely redundant.
Table Different weighting strategies for weak models: experiments using average, dynamic and fixed weights.
SLU-FI
PGR
Average 0.715 + 0.019
0.589 + 0.237
Dynamic weight
0.687 0.013
0.241 + 0.163
Fixed weight
0.721 + 0.013
0.668 + 0.166
G PERFORMANCE OF JOINT DECODING
G.1 THE IMPACT OF THE QUALITY OF WEAK MODELS
This section provides additional experiments where different weak model combinations are used: (i) three weak models (GPT2-Large, OPT-1.3B, Pythia-1.4B), (ii) four models (adding BLOOM-IBI): and (iii) all five models (further adding TinyLlama.vi.I) For TinyLlama.vl.. two setups were investigated: (i) only using it to generate weak target sequence but not for scoring (scoring done by other four models); (ii) using it for both generation and scoring. The results are listed in Table TŌ Compared to the siree-weak-mode system, introducing the 4th model shows 10% increase in PGR. However, incorporating TinyLlama as the 5th weak model undermines the performance (last row in Table I0). even nearly fails to surpass weak model performance. A possible reason is that TinyLlama performs poorly the text generation task. The impact of the TinyLlama can be reduced by excluding from scoring. As shown in the second to the last row of Table TO without TinyL scoring, the results with five weak models scoring ability prevents it from selecting a good target sequence among the candidates and the poor quality from certain weak model can largely impact the overall results for Joint Decoding methods. In contrast, as discussed in Section 6.2, the proposed Bayesian WeakS-to- -t-rong method is more robust against the quality of single weak model.
close those with three. This indicates that TinyLlama' 's poor
G.2 THE IMPACT OF BEAM SIZE
As introduced in Section 5.2 in Joint Decoding, each weak model generates M output sequences by beam search decoding. This section investigates Joint Decoding with different beam sizes M, as shown in Table II For WeakS-to-Strone to-Stron results with different M yields similar results. However, for WeakS-tar to-tt-Stt Soo-Strong-5, with TinyLlama included in the weak model set, the result with M = 10 is significantly worse than that with M 3 and M 5. The experiments on beam size further demonstrate that the poor results of the WeakS- WeakS-to-Stronn to-Strong-5 are due to its weak ability to select the target sequence, M 10 introduces more distractions than M 5.
16

---

Published as a conference paper at ICLR 2025
Table 10: Joint Decoding performance with different weak models. Experiments with three weak models (GPT2-Large, OPT-1.3B, Pythia-1.4B), four weak models (adding BLOOM-IBI), and all five weak models are conducted.
TinyLlama v1.1
3 Weak models
BLOOM-IBI IB1
SLU-FI
PGR
Generating Scoring
0.703 + 0.003
0,434 + 0.032
0.711 + 0.008
0.549 0.100
0.706 = 0.009
0,452 + 0.086
0.671 = 0.010
0.037 0.120
Table 11: Joint Decoding performance with different beam sizes in beam search when weak models generate labels. Both WeakS-to-Stroog-3 and WeakS-to-Strong-5 are explored.
SeakS-to-Strong3
WeakS-to-Stroo tt-Stttt
SLU-FI
PGR
SLU-FI
PGR
M=3
0.706 + 0.012
0.471
0.155
0.669 = 0.023
0.022 + 0.281
M=5
0.703 + 0.003
0.434 + 0.032
0.671 = 0.010
0.037 = 0.120
M=10
0.703 + 0.004
0.431 + 0.045
0.650 + 0.004
-0.224 + 0.050
H REGULARISING TERM OF EDL
As introduced in Section 3.3 the negative log-likelihood of a sample y with predicted Dirichlet prior with hyperparameter a is:
NNL = -lg P(yulT)p((T)a)))
te (2) (log(or)) - log(ar))
When a sample is not correctly classified, is expected the total evidence shrinks to zero for the sample. Taking this into consideration, Sensoy et al. (2018) added regularization term penalise the misleading evidence. The loss with this regularising term reads
LEDI. = LNLI + Cx (Dir( n (z) [Dr[|Di
where the KL term refers to the CREG in Section 3.3, Dir(rr() denotes a Dirichlet distribution with zero total evidence, ā y non-misleading evidence from predicted ar, and Xe is the annealing coefficient. By adding a KI.- divergence between the Dirichlet distribution with misleading evidence and zero total evidence, the total evidence is enforced to shrink to zero for the simple which is not correctly classified. The annealing coefficient increases by training step, enabling the model to explore the parameter space.
y)
the Dirichlet parameter after removal of the
I PROMPT USED EXPERIMENT
The prompt used for slot filling task is shown below, in which the input reference transcription refers to the reference transcription as the input. For example, with input "remind me about my business meeting at 3 and 45 pm", the expected output is " ("time"; "3 and 45 pm")'
J LICENSES FOR EXISTING ASSETS
Models and datasets we used in this work were all downloaded from HuggingFace website, expect SLURP which downloaded from https: //tgthtb.bb//t////// http://ww..eeefjjasaa////rrrr The licenses and paths for each asset used is listed below:
We provide the following links special licenses below:
Modified MIT License for GPT2-Large: brtps: com/ com/oppeai/ppt-2
blob//mstter//ICRN57
17

---

Published as a conference paper at ICLR 2025
USER:
Consider the following list of slot types provided to you:
"event name", "datc", "person", "time", "news topic", "relation", "list name", "media_tyy type", "business name", "weather descriptor", "music genre", "house place", "game name", "food type". "timeofday", "place_nat name", "definition word", "email address". "transport ageney", "movie name",,__mm "artist name", "transport " "joke type", "movie type", "time zone", "music descriptor", "device type". "color_type".. "meal type", "player setting", "podcast name", "email folder", "song name", "change amount", "business type", "personal info", "radio name", "coffèe type", "audiobook author", "audiobook name", "currency name", "playlist name", "podcast "deerrppor", "general frequency", "music album", "app name", "order stper, "transport name", "transport descriptor", "cooking type", "ingredient", "alarm pappe__tpp"__""_" "drink type", "sport "sport_type", "game_type"
Now consider the following sentencc(s) containing one or more of the above slot types. Can you extract slots belonging that slot list and their values in json format i.c. ("slot type": "value")? ONLY print out the json, only print { if no slot. "(input reference transcription3"
ASSISTANT:
Model/Dataset
License
Huggingface Path
GPT2-Large
Modified MIT License
openai aaanaai-commmiiiggp----aaag facebook/opt-L.36 EleutherAl/pythia-l..
OPT-1.3B
MIT license
Pythia-1.4B
Apache 2.0
BLOOM-1BI
RAIL License v1.0
bigsciencc/bloom-Ibe
TinyLlama v1.1
Apache 2.0
Tinymlama v1.1 Qwen/Qwen-7B
Qwen-7B
Tongyi Qianwen LICENSE AGREEMENT
Llama2-7B
Custom commercial license
mntta-llama/lllma-2776
SciQ
CC BY-NC 3.0 DEED CC BY 4.0
allenai/sciq
SLURP
N/A
CosmosQA
CC BY 4.0
allenai/cosmos.cg
RAIL License v1.0 for BLOOM-IBI bigscience/License
https//hugginffocc.c//gees
Tongyi Qianwen LICENSE AGREEMENT http::///gttabbb..co///eel/////
blob
Custom commercial license for Llama-2: https://ai.meta
18


Reviews:
[{"review_id": "DmJ4L0kC11", "reviewer": "Reviewer_WJwn", "summary": "This paper extends work in Weak-to-Strong generalization in two notable ways: (1) considering the setting in which there is an ensemble of multiple weak teachers; (2) expanding from more basic text classification to generation-based tasks. For (1), they show that using a Bayesian approach (based on EDL) is effective for learning from the ensemble of weak teachers, improving upon any of the weak teachers individually as well as simpler baselines that weight typical cross-entropy losses across teachers. For (2), they overcome the key issue that weak and strong models may have different tokenizers and also try applying a modified DPO stage on top, reporting a positive effect.", "questions": "**(Q1) How exactly were the the $\\lambda$’s set?** Related to (W3) above, the appendix does not say exactly what the final $\\lambda$'s are for the \"Fixed weight\" approach in Appendix E and how \"the performance of different weak models\" was used.\n\n**(Q2) How is PGR calculated when you have multiple weak models?** The definition provided in Sec 5.3 is clear when there is a single weak model but there are perhaps multiple ways to define in the WeakS-to-Strong setting (e.g. max v.s. average over the weak models in). \n\n**(Q3) Miscellaneous clarifications**\n- In Eq. 4, what is $\\hat{y}_w^{(i)}$? Should it not just be the label $k$; in general it seems something in the inner sum should depend on  $k$?\n- In Eq. 7, my understanding was that we wanted $P(s_1)P(s_2) = P(W)$ but here it seems like their sum will be equal to P(W) instead of their product.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**(S1) Originality and Significance** Studying the WeakS-to-Strong setting is well-motivated and using the proposed Bayesian method here is novel. Furthermore, the expansion to from binary verification tasks to generation is notable since the latter more realistically reflects LLM use cases. \n\n**(S2) Quality** The paper’s proposed method results in notable improvements on the studied tasks, outperforming the individual weak model teachers in isolation and the naive baseline that simply does a weighted average over the standard CE-based losses. I also appreciated the extra ablations the authors ran to verify the importance of certain aspects of their design (e.g., the aux loss, probability estimation for weak sequences and the $\\lambda$ weights across tasks). \n\n**(S3) Clarity** Overall, I found the writing to be clear, doing a good job of contextualizing methods/results in terms of prior work.", "weaknesses": "**(W1) Significance of some results.** I found that certain claims in the manuscript were not as strongly supported by the empirical results. For instance, it appears that the improvements obtained from the DPO-based stage weren't too big (i.e., within standard deviations across runs). Another case was the claim in L461 about the auxiliary loss improving Bayesian methods. That being said, this doesn't affect my opinion about what I consider to be the main result, i.e., the superiority of Bayesian approach. \n\n**(W2) Other baselines and tasks.** There are some other baselines that I think would be good to consider, namely:\n- Methods from data programming / programmatic weak supervision (e.g Snorkel, FlyingSquid) to aggregate the labels generated by different weak models (i.e, treating them as Labeling Functions). In any case, it would be nice to discuss this line of work given that they also operate in a setting where the goal is aggregate multiple weak label sources. \n- In the related work, the authors mentioned that some previous works (Liu & Asahi, 2024; Sang et al., 2024) tried traditional ensembling methods. Perhaps these would also be useful baselines?\n\nIn terms of evaluations, it would strengthen the paper if the authors could show similar results on more tasks per category (i.e., classification, generation). Currently the results are limited to one task for each. \n\n**(W3) Clarification on key assumptions.**  Some further discussion would be helpful on the following\n- From cross-referencing Appendix E with the main result tables, it appears that the best results rely on the \"Fixed Weight\" approach which is based on how good each of the weak models are individually. The assumption here (which I think should be made more explicit), seems to be that one is actually able to accurately estimate the weak model performances. I wonder how applicable this assumption is for real superhuman tasks where presumably getting ground-truth labels is difficult. Perhaps the authors can comment here? \n- Given one stated motivation for the study of WeakS-to-Strong was to consider the variation in human opinions, how might the methods/results generalize to when supervision actually comes from humans rather than weak models? In particular, it seems that in the current experiments, having confidence estimates associated with the weak model labels is important, but this may not readily exist for human-generated labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OdLLAWtKDe", "reviewer": "Reviewer_zdng", "summary": "The paper extends the weak-to-strong training to multi-weak-to-strong or weaks-to-strong. In a weak-to-strong setting, a smaller model provides a training signal for a larger model. This paper proposes to use multiple small models to provide the training signal. To squize out the best from the ensemble of small models, the paper proposes a Bayesian strategy based on evidential deep learning. The paper also performs sequence generation training, which was not done previously in the weak-to-strong setting. The results demonstrate that weaks-to-strong provides produces better results compared to weak-to-strong in both classification as well as generation.", "questions": "**Suggest**: As I understand, the regularization loss is critical. I suggest moving the corresponding discussion to the main paper. I also suggest mentioning and discussing the exact values of the weights in the loss to the main paper, as it can provide some insight into how the method works.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The paper is the first to successfully extend weak-to-strong to the weaks-to-strong setting. The paper also contains several useful ideas for performing sequence-level training using soft-labels when the student and the teacher have different tokenizers.\n\n**Clarity**: The paper is written well and is easy to follow.\n\n**Quality**: The experiments are well structured to answer the central questions posed by the paper, i.e., is weaks-to-strong better than weak-to-strong, and is the proposed method of performing weaks-to-strong training better than naive baselines? \n\n**Significance**: Based on the quality of the experiments and the topic addressed, I believe that a broad community will find the paper useful.", "weaknesses": "1. Some of the design choices are not well justified. For example, for the sequence-level training using soft labels, it is not clear why the strategy proposed in section 4.1 should work or is optimal. \n2. The datasets selected for the classification and generation tasks seem very specific. Is there a particular reason for selecting only these? The impact of the paper could be much better if model general datasets were included, at least for the generation task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ypA5MecCP", "reviewer": "Reviewer_JkWZ", "summary": "The paper introduces a Bayesian-based method inspired by the weak-to-strong approach for the classification task of language models. The author also extends the algorithm for the generation task by some simple transformations. To improve the performance of the generation task further, the author utilized DPO to finetune the model at the tail of the framework.", "questions": "- In Equation 3, are $\\alpha_k$s are trainable parameters?\n- Is the Equation 3 the loss for each sample?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposal is simple but works.", "weaknesses": "- The motivation for using Dirichlet distribution to model the prior is ambiguous.\n- The writing for the methodology description is unclear and hard to follow\n- The transformations in Equation 3 should be clarified.\n- What is the definition of $p_k$ in Equation 4?\n- In Section 4.1, the probability of the target token of the strong model changing after each update of network parameters because $C_s(s_1)$ and $C_s(s_2)$ shift. This means that the targets vary along the training, likely creating instability or leading to divergence. The time consumed for recomputing these terms is also one of my concerns.\n- All baselines that are naive and very simple algorithms are weak for both classification and generation tasks. It is suggested that stronger baselines be included for both tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2FShD8ArH4", "reviewer": "Reviewer_grX8", "summary": "This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS- to-Strong generalization.\n\nFurthermore, they broaden the use of the WeakS-to-Strong approach from its original focus on text classification to encompass text generation tasks(this point causes confusion for me, see in the weakness below), exploring more sophisticated supervision strategies. Additionally, direct preference optimization is employed to enhance the student model's ability to learn preferences, moving beyond the traditional teacher-forcing training framework(I think this contribution is not clear and verified by comprehensive experiments). \n\nI am not an expert in this domain, but I read this paper twice to get the paper's idea. I am open to other reviewers' opinion.", "questions": "1. I didn't get the idea behind \"The description about complementary of weak models\"(Section 6.3). I can tell that classification enjoy better agreement than generation(as generally expected), but I didn't get \"This suggests that the consistency among different weak models is low, thus they can complement each other well as the faults made by different weak models are not the same\"", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. They propose a Bayesian WeakS-to-Strong framework that improves weak supervison.(Although I tend to say this improvement is not well-proved). \n2. Extention of Weak-to-Strong framework from text classification to generation tasks.\n3. A method for estimating token-level probabilities is introduced to create soft labels for training robust models.", "weaknesses": "I have the following concerns regarding to this paper: \n\n1. The experiments is not comprehensive: the experiment they conduct are mainly on the ~1B weak models and the strong model are ~7B. I think from the experiment results conducted now, the claim on WeakS-to-Strong in the paper is slightly overclaimed. (**The major weakness in my opinion**)\n2. The improvement shown in the experiment is not observable: for instance, for accuracy: 0.843(baseline) vs 0.850(WeakS-to-Strong-3). The improvement is negligible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)