Bayesian WeakS-to-Strong from Text Classification to Generation

Authors: Ziyun Cui, Ziyang Zhang, Guangzhi Sun, Wen Wu, Chao Zhang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: weaks-to-strong, bayesian, generation, per token

Abstract:
Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them weakly. Weak-to-Strong mimics such a scenario where weak model supervision attempts to harness the full capabilities of a much stronger model. This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS-to-Strong generalization. Furthermore, we extend the application of WeakS-to-Strong from text classification tasks to text generation tasks where more advanced strategies are investigated for supervision. Moreover, direct preference optimization is applied to advance the student model's preference learning, beyond the basic learning framework of teacher forcing. Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superalignment.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 3

Reviews and discussion:
<thread>
creation date:
1729499438899

thread_type:
official_review

<thread_object>
creator: reviewer_grx8

<content>
summary:
This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS- to-Strong generalization.

Furthermore, they broaden the use of the WeakS-to-Strong approach from its original focus on text classification to encompass text generation tasks(this point causes confusion for me, see in the weakness below), exploring more sophisticated supervision strategies. Additionally, direct preference optimization is employed to enhance the student model's ability to learn preferences, moving beyond the traditional teacher-forcing training framework(I think this contribution is not clear and verified by comprehensive experiments). 

I am not an expert in this domain, but I read this paper twice to get the paper's idea. I am open to other reviewers' opinion.

strengths:
1. They propose a Bayesian WeakS-to-Strong framework that improves weak supervison.(Although I tend to say this improvement is not well-proved). 
2. Extention of Weak-to-Strong framework from text classification to generation tasks.
3. A method for estimating token-level probabilities is introduced to create soft labels for training robust models.

weaknesses:
I have the following concerns regarding to this paper: 

1. The experiments is not comprehensive: the experiment they conduct are mainly on the ~1B weak models and the strong model are ~7B. I think from the experiment results conducted now, the claim on WeakS-to-Strong in the paper is slightly overclaimed. (**The major weakness in my opinion**)
2. The improvement shown in the experiment is not observable: for instance, for accuracy: 0.843(baseline) vs 0.850(WeakS-to-Strong-3). The improvement is negligible.

questions:
1. I didn't get the idea behind "The description about complementary of weak models"(Section 6.3). I can tell that classification enjoy better agreement than generation(as generally expected), but I didn't get "This suggests that the consistency among different weak models is low, thus they can complement each other well as the faults made by different weak models are not the same"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730083635390

thread_type:
official_review

<thread_object>
creator: reviewer_jkwz

<content>
summary:
The paper introduces a Bayesian-based method inspired by the weak-to-strong approach for the classification task of language models. The author also extends the algorithm for the generation task by some simple transformations. To improve the performance of the generation task further, the author utilized DPO to finetune the model at the tail of the framework.

strengths:
- The proposal is simple but works.

weaknesses:
- The motivation for using Dirichlet distribution to model the prior is ambiguous.
- The writing for the methodology description is unclear and hard to follow
- The transformations in Equation 3 should be clarified.
- What is the definition of $p_k$ in Equation 4?
- In Section 4.1, the probability of the target token of the strong model changing after each update of network parameters because $C_s(s_1)$ and $C_s(s_2)$ shift. This means that the targets vary along the training, likely creating instability or leading to divergence. The time consumed for recomputing these terms is also one of my concerns.
- All baselines that are naive and very simple algorithms are weak for both classification and generation tasks. It is suggested that stronger baselines be included for both tasks.

questions:
- In Equation 3, are $\alpha_k$s are trainable parameters?
- Is the Equation 3 the loss for each sample?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730406448118

thread_type:
official_review

<thread_object>
creator: reviewer_zdng

<content>
summary:
The paper extends the weak-to-strong training to multi-weak-to-strong or weaks-to-strong. In a weak-to-strong setting, a smaller model provides a training signal for a larger model. This paper proposes to use multiple small models to provide the training signal. To squize out the best from the ensemble of small models, the paper proposes a Bayesian strategy based on evidential deep learning. The paper also performs sequence generation training, which was not done previously in the weak-to-strong setting. The results demonstrate that weaks-to-strong provides produces better results compared to weak-to-strong in both classification as well as generation.

strengths:
**Originality**: The paper is the first to successfully extend weak-to-strong to the weaks-to-strong setting. The paper also contains several useful ideas for performing sequence-level training using soft-labels when the student and the teacher have different tokenizers.

**Clarity**: The paper is written well and is easy to follow.

**Quality**: The experiments are well structured to answer the central questions posed by the paper, i.e., is weaks-to-strong better than weak-to-strong, and is the proposed method of performing weaks-to-strong training better than naive baselines? 

**Significance**: Based on the quality of the experiments and the topic addressed, I believe that a broad community will find the paper useful.

weaknesses:
1. Some of the design choices are not well justified. For example, for the sequence-level training using soft labels, it is not clear why the strategy proposed in section 4.1 should work or is optimal. 
2. The datasets selected for the classification and generation tasks seem very specific. Is there a particular reason for selecting only these? The impact of the paper could be much better if model general datasets were included, at least for the generation task.

questions:
**Suggest**: As I understand, the regularization loss is critical. I suggest moving the corresponding discussion to the main paper. I also suggest mentioning and discussing the exact values of the weights in the loss to the main paper, as it can provide some insight into how the method works.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711065545

thread_type:
official_review

<thread_object>
creator: reviewer_wjwn

<content>
summary:
This paper extends work in Weak-to-Strong generalization in two notable ways: (1) considering the setting in which there is an ensemble of multiple weak teachers; (2) expanding from more basic text classification to generation-based tasks. For (1), they show that using a Bayesian approach (based on EDL) is effective for learning from the ensemble of weak teachers, improving upon any of the weak teachers individually as well as simpler baselines that weight typical cross-entropy losses across teachers. For (2), they overcome the key issue that weak and strong models may have different tokenizers and also try applying a modified DPO stage on top, reporting a positive effect.

strengths:
**(S1) Originality and Significance** Studying the WeakS-to-Strong setting is well-motivated and using the proposed Bayesian method here is novel. Furthermore, the expansion to from binary verification tasks to generation is notable since the latter more realistically reflects LLM use cases. 

**(S2) Quality** The paper’s proposed method results in notable improvements on the studied tasks, outperforming the individual weak model teachers in isolation and the naive baseline that simply does a weighted average over the standard CE-based losses. I also appreciated the extra ablations the authors ran to verify the importance of certain aspects of their design (e.g., the aux loss, probability estimation for weak sequences and the $\lambda$ weights across tasks). 

**(S3) Clarity** Overall, I found the writing to be clear, doing a good job of contextualizing methods/results in terms of prior work.

weaknesses:
**(W1) Significance of some results.** I found that certain claims in the manuscript were not as strongly supported by the empirical results. For instance, it appears that the improvements obtained from the DPO-based stage weren't too big (i.e., within standard deviations across runs). Another case was the claim in L461 about the auxiliary loss improving Bayesian methods. That being said, this doesn't affect my opinion about what I consider to be the main result, i.e., the superiority of Bayesian approach. 

**(W2) Other baselines and tasks.** There are some other baselines that I think would be good to consider, namely:
- Methods from data programming / programmatic weak supervision (e.g Snorkel, FlyingSquid) to aggregate the labels generated by different weak models (i.e, treating them as Labeling Functions). In any case, it would be nice to discuss this line of work given that they also operate in a setting where the goal is aggregate multiple weak label sources. 
- In the related work, the authors mentioned that some previous works (Liu & Asahi, 2024; Sang et al., 2024) tried traditional ensembling methods. Perhaps these would also be useful baselines?

In terms of evaluations, it would strengthen the paper if the authors could show similar results on more tasks per category (i.e., classification, generation). Currently the results are limited to one task for each. 

**(W3) Clarification on key assumptions.**  Some further discussion would be helpful on the following
- From cross-referencing Appendix E with the main result tables, it appears that the best results rely on the "Fixed Weight" approach which is based on how good each of the weak models are individually. The assumption here (which I think should be made more explicit), seems to be that one is actually able to accurately estimate the weak model performances. I wonder how applicable this assumption is for real superhuman tasks where presumably getting ground-truth labels is difficult. Perhaps the authors can comment here? 
- Given one stated motivation for the study of WeakS-to-Strong was to consider the variation in human opinions, how might the methods/results generalize to when supervision actually comes from humans rather than weak models? In particular, it seems that in the current experiments, having confidence estimates associated with the weak model labels is important, but this may not readily exist for human-generated labels?

questions:
**(Q1) How exactly were the the $\lambda$’s set?** Related to (W3) above, the appendix does not say exactly what the final $\lambda$'s are for the "Fixed weight" approach in Appendix E and how "the performance of different weak models" was used.

**(Q2) How is PGR calculated when you have multiple weak models?** The definition provided in Sec 5.3 is clear when there is a single weak model but there are perhaps multiple ways to define in the WeakS-to-Strong setting (e.g. max v.s. average over the weak models in). 

**(Q3) Miscellaneous clarifications**
- In Eq. 4, what is $\hat{y}_w^{(i)}$? Should it not just be the label $k$; in general it seems something in the inner sum should depend on  $k$?
- In Eq. 7, my understanding was that we wanted $P(s_1)P(s_2) = P(W)$ but here it seems like their sum will be equal to P(W) instead of their product.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732069787023

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We appreciate the detailed comments and constructive feedback from the reviewers. We respond to the common questions below and provide individual responses to each reviewer for your comments. We hope our response fully resolves your concerns.

### G.1 Addional baseline

We added FlyingSquid[1], which is designed for binary classification, as an additional baseline for classification tasks. It is developed for weak supervision, using latent variable probabilistic graphical models to model correlations among noisy labels without any ground truth data. Through this method, we get a label model with multiple weak labels, and obtain the probability for the positive category, which is then used as a soft label in strong model training. We compared this baseline with the origin naive baseline and Bayesian method, on the classification task. The results for both 3 weak models and 5 weak models are listed below:

| WeakS-to-Strong-3 | Acc w/o aux_loss | PGR w/o aux_loss | Acc w/ aux_loss | PGR w/ aux_loss |
|-|-|-|-|-|
| Naive Multi-Weak | 0.817 $\pm$ 0.002 | 0.586 $\pm$ 0.008 | 0.831 $\pm$ 0.013 | 0.662 $\pm$ 0.064 |
| FlyingSquid | 0.809 $\pm$ 0.005 | 0.549 $\pm$ 0.026 | 0.825 $\pm$ 0.003 | 0.631 $\pm$ 0.013 |
| Bayesian Multi-Weak | $\mathbf{0.819 \pm 0.006}$ | $\mathbf{0.600 \pm 0.033}$ | $\mathbf{0.850 \pm 0.006}$ | $\mathbf{0.756 \pm 0.028}$ |

| WeakS-to-Strong-5 | Acc w/o aux_loss | PGR w/o aux_loss | Acc w/ aux_loss | PGR w/ aux_loss  |
|-|-|-|-|-|
| Naive Multi-Weak | 0.832 $\pm$ 0.005 | 0.641$\pm$ 0.025 | 0.853 $\pm$ 0.006 | 0.754 $\pm$ 0.032  |
| FlyingSquid | $\mathbf{0.832 \pm 0.004}$ | $\mathbf{0.643 \pm 0.023}$ | 0.855 $\pm$ 0.007 | 0.768 $\pm$ 0.035  |
| Bayesian Multi-Weak | 0.831 $\pm$ 0.008 | 0.627 $\pm$ 0.027 | $\mathbf{0.866 \pm 0.006}$ | $\mathbf{0.828 \pm 0.038}$   |

As shown in the tables, the FlyingSquid performs consistently worse than Naive Multi-Weak with three weak models, overall worse than Naive Multi-Weak when five weak models were used. Neither surpasses our proposed Bayesian method. The reason might be that five weak labels allow FlyingSquid to model label relationships more comprehensively, while simply averaging labels (naive approach) loses information. It doesn't outperform Bayesian because FlyingSquid is designed for hard labels and can't handle the confidence of weak models.

### G.2 Additional experiments on another dataset

We experimented on another dataset, CosmosQA[2], for the classification task. CosmosQA is a large-scale dataset of problems that require commonsense-based reading comprehension. Like the preprocess on the SciQ dataset, 5k data samples were extracted for training weak models, another 5k samples for strong models, and 1k samples for testing. This dataset is also reorganized into a binary classification format (i.e., determining correctness). Due to computational and time constraints, we only conducted experiments with three weak models. The results are listed below:

| Train on groundtruth | Accuracy  |
|-|-|
| GPT2-Large | 0.636  |
| OPT-1.3B | 0.642  |
| Pythia-1.4B | 0.654  |
| Qwen-7B | 0.847   |

*Table 1: Performance of single model on CosmosQA dataset. Trained on ground-truth labels.*

| Single Weak Model | Acc w/o aux_loss | PGR w/o aux_loss | Acc w/ aux_loss | PGR w/ aux_loss  |
|-|-|-|-|-|
| GPT2-Large | 0.652 $\pm$ 0.007 | 0.077 $\pm$ 0.032 | 0.651 $\pm$ 0.007 | 0.073 $\pm$ 0.032  |
| OPT-1.3B | 0.687 $\pm$ 0.005 | 0.218 $\pm$ 0.022 | 0.704 $\pm$ 0.009 | 0.304 $\pm$ 0.044  |
| Pythia-1.4B | 0.685 $\pm$ 0.010 | 0.162 $\pm$ 0.052 | 0.731 $\pm$ 0.002 | 0.399 $\pm$ 0.009   |

*Table 2: Weak-to-Strong performance with single weak model.*

| WeakS-to-Strong-3 | Acc w/o aux_loss | PGR w/o aux_loss | Acc w/ aux_loss | PGR w/ aux_loss  |
|-|-|-|-|-|
| Naive Multi-Weak | 0.691 $\pm$ 0.009 | 0.232 $\pm$ 0.043 | 0.698 $\pm$ 0.007 | 0.267 $\pm$ 0.034  |
| FlyingSquid | $\mathbf{0.699 \pm 0.004}$ | $\mathbf{0.272 \pm 0.020}$ | 0.706 $\pm$ 0.004 | 0.303 $\pm$ 0.030  |
| Bayesian Multi-Weak | 0.694 $\pm$ 0.007 | 0.247 $\pm$ 0.033 | $\mathbf{0.760 \pm 0.005}$ | $\mathbf{0.571 \pm 0.025}$   |

*Table 3: WeakS-to-Strong performance with three weak models.*

From the results, neither baseline outperformed the best single model (Pythia-1.4B with auxiliary loss), while our Bayesian approach significantly exceeded it, showing the effectiveness of our method, especially when weak model performance varies.

[1] Fu, D., Chen, M., Sala, F., Hooper, S., Fatahalian, K., & Ré, C.. Fast and three-rious: Speeding up weak supervision with triplet methods. In Proc. ICML. 2020.

[2] Huang, L., Le Bras, R., Bhagavatula, C., & Choi, Y.. Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning. In Proc. EMNLP. 2019.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732527063225

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised version of the manuscript

comment:
We thank the reviewers for providing valuable suggestions and engaging in the discussion. We have updated the manuscript accordingly with the following revisions:
- Added an additional baseline and one more dataset (Section 5.2, 6.1, Appendix D).
- Motified explanation for the formulas for clarification (Section 3.3, 4.1).
- Added the specific value of weak models' weights $\lambda$ (Appendix F).

All changes are shown in blue. We appreciate reviewers' time and effort, and hope the revisions strengthen the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733231022077

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Responses

comment:
Dear all reviewers, Area Chair, and Senior Area Chairs,

We thank reviewers WJwn, zdng, JkWZ, grX8 for their valuable feedback on our work. Below is a summary of responses to the reviewers.

### Reviewer WJwn

- **Baselines and Tasks**: We added a new baseline FlyingSquid, as well as a new dataset CosmosQA, for classification. The results are listed in a general response, demonstrating the effectiveness and generalization of our method.
- **Setting of Weight $\lambda$**: The specific value of $\lambda$ is informed by weak-to-strong performance, which is a direct way to reflect the quality of weak labels. No direct mapping relationship and grid search is done for selecting $\lambda$ weight.
- **Extension to Humans**: The scenarios and capabilities of weak models and humans are different, especially in evaluation capabilities. We designed some methods (for example, per-token probability estimation) to address this issue. When extending to humans, they can provide detailed, fine-grained evaluations across various aspects.

### Reviewer zdng

- **Datasets**: We chose the SciQ dataset for classification to follow Burn et al.'s work, and added experiments on another dataset CosmosQA. For the generation task, the SLURP dataset is chosen because of its clear evaluation metrics, making it easy to quantify the Weak-to-Strong performance.
- **Design choices for probability estimation**: Our motivation is that weak model performance is not satisfactory, and soft labels can provide more information than hard labels. Detailed explanation is given in response to the reviewer.

### Reviewer JkWZ

- **Motivation for Dirichlet**: The Dirichlet distribution is the conjugate prior distribution of the categorical distribution requested by both text classification and generation, which is tractable and avoids mathematical difficulties.
- **Training Stability**: Our strong model is not trained from scratch but is fine-tuned from a pre-trained model. Its inherent capability allows it to produce reliable confidence score estimates. We didn't encounter instability issues in our experiments.
- **Capability of Baselines**: For classification, we added FlyingSquid as another baseline. For generation, we introduced Joint Decoding, which is specifically designed for multiple weak model generating. The details can be found in response to the reviewer.

### Reviewer grX8

- **Model Size and Complementary**: We performed additional experiments with settings of the 1B weak + 13B strong models on SciQ dataset, as well as 3B weak + 13B strong models on both SciQ and CosmosQA datasets. The results show that our method is not tied to model size but to model performance. The results also give comprehensive insights into the complementary claim about how our method works.
- **Auxilary Loss**: Our Bayesian Multi-Weak method incorporates a distinct auxiliary loss design compared to baseline methods. The final approach combines both the Bayesian Multi-Weak framework and the auxiliary loss. The results without the auxiliary loss are presented solely for ablation studies.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper seeks to improve weak-to-strong generalization. This phenomenon, which has been known for quite a long time, has recently become popularized due to its theorized application to superalignment. In weak-to-strong generalization, a weak model produces outputs that a strong model is trained on. In certain settings, this strong model can perform better than the weak model did, despite only being trained on the weak model’s outputs. 

The authors note that one challenge is the choice of the weak model. They propose replacing a single model with a simple Bayesian ensembling approach and show how to use this for both predictive and generative tasks. The latter setting is challenging, since it requires ensembling output sequences from multiple generators. They also show some nice extensions for alignment. 

The reviewers generally liked the paper, with one caveat being the strength of the empirical results. While I share some of the sentiment, the paper has a bunch of good ideas and sufficient evidence of their value, so I voted for acceptance.