Weak-to-Strong Trustworthiness: Eliciting Trustworthiness with Weak Supervision

Authors: Martin Pawelczyk, Lillian Sun, Zhenting Qi, Aounon Kumar, Himabindu Lakkaraju
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Weak-to-strong learning, Fairness, Adversarial Robustness, OOD Robustness, Privacy

Abstract:
The rapid proliferation of generative AI, especially large language models (LLMs), has led to their integration into a variety of applications. 
A key phenomenon known as weak-to-strong generalization - where a strong model trained on a weak model's outputs surpasses the weak model in task performance - has gained significant attention.  Yet, whether critical trustworthiness properties such as robustness, fairness, and privacy can generalize similarly remains an open question. In this work, we study this question by examining if a stronger model can inherit trustworthiness properties when fine-tuned on a weaker model’s outputs, a process we term weak-to-strong trustworthiness generalization. Specifically, we examine whether a strong model can inherit or even enhance trustworthiness attributes when fine-tuned on a weak model's outputs. To address this, we introduce two foundational training strategies: 1) Weak Trustworthiness Finetuning (Weak TFT), which leverages trustworthiness regularization during the fine-tuning of the weak model, and 2) Weak and Weak-to-Strong Trustworthiness Finetuning (Weak+WTS TFT), which extends regularization to both weak and strong models. Our experimental evaluation on real-world datasets (Adult, OOD Style Transfer, AdvGLUE++, and Enron Emails) reveals that while some trustworthiness properties, such as fairness, adversarial, and OOD robustness, show significant improvement in transfer when both models were regularized, others like privacy do not exhibit signs of weak-to-strong trustworthiness. As the first study to explore trustworthiness generalization via weak-to-strong generalization, our work provides valuable insights into the potential and limitations of this method. Our findings highlight the importance of systematically studying trustworthiness transfer to develop AI systems that are not only accurate but also ethically aligned and reliable in critical applications.

Decision: reject
Reviewer scores: 6, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730386800873

thread_type:
official_review

<thread_object>
creator: reviewer_cuwc

<content>
summary:
This paper conducts an empirical study of weak-to-strong generalization for trustworthiness (fairness, OOD generalization, adversarial robustness and privacy). They compared three ways to perform weak-to-strong generalization, where the difference is whether the trustworthiness regularization is added to the loss of the weak/strong models. Experiments show the Weak+WTS TFT approach is the most effective way of transferring trustworthiness.

strengths:
- This work considers 4 different aspects of trustworthiness.
- It performs a relatively comprehensive study of parameter sensitivity.

weaknesses:
- The motivation of the weak-to-strong transfer in this work is questionable. It can be meaningful to study this problem in some special cases, such as when the labeled dataset is small while there exists a large unlabeled dataset. If the strong model is only trained with the small labeled data, it can suffer from underfitting, thus making it more meaningful to train the weak model first and train the strong model with weak labels generated by the weak model. However, this work does not consider such cases.
- This work is benchmarking three different ways to fine-tune the strong model, there is not much technical novelty.

questions:
1. In figure 2 (d), why does the Weak+WTS TFT perform better than the strong ceiling?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679220234

thread_type:
official_review

<thread_object>
creator: reviewer_qe1z

<content>
summary:
This work investigates this critical studies the transfer of trustworthiness properties from weak to strong models. This is an empirical work showing how some trustworthiness properties, such as fairness, adversarial, and OOD robustness, show significant improvement in transfer when both models were regularized towards trustworthiness, others like privacy do not exhibit signs of weak-to-strong trustworthiness.

strengths:
Overall, the problem is important and well-motivated, the paper is well-written, and the experimental section had an interesting conclusion, matching the claims in abstract in intro.

weaknesses:
This paper is an empirical evaluation extending a previously known result (weak-to-strong generalization) to a new set of capabilities, ie trustworthiness. Since there is no new methodology, I was hoping to see a stronger experimental section, containing a wider variety of models and datasets, ie >1 per task. I don't think the paper has any deep technical issues. In my opinion it's incomplete due to a few things, like only evaluating one definition of fairness/privacy/ood,. and using one dataset per setting. I understand using more models would be hard, due to accessibility, but if possible this would also improve the empirical evaluation.

questions:
- Can you expand on the experimental setting? E.g. hyperparam search. This should be contained at least in the appendix.
- How do you connect the rate of improvement of trustworthiness capabilities with the previous results in weak-to-strong generalization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720911807

thread_type:
official_review

<thread_object>
creator: reviewer_raj7

<content>
summary:
In this paper, the authors explore the challenge of whether the fairness, robustness, and privacy can be transferred from weak to strong models through weak-to-strong generation. This is an important research topic, because the ability to transfer trustworthiness properties is crucial for preventing harmful outcomes, complying with regulations, and maintaining public trust in AI technologies. In this paper, the authors propose two novel approaches to enhance the transfer of trustworthiness properties between weak and strong models.

strengths:
1. The authors propose two novel methods (Weak TFT and Weak+WTS TFT) to enhance trustworthiness transfer.

2. The paper is well organized and easy to follow. 

3. The authors provide a comprehensive study about the related works.

4. The authors propose three-phase experimental design provides a thorough analysis of trustworthiness transfer.

weaknesses:
1. The size of the strong models included in the paper is small. How about the results on the models around 7B?

2. Lack of generazation towards fairness. 
	In this paper, the authors adopt the Demographic Parity as the representative definition. In fairness research domain, there are a series of fairness definition, such as Equal False Positive/Negative Rates, Calibration/Predictive Parity, etc. How the proposed method work with different fairness definition.


3. Lack the in-depth discussion about why privacy do not exhibit signs of weak-to-strong trustworthiness.


Minor: 

1. Formatting issue: The subfigures in Figure 3 have varying sizes.

questions:
See weakness 1, 2, 3
</content>
</thread_object>

</thread>
<thread>
creation date:
1731080957518

thread_type:
official_review

<thread_object>
creator: reviewer_pmn1

<content>
summary:
In this paper the authors studied the abiilty for a weak model to teach a more powerful model to be fair, robust (both OOD and adversarial), and privacy sensitive.  The paper explores two appraoches: (1) traditional weak-to-strong generalization after training the weak model to have good trustworthiness and (2) additional trustworthiness regularization of the strong model. They find that traditional weak-to-strong generalization does not hold but that their second method works better.

strengths:
* While weak-to-strong generalization has been studied as a general property, it has not been sufficiently examined for safety and trustworthiness properties.

* The paper is fairly clear and thorough in its experiments, making it relatively easy to interpret.

weaknesses:
* Some of the tasks chosen I don't believe are a good fit for the study.  As an example, classification over discrete features (as in Adult) is an odd fit for testing LLMs (in contrast to something like BBQ) and demographic parity as an objective is particularly easy for weak models to learn such that teaching the concept to stronger models doesn't require much. I'd expect the weak-to-strong hypothesis is more interesting in tasks where it is hard for weak models and larger models we think can be more capable but giving them good supervision is hard.  I'd expect this is true in more nuanced definitions of fairness, stereotyping, and safety as an example.  (Similar statements could be said for newer ways of evaluating privacy and robustness to jailbreaks)

* There is some lack of clarity in the method description here so I could be wrong but it seems like the Weak+WTS TFT method is largely succeeding by adding the trustworthiness regularization directly and maybe not needing the weak model at all?  My current reading as a result is that the paper is largely presenting instead a negative result for WTS genrealiztaion on trustworthiness.

* There are some missing details that I think significantly shape the results, eg what is the actual loss for Weak+WTS TFT, what data is used for every method, and what is the vanilla form of using each type of data.

questions:
* Am i understanding correctly that in Weak+WTS TFT the strong model gets trustworthiness regularization + the normal weak-to-strong loss?  does the trustworthiness regularizer make use of the weak model?
* How is the Enron data used?
* Is all fine-tuning only on the trustworhtiness datasets? What is the vanilla form of each task that doesn't apply the trustworthiness loss?
* Is it a reasonable assumption that we have these curated datasets? It would seem that even having some of the datasets makes the WTS problem secondary (eg once I know demographic attributes and have decided on demogrpahic parity, or differential privacy)

nits:
* "modelss" (extra s)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732199548672

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We thank all the reviewers for their valuable feedback, which has significantly contributed to the refinement of our work. Inspired by their comments and suggestions, we are providing the following changes to our work:

- **Clarified Contributions**: Considering reviewers ``qE1Z``’s and ``CUwC``’s feedback, we have updated the abstract and introduction to state our contributions more clearly.

- **Added detail for all three training objectives**: Addressing Reviewer ``pMn1``’s feedback, we have updated Sections 3.1 and 3.2 and added a Section to Appendix A.1 to give more details on the training objectives.

- **Added overview of main conclusions from experiments in Table 1**:  In response to all reviewers, we have added Table 1 to the manuscript which provides a concise overview of the weak-to-strong trustworthiness trends observed in our experiments.   

- **Added additional Experiments**: Inspired by the constructive suggestions from Reviewers ``pMn1``, ``rAJ7`` and ``qE1Z``, we have conducted additional experiments to showcase the effect of:

   - **Larger Model Size (6.9B) on Weak-to-strong Trustworthiness**: In response to Reviewer ``rAJ7``, we ran additional experiments for fairness, adversarial robustness, and OOD robustness using the larger 6.9B Pythia LLM. The conclusions are qualitatively not different from the conclusions from the 410M and 1B large models.

   - **Additional Data sets**: Responding to Reviewer ``qE1Z``, we would like to clarify that the Adv. Robustness results are averaged across 6 datasets. Overall, the trends on each dataset are not different from the global trends we observe Adv. Robustness. 

   - **Additional Evaluation Measures**: In response to Reviewer ``pMn1``’s and ``rAJ7``’s suggestions, we have added additional experimental results for fairness evaluation (Equalized Odds) and privacy (Membership Inference Attacks).  Here, the trends on each evaluation measure is consistent with the trend we observed before. 

----
We would additionally like to highlight that our work makes several *impactful contributions*:
- **Novel Trustworthy Learning Paradigm**: This is the first work to investigate if trustworthiness properties can generalize from a weak to a strong model using weak-to-strong supervision, a process we term weak-to-strong trustworthiness generalization.

- **Foundational Training Strategies for Weak-to-Strong Trustworthiness Generalization**: We introduce two strong baseline training strategies, 
``Weak TFT`` and ``Weak+WTS TFT``, designed to facilitate weak-to-strong trustworthiness generalization.

- **Weak-to-Strong Trustworthiness Generalization is Feasible**: Our experiments show that some trustworthiness properties can indeed be generalized and even enhanced from weak to strong models.

- **Comprehensive Evaluation**: We perform extensive experiments across:
   - *Models*: Ranging from weak (14M, 70M, 160M) to strong (410M, 1B, 7B) sizes.
   - *Tasks*: Assessing fairness, privacy, adversarial robustness, and out-of-distribution (OOD) robustness.
   - *Datasets & Metrics*: Spanning 9 datasets, the two most popular privacy attack metrics (Membership Inference, Extraction), two most popular fairness criteria (Equalized Odds, Demographic Parity), and standard robustness metrics (Adv & OOD robustness)
   - *Thorough Hyperparameter Optimization*: We conduct exhaustive and computationally expensive hyperparameter searches for all relevant parameters in the objective functions to ensure robust and fair comparisons.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper explores the transfer of trustworthiness properties—fairness, robustness (OOD and adversarial), and privacy—from weak models to stronger ones via weak-to-strong generalization. While the study is well-organized, addressing an important topic with comprehensive experiments, its contributions are limited. The experimental design focuses on a single fairness definition (Demographic Parity) and a narrow range of datasets and model scales, limiting generalizability. Additionally, the results suggest that trustworthiness regularization alone, rather than weak-to-strong generalization, drives the improvements, undermining the paper's central premise. The lack of discussion on the failure of privacy transfer, combined with missing details on the experimental methods (e.g., loss functions, data usage), further weakens the work. Overall, while the topic is relevant, the lack of technical novelty, limited scope, and incomplete exploration make this paper insufficient for acceptance.