Debate Helps Weak-to-Strong Generalization
Authors: name: Hao Lang; name: Fei Huang; name: Yongbin Li
Published: 2025-01-21 05:36:13

Abstract: Common methods for aligning already-capable models with desired behavior rely
on the ability of humans to provide supervision. However, future superhuman
models will surpass the capability of humans. Therefore, humans will only be
able to weakly supervise superhuman models. This expected deficiency of human
evaluation would weaken the safety of future AI systems. Scalable oversight and
weak-to-strong generalization are two complementary approaches to tackle this
issue. In this paper, we attempt to combine the strengths of these two
approaches to further improve alignment. Specifically, we investigate ways of
improving human supervision with a strong pretrained model and then supervise
the strong model with enhanced weak human supervision. To make iterative
empirical progress, we consider an analogy: can we use a strong model to
improve weak model supervision and then use it to supervise the strong model?
We empirically test it by finetuning a small weak model on ground truth labels
with the additional help from a large strong model, and then finetuning the
strong model on labels generated by the weak model. We find that debate can
assist a weak model in extracting trustworthy information from an untrustworthy
strong model, which provides leverage as context on samples when training a
weak model. We also show that an ensemble of weak models helps exploit long
arguments generated by strong model debaters and obtain a more robust
supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP
benchmarks show that the combination approach leads to better alignment, which
indicates that debate has the potential to help weak-to-strong generalization.