Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak
  Supervision
Authors: name: Collin Burns; name: Pavel Izmailov; name: Jan Hendrik Kirchner; name: Bowen Baker; name: Leo Gao; name: Leopold Aschenbrenner; name: Yining Chen; name: Adrien Ecoffet; name: Manas Joglekar; name: Jan Leike; name: Ilya Sutskever; name: Jeff Wu
Published: 2023-12-14 23:07:33

Abstract: Widely used alignment techniques, such as reinforcement learning from human
feedback (RLHF), rely on the ability of humans to supervise model behavior -
for example, to evaluate whether a model faithfully followed instructions or
generated safe outputs. However, future superhuman models will behave in
complex ways too difficult for humans to reliably evaluate; humans will only be
able to weakly supervise superhuman models. We study an analogy to this
problem: can weak model supervision elicit the full capabilities of a much
stronger model? We test this using a range of pretrained language models in the
GPT-4 family on natural language processing (NLP), chess, and reward modeling
tasks. We find that when we naively finetune strong pretrained models on labels
generated by a weak model, they consistently perform better than their weak
supervisors, a phenomenon we call weak-to-strong generalization. However, we
are still far from recovering the full capabilities of strong models with naive
finetuning alone, suggesting that techniques like RLHF may scale poorly to
superhuman models without further work. We find that simple methods can often
significantly improve weak-to-strong generalization: for example, when
finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence
loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our
results suggest that it is feasible to make empirical progress today on a
fundamental challenge of aligning superhuman models.