Beyond Human Data: Scaling Self-Training for Problem-Solving with
  Language Models
Authors: name: Avi Singh; name: John D. Co-Reyes; name: Rishabh Agarwal; name: Ankesh Anand; name: Piyush Patil; name: Peter J. Liu; name: James Harrison; name: Jaehoon Lee; name: Kelvin Xu; name: Aaron Parisi; name: Abhishek Kumar; name: Alex Alemi; name: Alex Rizkowsky; name: Azade Nova; name: Ben Adlam; name: Bernd Bohnet; name: Hanie Sedghi; name: Igor Mordatch; name: Isabelle Simpson; name: Izzeddin Gur; name: Jasper Snoek; name: Jeffrey Pennington; name: Jiri Hron; name: Kathleen Kenealy; name: Kevin Swersky; name: Kshiteej Mahajan; name: Laura Culp; name: Lechao Xiao; name: Maxwell L. Bileschi; name: Noah Constant; name: Roman Novak; name: Rosanne Liu; name: Tris Warkentin; name: Yundi Qian; name: Ethan Dyer; name: Behnam Neyshabur; name: Jascha Sohl-Dickstein; name: Noah Fiedel
Published: 2023-12-11 18:17:43

Abstract: Fine-tuning language models~(LMs) on human-generated data remains a prevalent
practice. However, the performance of such models is often limited by the
quantity and diversity of high-quality human data. In this paper, we explore
whether we can go beyond human data on tasks where we have access to scalar
feedback, for example, on math problems where one can verify correctness. To do
so, we investigate a simple self-training method based on
expectation-maximization, which we call ReST^{EM}, where we (1) generate
samples from the model and filter them using binary feedback, (2) fine-tune the
model on these samples, and (3) repeat this process a few times. Testing on
advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find
that ReST^{EM} scales favorably with model size and significantly surpasses
fine-tuning only on human data. Overall, our findings suggest self-training
with feedback can substantially reduce dependence on human-generated data.