Self-Alignment with Instruction Backtranslation
Authors: name: Xian Li; name: Ping Yu; name: Chunting Zhou; name: Timo Schick; name: Luke Zettlemoyer; name: Omer Levy; name: Jason Weston; name: Mike Lewis
Published: 2023-08-11 17:47:54

Abstract: We present a scalable method to build a high quality instruction following
language model by automatically labelling human-written text with corresponding
instructions. Our approach, named instruction backtranslation, starts with a
language model finetuned on a small amount of seed data, and a given web
corpus. The seed model is used to construct training examples by generating
instruction prompts for web documents (self-augmentation), and then selecting
high quality examples from among these candidates (self-curation). This data is
then used to finetune a stronger model. Finetuning LLaMa on two iterations of
our approach yields a model that outperforms all other LLaMa-based models on
the Alpaca leaderboard not relying on distillation data, demonstrating highly
effective self-alignment.