LIMA: Less Is More for Alignment
Authors: name: Chunting Zhou; name: Pengfei Liu; name: Puxin Xu; name: Srini Iyer; name: Jiao Sun; name: Yuning Mao; name: Xuezhe Ma; name: Avia Efrat; name: Ping Yu; name: Lili Yu; name: Susan Zhang; name: Gargi Ghosh; name: Mike Lewis; name: Luke Zettlemoyer; name: Omer Levy
Published: 2023-05-18 17:45:22

Abstract: Large language models are trained in two stages: (1) unsupervised pretraining
from raw text, to learn general-purpose representations, and (2) large scale
instruction tuning and reinforcement learning, to better align to end tasks and
user preferences. We measure the relative importance of these two stages by
training LIMA, a 65B parameter LLaMa language model fine-tuned with the
standard supervised loss on only 1,000 carefully curated prompts and responses,
without any reinforcement learning or human preference modeling. LIMA
demonstrates remarkably strong performance, learning to follow specific
response formats from only a handful of examples in the training data,
including complex queries that range from planning trip itineraries to
speculating about alternate history. Moreover, the model tends to generalize
well to unseen tasks that did not appear in the training data. In a controlled
human study, responses from LIMA are either equivalent or strictly preferred to
GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard
and 65% versus DaVinci003, which was trained with human feedback. Taken
together, these results strongly suggest that almost all knowledge in large
language models is learned during pretraining, and only limited instruction
tuning data is necessary to teach models to produce high quality output.