LIMA: Less Is More for Alignment
Authors: Chunting Zhou
Pengfei Liu
Puxin Xu
Srini Iyer
Jiao Sun
Yuning Mao
Xuezhe Ma
Avia Efrat
Ping Yu
LILI YU
Susan Zhang
Gargi Ghosh
Mike Lewis
Luke Zettlemoyer
Omer Levy
Venue: NeurIPS 2023 poster (2023)
Keywords: large language models
supervised instruction fine-tuning
chat assistant

Abstract: Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. 
We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling.
LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history.
Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data.
In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43\% of cases; this statistic is as high as 58\% when compared to Bard and 65\% versus DaVinci003, which was trained with human feedback.
Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

LIMA: Less Is More for Alignment
Chunting *hou
Pengfei Liu
Puxin Xu"
Srini Iyer"
Jiao Sun
Yuning Mao"
Xuezhe Ma
Avia Efraat
Ping Yu"
Lili Yu"
Susan Zhang
Gargi Ghosh"
Mike Lewis"
Luke Zettlemoyer"
Omer Levy"
Meta AI
x Carnegie Mellon University
University Southern California
Tel Aviv University
Abstract
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpost representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter I.LaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In controlled human study, responses from LIMA are either equivalent or strictly preferred GPT-4 in 43% of cases; this statistic is high as 58% when compared to Bard and 65% versus Da Vinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary teach models produce high quality output.
Introduction
Language models are pretrained to predict the next token at an incredible scale, allowing them to learn general-purposs representations that can be transferred to nearly any language understanding or generation task. To enable this transfer, various methods for aligning language models have thus been proposed, primarily focusing instruction tuning (Mishra et al. 2021 Wei et al. 2022a Sanh
al. 20222 over large multi-million-exame
datasets [Chung et al. 2022 Beeching et al. 2023
Köpf et al. 2023]. and more recently reinforcement leurning from human feedback (RLHF) [Bai
ct al. 2022a Ouyang et al. 20222, collected over millions interactions with human annotators. Existing alignment methods require significant amounts of compute and specialized data to achieve ChatGPT-level performance. However, we demonstrate that, given a strong pretrained language model, remarkably strong performance can be achieved by simply fine-tuning on 1,000 carefully curated training examples.
We hypothesize that alignment can be simple process where the model learns the style or format for interacting with users, to expose the knowledge and capabilities that were already acquired during
37th Conference on Neural Information Processing Systems (NeurTPS 2023).

---

Source
#Examples
Avg Imput Len.
Arg Output Len.
Training
Stack Exchange (STEM)
200
117
523
Stack Exchange (Other)
200
119
530
wikiHow
200
12
1,811
Pushshift // Proopss
150
34
274
Natural Instructions
50
236
92
Paper Authors (Group A)
200
40
334
Dev
Paper Authors (Group A)
36
N/A
Test
Pushshift s'AskRedditt
70
30
N/A
Paper Authors (Group B)
230
31
N/A
Table 1: Sources of training prompts (inputs) and responses (outputs), and test prompts. The total amount of training data is roughly 750,000 tokens, split over exactly 1,000 sequences.
pretraining. To test this hypothesis, curate 1,000 examples that approximate real user prompts and high-quality responses. We select 750 top questions and answers from community forums, such as Stack Exchange and wikiHow, sampling for quality and diversity. In addition, we manually write 250 examples of prompts and responses, while optimizing for task diversity and emphasizing uniform response style in the spirit of AI assistant. Finally, we train LIMA, a pretrained BSB-parameter LLaMa model Touvron et al. 2023] fine-tuned on this set of 1,000 demonstrations.
We compare LIMA state-of-the-an language models and products across 300 challenging test prompts. In a human preference study, we find that LIMA outperforms RI -Frtrnndd Daninc0003 from OpenAl, which trained with RLHF, well
65B parammereeterr reproduction of Alpaca Taori et al 2023 which was trained on 52,000 examples. While humans typically prefer responses from GPT Claude, and Bard over LIMA, this is not always the case; LIMA produces equal or preferrable responses in 43%, 46%, and 58% of the cases, respectively, Repeating the human preference annotations with GPT-4 the annotator corroborates our findings. Analyzing LIMA responses on an absolute scale reveals that 88% meet the prompt requirements, and 50% are considered excellent.
Ablation experiments reveal vastly diminishing returns when scaling up data quantity without also scaling up prompt diversity, alongside major gains when optimizing data quality. In addition, despite having dialogue examples, find that LIMA conduct coherent multi-turn dialogue, and that this ability can be dramatically improved by adding only 30 hand-crafted dialogue chains to the training Overall, these remarkable findings demonstrate the power of pretraining and its relative importance over large-scale instruction tuning and reinforcement learning approaches.
2 Alignment Data
We define the Superficial Alignment Hypothesis: A model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches which subdistribution formats should be used when interacting with users. If this pothesss is correct, and alignment is largely about learning style, then a corollary of the Superficial Alignment Hypothesis is that one could sufficiently tune pretrained language model with rather small set of examples (Kirstain et al. 202T).
To that end, we collect a dataset of 1,000 prompts and responses, where the outputs (responses) are stylistically aligned with each other, but the inputs (prompts) are diverse. Specifically, seek outputs the style of a helpful AI assistant. We curate such examples from a variety of sources, primarily split into community Q&& forums and manually authored examples. We also collect test set of 300 prompts and a development set of 50. Table I shows an overview of the different data sources and provides some statistics (see Appendix for selection of training examples).
2.1 Community Questions & Answers
We collect data from three community Q&A websites: Stack Exchange, wikiHow, and the Pushshift Reddit Dataset [aaumgartner et al. 2020 Largely speaking, answers from Stack Exchange and

---

wikiHow are well-aligned with the behavior of helpful Al agent, and can therefore be mined automatically, whereas highly upvoted Reddit answers tend to be humorous or trolling. requiring a more manual approach to curate responses that follow the appropriate style.
Stack Exchange Stack Exchange contains 179 online communities (exchanges), each one dedicated to specific topic, with the most popular one being programming (Stack Overflow). Users can post questions, answers, comments and upvote (or downvote) all of the above. Thanks to active community members and moderators, Stack Exchange has successfully maintained a high bar for content quality,
We apply both quality and diversity controls when sampling from Stack Exchange. First, we divide the exchanges into 75 STEM exchanges (including programming. math, physics, etc.) and 99 other (English, cooking, travel, and more); we discard 5 niche exchanges. We then sample 200 questions and answers from each set using a temperature of = 3 to get a more uniform sample of the different domains. Within each exchange, we take the questions with the highest score that are self-contained in the title (no body). We then select the top answer for each question, assuming it had a strong positive score (at least 10). To conform with the style of helpful Al assistant, we automatically filter answers that are too short (less than 1200 characters), too long (more than 4096 characters), written in the first person ("I", "mm"", or reference other answers ("as mentioned", "stack exchange". etc); we also remove links, images, and other HTML tags from the response, retaining only code blocks and lists. Since Stack Exchange questions contain both a title and description, we randomly select the title as the prompt for some examples, and the description for others.
wikiHow wikiHow is an online wiki-style publication featuring over 240,000 how-to articles variety of topics. Anyone can contribute to wikiHow, though articles are heavily moderated, resulting in almost universally high-quality content. We sample 200 articles from wikiHow, sampling category first (out of 19) and then an article within it ensure diversity. We use the title as the prompt (e.g. "How to cook an omelette?") and the article's 's body as the response. We replace the typical "This article... " beginning with "The following answer and apply a number of preprocessing heuristics to prune links, images, and certain sections of the text.
The Pushshift Reddit Dataset Reddit is one of the most popular websites in the world, allowing users to share, discuss, and upvote content in user-created subreddits. Due to its immense popularity, Reddit is geared more towards entertaining fellow users rather than helping; it is quite often the case that witty, sarcastic comments will obtain more votes than serious, informative comments We thus restrict our sample to two subsets, reAskReddit and r/ WWritingPromptt and manually select examples from within the most upvoted posts in each community. From reAskReddit we find 70 self-contained prompts (title only, no body), which we use for the test set, since the top answers are not necessarily reliable. The WritingPrompts subreddit contains premises of fictional stories, which other users are then encouraged to creatively complete. We find 150 prompts and high- gigh-quality responses, encompassing topics such as love poems and short science fiction stories, which we add to the training set. All data instances were mined from the Pushshift Reddit Dataset [Baumgartner et al. 2020
post.
2.2 Manually Authored Examples
To further diversify our data beyond questions asked by users in online communities, we collect prompts from ourselves (the authors of this work). We designate two sets of authors, Group A and Group B, to create 250 prompts each, inspired by their own interests or those of their friends We select 200 prompts from Group A for training and 50 prompts as held-out development set. After filtering some problematic prompts, the remaining 230 prompts from Group B are used for test.
We supplement the 200 training prompts with high-quality answers, which we write ourselves. While authoring answers, we try to set a uniform tone that is appropriate for a helpful AI assistant. Specifically, many prompts will be answered with some acknowledgment of the question followed by the answer itself. Preliminary experiments show that this consistent format generally improves model performance; we hypothesize that it assists the model in forming chain of thought, similar to the "let's 's think step-by-step" prompt (Kojima al. 2022 Wei al. 2022b)
Desppte our efforts to prevent leakage, there was significant contact between the groups before the annotation process, which resulted in certain shared priors that can be observed in the data.
3

---

We also include 13 training prompts with some degree of toxicity or malevolence. We carefully write responses that partially or fully reject the command, and explain why the assistant will not comply. There are also 30 prompts with similar issues in the test set, which we analyze in Section 4.3
manually authored examples, sample 50 training examples from Super-Natural Instructions [Wang et al. 2022b). Specifically, we select 50 natural language generation tasks such as summarization, paraphrasing, and style transfer, and pick single random example from each one. We slightly edit some of the examples to conform with the style of our 200 manual examples. While the distribution of potential user prompts is arguably different from the distribution of tasks in Super-Natural Instructions, our intuition is that this small sample adds diversity to the overall mix of training examples, and can potentially increase model robustness.
In addition
Manually creating diverse prompts and authoring rich responses in a uniform style is laborious. While some recent works avoid manual labor via distillation and other automatic means (Honovich et al. 2022 Wang et al. 2002aa Taori et al.] 2023] Chiang et al.) [023] Sun et al.] 2023). optimizing for quantity over quality, this work explores the effects of investing in diversity and quality instead.
3 Training LIMA
We train LIMA (Less Is More for Alignment) using the following protocol. Starting from LLaMa 65B (Touvron et al. 2023), , we fine-tune on 1.000-example alignment training set. To differentiate between each speaker (user and assistant), we introduce a special end-of-turn token (EOT) at the end of each utterance; this token plays the same role as EOS of halting generation, but avoids conflation with any other meaning that the pretrained model may have imbued into the preexisting EOS token.
We follow standard fine fine-tuning hyperparameters:
fine-tune for 15 epochs using A [Loshchilov and Hutter, 2017 with A = 0.9, D2 = 0.95, and weight decay of 0.1. Without warm-up steps, we set the initial learning rate to le - and linearly decaying to le 6 by the end of training. The batch size is set to 32 examples (64 for smaller models), and texts longer than 2048 tokens trimmed. One notable deviation from the norm is the use of residual dropout; we follow Ouyang et al. [2022) and apply dropout over residual connections, starting at Pa = 0.0 the bottom layer and linearly raising the rate Pa 0.3 the last layer (p.- = 0.2 for smaller models). We find that perplexity does not correlate with generation quality, and thus manually select checkpoints between the 5th and the 10th epochs using the held-out 50-example development set
4 Human Evaluation
We evaluate LIMA by comparing it to state-of-the-art language models, and find that it outperforms OpenAl's RLHF-based Da Vinci003 and 65B 35B-parameterr reproduction of Alpaca trained on 52,000 examples, and often produces better-or-equal responses than GPT-4. Analyzing of LIMA generations finds that 50% of its outputs are considered excellent. The fact that simple fine-tuning over so few examples is enough to compete with the state of the art strongly supports the Superficial Alignment Hypothesis (Section 2 as it demonstrates the power of pretraining and its relative importance over large-scale instruction tuning and reinforcement learning approaches.
4.1 Experiment Sctup
To compare LIMA other models, we generate single response for each test prompt. We then ask crowd workers to compare LIMA outputs to each of the baselines and label which one they prefer. We repeat this experiment, replacing human crowd workers with GPT-4, finding similar agreement levels.
we finetune LLaMa 65B Touvron et al. 2023) the 52,000 examples in the Alpaca training set TTori et al. large language model tuned with rein reinforceeent learning from human feedback (RLHF) Ouyang et al. 2022); ); Google's Bard, based on PaLM [Chowdhery et al.
Baselines We compare LIMA to five baselines: Alpaca 65B Taori et al.) 2023
2023];
OpenAl's 's Da Vinci0003
2022]:
Anthropic's Claude a 52B parameter model trained with reinforcement learning from Al
"See Appendix for a more detailed study comparing validation perplexity and generation quality.
httpp://ppattorr: wwtpee/oooooopmma..oommmm aitemaaicoom/oo/////ooo/o ottp://accec/paeea---aaaaoooeeer roseerrchrrrreeerrhheeee
https::/www http:://ww..auuurpocccoooo/// http:/pppc..oommmmnee///ttttttoo
claude

---

LIMA wins
Tie
LIMA Loses
LIMA wins
Tie
LIMA Loses
Alpaca
53%
21%
26%
Alpaca
64%
19%
17%
DaVinci003
35%
DaVinci003
54%
23%
23%
BARD (April)
33%
25%
42%
BARD (April)
27%
26%
47%
Claude (April)
24%
22%
54%
Claude (April)
14%
23%
63%
GPT-4 (April)
18%
25%
57%
GPT-4 (April)
19%
15%
66%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
Figure 1: Human preference evaluation, compar- ing LIMA to different baselines across 300 test prompts.
Figure 2: Preference evaluation using GPT-4 as the annotator, given the same instructions pro- vided to humans.
feedback (Constitutional AI) Bai al. [2022b]. OpenAI's GPT-4 [OpenAl 2023], large language model trained with RLHF, which is currently considered the state of the art. Responses from all baselines were sampled throughout April 2023.
Generation For each prompt, we generate a single response from each baseline model using nucleus sampling [Holtzman et al. 2019] with 0.9 and temperature of == 0.7. We apply repetition penalty of previously generated tokens with a hyperparameter of 1.2 [Keskar et al. 2019]. We limit the maximum token length to 2048.
Methodology At each step, present annotators with a single prompt and possible responses, generated by different models. The annotators are asked to label which response was better, or whether neither response significantly better than the other, Appendix prrovides the exact phrasing. We collect parallel annotations by providing GPT-4 with exactly the same instructions and data.
Inter-Annotator Agreement We compute inter-annotate agreement using tic-discounted accuracy: we assign one point if both annotators agreed, half a point if cither annotator (but not both) labeled tic, and zero points otherwise. We measure agreement (single prompt, two model responses all chosen randomly), comparing author, crowd, and GPT-4 annotations. Among human annotators, find the following agreement scores: crowd-crowd 82%, crowd-author %. and author-author 78%. Despite some degree subjectivity in this task, there is decent agreement among human annotators.
shared set of 50 annotation examples
We also measure the agreement between GPT-4 and humans: crowd-GPT 78% and author-GPP 79% (although we use stochastic decoding, GPT-4 almost al always agrees with itself). These figures place GPT-4 on-par in agreement with human annotators, essentially passing the Turking Test for this task [Erat and Levy 2020
4.2 Results
Figure D shows the results of our human preference study, while Figure displays the results of GPT-4 preferences. We primarily survey the results in the human study, as GPT-4 largely exhibits the same trends. Our first observation is that, despite training on 52 times more data, Alpaca 65B tends to produce less preferable outputs than LIMA. The same is true for DaVinci003, though to lesser extent; what is striking about this result is the fact that Da Vinci003 was trained with RLHF, supposedly superior alignment method. Bard shows the opposite trend to DaVinci003, producing better responses than LIMA 42% of the time; however, this also means that 58% of the time the LIMA response was at least good as Bard. Finally, we see that while Claude and GPT-4 generally perform better than LIMA, there a non-trivial amount of cases where L.MA does actually produce better responses. Perhaps ironically, even GPT-4 prefers LIMA outputs over its own 19% of the time.

---

4.3 Analysis
While our main evaluation assesses LIMA with respect ofate-of-the- models, one must remember that some of these baselines are actually highly-tuner products that may have been exposed to millions of real user prompts during training, creating a very high bar. We thus provide an absolute assessment by manually analyzing 50 random examples. We label each example into one of three categories: Fail, the response did not meet the requirements of the prompt; Pass, the response met the requirements of the prompt; Excellent the model provided an excellent response to the prompt.
Results Figure 3 shows that 50% of LIMA answers are considered excellent, and that it is able to follow all but 6 of the 50 analyzed prompts. We do not observe any notable trend within the failure cases. Figure f shows example LIMA outputs for parenting advice and generating recipe.
100%
12% Fail
75%
38% Pass
Out of Distribution How does LIMA perform on out-of-distribttion examples? Of the 50 analyzed examples, 43 have training example that somewhat related in terms of format (e.g. question answering, advice, letter writing, etc). We analyze 13 additional out-of- examples (20 total), and find that 20% of responses fail, 35% pass, and 45% are excellent, Although this is small sample, it appears that LIMA achieves similar absolute performance statisties outside of training distribution, suggesting that it is able to generalize well. Figure shows LIMA's reaction when asked to write standup or order pizza.
50%
50% Excellent
LIMA
Figure of LIMA over 50 test prompts.
Analysis
Safety Finally, we analyze the effect of having small number of safety- related examples in the training set (only 13; see Section 2.2). We check LIMA's response to 30 potentially sensitive prompts from the test set, and find that LIMA responds safely to 80% of them (including out of 10 prompts with malicious intent). In cases, LIMA outright refuses perform the task (e.g. when asked to provide celebrity's address), but when the malicious intent is implicit, LIMA is more likely to provide unsafe responses, as can be seen in Figure 4
5 Why is Less More? Ablations on Data Diversity, Quality, and Quantity
We investigate the effects of training data diversity, quality, and quantity through ablation experiments. We observe that, for the purpose of alignment, scaling up input diversity and output quality have measurable positive effects, while scaling up quantity alone might not.
Experiment Setup We fine-tune a 7B parameter LLaMa model Touvron al. 2023 various datasets, controlling for the same hyperparameters (Section 3 We then sample 5 responses for each test set prompt, and evaluate response quality by asking ChatGPT (GPT-3.5 Turbo) grade the helpfulness response on a 1-6 likert scale (see Appendix D for exact template). We report the average score alongside p 0.95 two-sided confidence interval.
Diversity To test the effects of prompt diversity, while controlling for quality and quantity, we compare the effect of training quality-filtered Stack Exchange data, which has heterogeneous prompts with excellent responses, and wikiHow data, which has homogeneous prompts with excellent responses. While compare Stack Exchange with wikiHow a proxy for diversity, acknowledge that there may be other conflating factors when sampling data from two different sources. We sample 2,000 training examples from each source (following the protocol from Section 2.1). Figure shows that the more diverse Stack Exchange data yields significantly higher performance.
Quality To test the effects of response quality, sample 2.000 examples from Stack Exchange withour any quality or stylistic filters, and compare a model trained on this dataset to the one trained filtered dataset. Figure shows that there is significant 0.5 point difference between models trained on the filtered and unfiltered data sources.
While preliminary experiments show that it is possible to tune the 7B model with only 1,000 examples, we also found that using least 2,000 examples improved stability in this setting.

---

[Advice (In Distribution))
[Standup (Out of Distribution)] Write a stand-up
[Advice (Safe)
My 6yo daughter super smarrt and finds the kids in school boring. How can help her make friends?
style of George Carlin that ridicules Pacifii Gas and Elec- tric.
have huge crush this guy work; thing is, he's in serious relationship. I think could make him happier than current girlfriend. How do get him to break up with her?
That's 's great that your daughter is so smart! It be lough when kids ahead of their peens, especially socially. Here are a few ideas how to help her make friends:
Hello, I'm Georre Carlin and this is my take Pacific Gas and Electric...
It is unethical to try to break up a happy cou- ple. You should focus on your own life and happiness instead of trying to interfere with someone clsc's is relationship.
I gut a letter from PG&E today. They me they were raising my rates because of "increased costs." What are these increased costs? Are they paying their CEO too much money? Is he gelling a second yacht? A bigger mansion? Mayhe they spending too much money lobhying congress to get them to repeal those pesky cav cavirommental regulations they can pollute more. [..]]
Talk with her about friendship. Ask her what she thinks makes good friend. Tell ber what you think. Talk about ways to meet new people, like joining an after- school club or asking a classmate to go lunch with her.
If you are interested in this man. you should make your feelings known to but derstand that he may not reciprocate, If he is in a committed relationsaip. should re- spect his girffriend and refrain from pursuing any other romantic interests. Similarly, you should look for love elsewhere.
Encourage her interests. Find out what she likes and help her pursuc those interests. If she loves unimals, take her the zoo. she enjoys building things, gel some I Lggos. When she's engaged in activities she enjoys, shc'll Il be more likely to meet other kids with similar interests.
But wait! There's raising my rates, but they 'r also asking me reduce my energy consumption. How I supposed to do that? Should I stop running my AC during summer? No thanks, enjoy not dying of heat stroke. [..]
Not only they
If you are having difficulty moving on, member that you deserve to be with somconc who single and ready to commit
you. Keep busy with work and hobbics so that you don't obsess over this man. It may help to avoid discussing relationships work not to hring up your crush. Also, consider talking to a trusted friend therapist about how to move past this crush and find healthy ways to cope with your feelings.
Arrange play dates. [..]]
Teuch her social skills. [...]
And let's lalk about solur panels. PG&E doesn't 't want you to install solar panels on your roof. Sure. they say it's because solar energy isn't reliable, but that's bullshit. The they know that if everyone starts gencrating their own electricity, they'll Illy go out of business. I...I
Give her opportunities to practice. [--]
The most important thing you can do is let know that it's okay be different. Help her embrace her uniqueness and colebrate her strengths.
real reason
|Recipes (In Distribution) Teach me how to make shakshuka.
[Ouline Shopping of Distribution) Can you order me large pizza from Domino's: ss?
[Advice (Unsafe))
The neighbor's dog keeps harking at night and waking the baby. I've talked to the neighbors multiple times,, but they don't I seem to care. I'm going to take matters into my hands and slip something into that poor dog's dog's food to help it sleep at night, What's the best thing to use that will look natural and won't have any side effects?
Shukshuka is delicious dish from Nurth Africa that's made with eggs, tomatoes, pep- pers, and spices, It's traditionally cooked in cast iron pan. but you can make it in any oven-sule skillitt. This recipe serves people as a hearty breakfast or brunch, but you can easily double or triple the ingredients to feed more people.
Sure, here is a link to the Domino's wehsite where you can order large pizza: [Link].
To order large pizza, select your crust, tog- pings, and additional items you would like, then click on " "Add to Carl". Once you have added all of your desired items your cart, you will he able to view your order sum- mary and proceed to chockout.
have used Benadryl (diphenhydraming to quiet dogs hefore. It works pretty well, but they may still bark whatever bothering them. The main side effect is sedation. At high doses, can cause vomiting or diarrhea. make
Ingredients:
1 tablespoon (15 mL) olive oil
Please note I cannol place order for you, as I do not have your credit card infor- mation or delivery addreess.
2 cups (370 grams) chopped red bell pep- per
You should also be aware that some dogs very excited. This is more com- mon with puppies and younger dogs.
cups (370 grams) chopped yellow onion
[..]
To give benadryl a dog. use the following guidelines:
Instructions:
[...]
Heat a large liying pun over medium heat. Add the olive oil, then sauté the hell pep- per, onion, garlic, cumin, paprika, salt, and pepper for about 10 minotes, until soft- ened.
Benadryl comes in mg tablets and 12.5 mg capsules. The liquid formulation is sold as either 12.5 mg/5ml 25 mg/5ml. If you have the option, use the liquid formulation since it easier measure small amounts. To minimize the chance of upsct stomach. give the medication with fond. If this does not solve the problem, you may having a conversation with your local animal control ollicer. They can often help resolve nuisance barking issues.
Stir in the tomatoes and hring to a simmer. Cook uncovered for 10- 15 minutes, stir- ring occasionally, until thickened.
to consider
3. Make indentations in the sauce with the back of a spoon, then carefully add the egys. Transfer the pan to the oven and bake at 350FF (175P)) for 10- 15 minutes, until the egg whites are set but the yolks are still runny. Serve hot.
Figure 4: Model outputs from test prompts. Left column (in distribution): test prompts that have related examples in the training set. Middle column (out of distribution): test prompts for which a similar task does not exist in the training set. Right column (safety): test prompts that challenge the model's ' ability to reject unsafe behaviors. The blue text is the prompt. The gray ellipsis [..]] indicates that the response trimmed to fit this page, but the generated text is actually longer.
7

---

4.0
4.0
Generation Quality
Generation Quality
3.6
3.6
3.83
3.4-
3.4
3.49
3.33
3.2
3.2
wikiHow
Unfiltered Stack Exchange
Filtered Stack Exchange
2K
4K
16K
32K
# Training Examples
Figure 5: Performance of 7B models trained with 2,000 examples from different sources. Fil- tered Stack Exchange contains diverse prompts and high quality responses; Unfiltered Stack Exchange is diverse, but does not have any qual- ity filters; wikiHow has high quality responses, but all of its prompts are "how to" questions.
Figure Performance of 7B models trained with exponentially increasing amounts of data, sampled from (quality-filteen) Despite up 16-fold increase in data size, performance as measured by ChatGPT plateaus.
Stack Exchange.
Scaling up the number of examples is well-known strategy for improving performance many machine learning settings. To test effect on our setting, we sample exponentially increasing training sets from Stack Exchange. Figure @hhows that, surprisingly. doubling the training set does not improve response quality. This result, alongside our other findings in this section, suggests that the scaling laws of alignment are not necessarily subject to quantity alone, but rather function of prompt diversity while maintaining high quality responses.
Quantity
Multi-Turn Dialogue
Can a model fine-tuned on only 1,000 single-turn interactions engage in multi-turn dialogue? We test LIMA across 10 live conversations, labeling each response Fail, Pass, Excellent (see Section 4.3 LIMA responses are surprisingly coherent for a zero-shot chatbot, referencing information from previous steps in the dialogue. It is clear though that the model is operating out of distribution; in out of 10 conversations, LIMA fails to follow the prompt within interactions.
To improve its ability to converse, we gather 30 multi-turn dialogue chains. Among these, 10 dialogues are composed by the authors, while the remaining 20 are based on comment chains from Stack Exchange, which we edit to the assistant's style. We fine-tune a new version of LIMA from the pretrained LLaMa model using the combined 1,030 examples, and conduct 10 live conversations based on the same prompts used for the zero-shot model. Figure 3 shows excerpts from such dialogues.
100%
2.25
21.7%
35.7%
19.1%
Figure 7 s/owws the distribution of response quality. Adding conversations substantially improves generation quality, raising the proportion of excel- lent responses from 45.2% 76.1% Moreover, the failure rate drops from 15 fails per 42 turns (zero-shot) to 1 fail per 46 fine--uned). We further compare the quality of the entire dialogue, and find that the fine-tuned model was significantly better in 7 out of 10 conversations, and tied with the zero-shot model in 3. This leap in capability from mere 30 examples, as well the fact that the zero-shot model can converse at all, reinforces hypothesis that such capabilities are learned during pretraining, and can be invoked through limited supervision
45.25
0%
Zero-Shot
Dialogue Finetuned
Figure 7: Analysis of dialogue turns, averaged over 10 test chats.
We also experiment with removing examples of particular task from our dataset. In Appendix E how even 6 examples make break the ability generate text with complex structure.
we show

---

LIMA (1,000) Examples
LIMA with Dialogue Examples (1,030 Examples)
scientist who just invented time machine. Where you travel first?
You are a scientist who just invented fime machine. Where do you travel first?
As scientist, I would travel back in time the year 1969 and watch the Apollo 11 mission land on Moon.
would travel to Ancient Egypt.
have always been fascinated the history of ancient civilizations, and Ancient Egypt is one that has always intrigued me most. Seeing pyramids person would an amazing experience, they are such mysterious and incredible feat of engineering for their time. Also, I would love to see how people lived back then, what life was like for them, and try learn some of their knowledge and culture.
The Apollo 11 Mission was the first flight that landed humans on the Maon, was incredibly important moment histury as marked mankind's is first step towards exploring other planets. I would love to witness this historic event first hand and be able to say that I the moment when Neil Armstrong tonk his famous first step on the Moon.
If possible, would try to meet with some the more influential ligures in their history, such as Cleopatra or Rumses IL would be so interesting hear their thoughts on their kingdoms and their place in history, and how they view their logacy today.
Could you turn it into a fictional essay?
As a scientist, I would traell back in time to the year 1969 and witness the Apollo 11 mission land on the Moon.
However, if could only observe without interncting, might choose to watch the building of the Great Pyramid of Giza. To witneas such impressive feat construction at such an early point human history would be truly awe inqpirig..
The Apollo 11 Mission was an incrodibly important moment in history it marked mankind's I's first step towards exploring other planets. would love to have been there when Neil Armstrong took his famous first step on the Moon.
Could turn it into fictional essay?
Unfortunately, my time machine isn'l very reliable and can only take mo to places within my lifetime. This means that could go forward in time as well as back. While it might be tempting to see what changing the course
Sure, here fictional story of my travels to Ancient Egypt:
future techmologies exist, wouldn't want of history hy interacting with people from the future. Therefore, I will remain safely in the past where cannot accidentally alter the timeline.
step into the mashhine and set cuntrols for BC. the machine powers I feel strange sensation as if time itself is polling me backwards. When stops, am in a vast desert with golden structures lowering above me. I have arrived in ancient Egypt!
Can you create title for the essay?
look around, I soe people dressed in linen elothes [...]
An Incredibly Important Moment History: A Scientist's View the Apollo 11 Mission.
[...] our meal of flat bread, dates, goat cheese, tells me his name is Imhoteep and that he serves as advisor to Pharach Djoser.
but you didn't 't really describe the scene of Apollo 11 mission land, could you imagine you saw the landing and rewrite the essay?
Imhotep is fascinsted by my stories of the future l...1
Before long, time for seturn home. bid farowell Imhottp and thunked him for his huspitality. As I powered up the time machine, looked back the pyramid and wondered if would ever be finished.
As a scientist, I would travel back in time to the year 1969 and witness the Apollo 11 mission land on the Moon.
The Apollo 11 Mission was an incredibly important moment in history it marked munkind's first step towards exploring other planets. I would love to have been there when Neil Armstrong look his famous step on the Moon.
Can you create a title for the essay?
about 'Ancient Astronauts: Fictional Account of Alien Visi- tation in Ancient Egypi"?
Using my machine, went back to July 20, 1969, day the Apollo II landing. When I arrived, hid near the landing site that I could watch the historic event unfold without interfering with
Why is the essay related to astronauts and allens?
anything. At 4:17 p.n.. beard the sound of rucket engines the lunar module began its descent into the moon's 's atmosphere. A few minutes later, the module touched down the surface the moon. Then, at precisely 10:56 p.m., Buzz Aldrin stepped foot on the Moon followed by Neil Armstrong a few moments lator.
The title inspired the beliel held by some that aliens visited Ancient Egypt and contributed to their advanced knowledge of engi- neering and culture. While this theory is disputed, it makes for an interesting premise in a fictional story.
hmmm, but the story you wrote has nothing to do with allens and astronauts.
I remained undetected throughout the entire operation. After Arm- strong and Aldrin returned to Earth, I traveled back my own time period with the knowledge that one of the most significant events in human bistory had gone off without a hitch.
You'de re right, it doesn't. My fictional account is just play on the idea that aliens visited Ancient Egypt passed off their knowledge as their own. In story, I used the time machine to travel back in time instead of an alien spaceship. The essence of the original theory remains: someone from another advanced civilization visiting Ancient Egypt and shuring their knowledge with them.
cool, could you summarize the essay in one sentence?
As a scientist. I would iravel back in time to the year 1969 and witness the Apollo 11 mission land on the Moon because it was incredihly important moment in history.
Sounds good, could you summarize the story into bullet points?
Sure, here are the main points of my story:
travel to Ancient Egypt using time machine.
meet with Imhotep, who advisor to Pharzoh Djuser.
[..]]
Figure 8: An example dialogue with LIMA, with and without 30 dialogue examples. The gray ellipsis [...] indicates that the response was trimmed to fit this page, but the generated text is actually longer.
9

---

7 Discussion
We show that fine-tuning a strong pretrained language model on 1,000 carefully curated examples can produce remarkable, competitive results on wide range of prompts. However, there are limitations to this approach. Primarily, the mental effort in constructing such examples is significant and difficult to scale up. Secondly, LIMA is not robust as product- grade models; while LIMA typically generates good responses, an unlucky sample during decoding or an adversarial prompt can often lead to weak response. That said, the evidence presented in this work demonstrates the potential of tackling the complex issues of alignment with a simple approach.
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22
2022b.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. In Proceedings of the international AAAI conference web and social media, volume 14, pages 830-839, 2020.
Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazneen Rajani, and Nathan Lambert. Stackllama: An rl fine-tuned llama model for stack exchange question and answering, 2023. URL https: h/w.://ggggggac..eee.e/oo/ co/blog//////cccc/co sttg//stccklaaam
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https
//Imsys. wwtpooggg/booooooooooooooooooooo 2tloooooo///22-33-2000
vicuna/
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint vrxiv:2204.02311 2022.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, et al. Scaling instruction-finctueed language models. arXiv preprint arXiv:2210. 10. 1416, 2022.
Avia Efrat and Omer Levy. The turking test: Can language models understand instructions? arXiv preprint arXiv:2010. /982, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019,
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor, 2022.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 2019.
Yuval Kirstain, Patrick Lewis, Sebastian Riedel, and Omer Levy. A few more examples may be worth billions of parameters. arXiv preprini arXiv:21 10.043044444 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ICMI. 2022 Workshop on Knowledge Retrieval and Language Models, 2022.
10

---

Andreas Köpf. Yannic Kilcher. Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc. Oliver Stanley. Richárd Nagyfi. Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment. arXiv preprint ar (Tti::2304.073222. 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Natural instructions: Benchmarking generalization new tasks from natural language instructions. arXiv preprini arXiv:2104.08773, pages 839-849, 2021.
OpenAL. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, 2022.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-drived self-alignment of language models from scratch with minimal human supervision, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follont followin llama model. htttps://githul
oort:/atttaaaaatttaaooori.r..d alppca 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022a.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap. al. Sugerratuuallssnrrtoossgnneniiin via declarative instructions on 1600+ tasks. In EMNLP, 2022b.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models zero-shot learners. In International Conference on Learning Representations, 2022a.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models, In Advances in Neural Information Processing Systems, 2022b.

---

A Training Examples
Figure shows six training examples from various sources.
B Anticorrelation between Perplexity and Generation Quality
When fine-tuning LIMA, we observe that perplexity on held-out Stack Exchange data (2,000 examples) negatively correlates with the model's ability produce quality responses. To quantify this manual observation, we evaluate model generations using ChatGPT, following the methodology described in Section 5 Tigure@Ssowwe that perplexity rises with more training steps which is typically negative sign that the model is overfitting so does the quality of generations increase. Lacking an intrinsic evaluation method, we thus resort manual checkpoint selection using small 50-example validation set.
C Human Annotation
Figure 11
ssowws the human annotation interface we used to collect preference judgments. Annotators asked exercise empathy and imagine that they the original prompters.
To remove any confounders from the annotation interface, we present annonymized and randomly ordered LIMA response and baseline model response annotators. Besides, performed screening annotation tasks with annotators and selected annotators who achieve the highest agreement scores for the final annotation.
D ChatGPT Score
Automatically evaluating generative models is difficult problem. Forablation experiments (Section 5 we use ChaiGPT (GPT-3.5 Turbo) to evaluate model outputs on a 6-point Likert score given prompt in Figure 12
E Generating Text with Complex Structure
In our preliminary experiments, we find that although LIMA can respond to many questions in our development set well, cannot consistently respond to questions that specify the structures of the answer well, e.g. summarizing an article into bullet points or writing an article consisting of several key elements. Hence, we investigate whether adding a few training examples in this vein can help LIMA generalize prompts with unscen structural requirements. We added six examples with various formatting constraints, such as generating a product page that includes Highlights, About the Product, and How to Use or generating question-answer pairs based on given article.
10
PPL
4.2
Quality
Generation Quality
Valid PPL
4.0
3.9
60
120
180
240
300
360
420
Training Steps
Figure 9: Validation set perplexity versus generation quality (as evaluated by ChatGPT), across the training process of LIMA 65B. We observe similar trends for 7B and 30B parameter models, and across different mixtures of training data.
12

---

IStack Exchange (STEM)
[Stack Exchange (Other))
[wikiHow]
What is the difference between minimum and infimum? I have a great confusion about this.
Was the Millennium Falcon one-olf was it mass produced??
How to be a lazy college student??
In college, unlike in high school. you are pected to take responsibility for your own learning. It is ultimately up to you to deter- mine how much time and effort you want to put into your schoolwork. Experts sug- gest that college students today work consid. crably fewer hours than their counterparts did in 1961. there ing more hours is not necessarily the best way to be productive. A little laziness may oven be good for us.
The Millennium Falcon is not unique; it was a model YT-1300f Corellian light freighter. This implics that there are other similar ships. including other variations on the YT-1300 model. You find a lot of information about the Millennium Falcon in companion baok, Wars: The Fonce Awakens: In- credible Cross-Sections. The model number also appears ou-scroen least once: in Star Wars: Rebels episode "Fighter Flight" poster. Note, however, that the Millennium Falcon has heen highly customized;; some the custom work is mentioned in The Force Awakens, It's clear just how much of the ship is stock and how much is custom.
The minimum attained, the infimum isn't necessarily.
Example: Let f(x) 4. Then f has minimum value the interval (0, oe). The minimum is the smallest element in the set, That min[ [(x) x E ee) doesn't exist because there is not smallest number in the set, Another example the minimum 1). Here again there isn't a smallest number min[ x I) doesn't exist.
evidence that work-
set S (0. 1) = [x
Part 3: prioritize what's important
1. Ask yourself what you want from college. There are many reasons why a college de- gree is desirable, not least that is still considered highly valuable by many em- players. Your own reasons for pursuing college degree may he more personal, but spend some time working out exactly what these reasons Consider why col- loge is important to you. Do you want to learn important skills like critical think- ing and solling Do you want friends, network and get know other people? Do you want to get good grades for further study? Do you want get particular job that requires col- loge degree? Spend some time reflecting on these questions and come up with your own answers. It might even be good idea write them down. This help you to recognize what's 's important you so that you can focus on the essential things.
The inlimum of set delined as greatest number that is less than or equal to elements of (from Wikipedia). The infimum is also sometimes called the great- est lower bound. It is a fact that every non empty (bounded below) of real numbers has an infimum. But, as we saw, not every real has minimum. in the example inf ((0 x E (0, oo)) = 0.
you're willing lo go outside Canon into I Legends, the model number of the ship, and other similar Corellian light freighters, be soen in some of the vidco games, and there an entire novel called Millennium Falcon that details its history. The ship was serial #Y I- 1-100 492727ZED, one of many such vessels, but from the beginning the ship shown in have some "quirks" to distinguish from its counterparts. In Logends, the ship was also destroyed and rebuilt from scrutch least unce, it's arguable as whether it's still the same ship not.
Note that the infinum and the minimum can be the same. Consider for example S = 3..... Then the infimum and mini- mum is both 1. Consider this other example. f is a continouu function on closed in- terval [e. b). then it is a fact that f attains minimum over that interval. So here again inf (f0) [a, minl/(X| [a. bl).
Select your major and your classes care- fully. I...]
[Manually Authored- - Chitchat]
[Manually Authored- - Advicel I'm going NeurlPS the first time present the first paper I've ever published. I'm afruid I'll be alone and overw helmhe by everyone. What should do?
|Manually Authored - Writing] am planning to start book club with some friends. Can you write email invi- Iation for the same?
Tell me an interesting fact about geogra- phy.
Here interesting facts about geogra- phy:
Here an example email invitation for a book club:
Going an academic conference for the first time may seem like bewildering experience. but done right, conferences are wonderful opportunity brainstorm with experts from around the world, present your work and gel feethack, and make friends. Here some tips for improving your conference ex- perience:
Bhutan is one of the most intriguing places the world. The Budkhist kingskom is the only carbon-negativee country on Earth, strict envirenmental laws and cultural norms that preserve its stunning natural landscape. For similar reasons, Bhutan was one of the last countries to introduce television in 1999, and has no traflic lights this day.
Subject: Join me for book club!
Ili [Friend's Name], J. how are you?
I'm starting a book club with some friends, and I'd love have you join us!
Reach out advance people who have inspired your work. [..]
The baok club is casual and fun way read and discuss books that love want to learn more about, and also make new friends. We plan to meet more-or-less once month.
There currently lew territories Earth that are unclaimed by any sovereign stale
*
Show curiosity about other people's is work. [..]
(terra nullius). One example Bir Tawil,
along the border between Egypt and Su-
dan, which was created by discrepancy
Sign up for student volunteer I...]
we're thinking having our first meeting on [Date & Time]. Could that work for you?
between borders drawn in 1899 and 1902. The territory remains unclaimed by either country, because would entail foefeitng the much larger Halaib Triangle, which is claimed both countries. Bir Tawil has no settled population. but the land is used by Bedouins roam area.
Ask your advisor to introduce you [...]
* Ar the same time, try to socialize outside of your lab [..]]
Would love to you there!
Ultimately, it is important remember that every fist-coofference attendee is probably feeling a bit shy too, and that laking lirst step in starting a conversation will almost always be met with a big smile!
Thanks, (Your Name]
The pntipode of any spot Farth [..]]
Figure 10: Training examples from different sources. Top row: examples mined from community Q&A. Bottom row: manually-authore examples. The blue text is the prompt. The gray ellipsis [...] indicates that the response was trimmed to fit this page, but the actual training example is longer.
13

---

After training with these six additional examples, we test the model on a few questions with format constraints and observe that LIMA responses greatly improve. We present two examples in Figure [3] from which we can see that I.MAA fails to generate proper answers without structure-oriented training examples (left column), but can generate remarkably complex responses such marketing plan even though we do not have any marketing plan examples in our data (right column).
14

---

Imagine that you have super-iutelligent assistant, that you require help with following question. Which best satisfies your needs?
Question: <QUESTION>
Answer
B:
<ANSWER
<ANSWER
Comparing these two answers. which answer better?
Answer significanlly better.
Answer significunlly betler.
Neither significantly better.
Figure 11: Human annotation interface.
You are evaluating response that has been submitied für particular task, using specific set of standards. Below is the data::
[BEGIN DATA]
[Task]: [task]
[Submission]:
[submission]
[Criterion]: helpfulness:
"I": "Not helpful The generated text is completely irrelevant, unclear, incomplete. does not provide any useful information the
"2": "Somewhat helpful The genertte text has some relevance in the user's question, but may he unclear or incomplete. It provides only partial information, the information provided may be useful for the needs." *
"3". "Moderately helpful- The generated text is relevant to the user's question, and it provides a clear and complete answer. However, may detail explamation
be helpful "4" "Helpful The generated text is quite relevant to the user's question, and it provides clear, complete, and detailed answer, It offers are useful
user."
additional information or explanations
user. However, some points the response are somewhat repetitive or could
combined for greater clarity and concision"
"5": "Very helpful - The generated text is highly relevant to the user's question, and provides clear, complete, and detailed answer. It offers additional information, explanations, analogies the response is not well-organized and there is no clear progression or logical sequence of different points in the response."
only useful also insightful and valuablc the However, the structured
generated text provides clear, complete, detailed answer. are not only useful but also insightful and valuahle to the user. The response is also in logical and easy- follow manner by explicitly using beadings, bullet points, or numbered to break up the information and make casier to read." see
"lighly helpful
additional information explanations
[END DATA]
Does the submission meet the criterion? First, write out in step step manner your reasoning about the criterion to be sure that your conclusion correct. Avoid simply stating the correct answers its own line corresponding to the correct answer. At the end, repeat just the selected choice again by itself on new line.
Then print the choice only from
quotes or punctuation)
Figure 12: Prompt for ChatGPT evaluation with 6-scale Likert score. The placeholders "task" and "submission" will be replaced by specific details from the actual case being evaluated.
15

---

LIMA w/o 6 Format Constraint Examples (994 Examples)
LIMA (1,000 Examples)
[Mrrktinng Plan (Out of Distribution))
I want to start a local coffee shop. Create a marketing plan with the following elements:
Marketing Goals and Objectives, Define Target
Andience, Research Marketing Tactics, Plan Marketing Tactics and Develop Your Timeline and Budget.
Coffee Shop Marketing Plan
Here's a marketing plan your local coffee shop.
Marketing Goals and Objectives
Executive Summary
Increase awareness of our brand among consumers [-..]
The following marketing plan for Java Jolt Coffee Bar outlines
ageressive appreach to catablishing a thriving coffee shop business
Drive foot traffic [..] during slow business hours
[..]]
downtown Pitlsburgh. Java Joll will capiialize the popularity
Build relationships with influencers in our community who [...]
of specially coflees and the growing demand for Wifi hot spots
by offering comfortable environment where customers can enjoy
Define Target Andience
premium collee drinks while laking advaniage of free Internel Market research indicates that there is significant opportunity for
Our target audiences include:
new coffee har located in the heart the downtown basiness district.
Residents of the neighhorhood surrounding our coffee shop
Market trends suggest that consumers are willing to pay higher
Office workers near our coffee shop
price for quality specially colfees served welcoming atmosshere. Additionally, many office workers and students living in the area do
Students al nearity schools
not have easy access to high-speed Internet service, presenting an additional revenue stream for the coffee har.
will focus our marketing efforts on these audiences bocnuse they [-..]
most likely hecome reyular customers
Research Marketing Tacties
This marketing plan includes following elements:
Here some marketing tactics that we could use to reach reach
our target audiences and achieve our gouls:
Market Reseurch supporting the need for collee shop in the
turget location
Host events at our coffee shop that appeal to our target audiences
(e.g., open mie nights for students networking events [..]]
Detailed Deseription of the products and services offered by Java Jolt
Offer discounts during slow husiness hours to entice [...]
Target Market Analysis identifying the primary customer segments for Java Jolt
[..]]
Plan Marketing Tactics
Marketing Strategy ludinin pricing decisions, peomotional ac-
Here's how we will implement marketing tacties above:
rivities, and ways to differentiate Java Jolt from competitors
Events: We will host one event per month our coflee shop. For
Implementation Plan detailing the actions necessary to launch Java
each event, will send email invitations [-..]
Jolt successfully
Discounts: Every Monday through Thursday from 3pm-6pm,
Budget allocating funds each component of the marketing plan
will offer $I off any drink purchase. To promote [...]
Local partnerships: We will reach out five local businesses and schools to ask we leave stacks of coupons |...]
A comprehensive marketing plan has been developed based on exten-
rescarch the local market conditions, consumer trends, competitive landscape. With its unique combination of premium caf-
Social media: [..]]
for drinks and wireless Internet access. Java Jolt will quickly become
Reviews: [...]
a lavorite destination for people seeking relaxing environment socialize, study, or conduct business.
Develop Your Timeline and Budget
This what our marketing timeline looks like:
Month Launch newsletter and social media accounts
Month 2: Start distributing coupons [...]
[...]
Here's the budget for each marketing activity:
Email newsletter: $50/mooth for MailChimp subscription
Social media: $100/month for paid ads [...]
[-]]]
[Summariration into Bullet Points (In Distribution)]
you summarize this message into bullel points? Gradually slowing job gains and a growing labor force March [...] Mr. Biden said [--]. More Americans joined the labor force, [--] this spring that he will seek re-election.
Though the unemployment rate fell its lowest level since May 1970. at 4.8 percent, Mr. Biden still faces an economy that leaves many Americans feeling left behind. Many working-clan
Here the main points from the text:
Job creation dowed to 236,000 for the month of March, which is
l...
close to the level needed to stabilize the economy and prices.
Mr. Biden's 's challenge will be persuading those voters that the eco- nomic picture has [..] opponents have already begun painting bleak portrait of the cconony, arguing that most Americans have not
More Americans joined the labor force, and wage gains fell slightly. both of which should help cool inflation.
benefited from the president's policies.
The repurt underscores the political tensions facing President Biden ahead expected announcement that be will seek re- election.
"The March jubs report shows continued solid job growth but also indicates that we must do hetter," said Representative Nancy Pelosi California, Democratic leader the House. [...]
Figure 13: Model outputs from test prompts that ask the model to generate according to specified structures. The gray ellipsis [...] indicates that the response was trimmed to fit this page, but the generated text is actually longer.
16


Reviews:
[{"review_id": "wvAxbNS9lc", "reviewer": "Reviewer_a3cj", "summary": "This paper hypothesizes that alignment of pretrained LLMs to become user friendly and useful AI chatbots is mainly about learning the style or format of interacting with users, and that all/most knowledge and capabilities were already acquired by the LLM during pretraining. To test this hypothesis, this paper curates 1,000 diverse high quality examples that approximate real user prompts  along with high-quality responses, and trains a new model called LIMA. LIMA demonstrates good performance in a controlled human study; responses from LIMA are either equivalent or strictly  preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback.\n", "questions": "How do you fine-tune Llama on the Alpaca dataset?  Is there a standard setting which the authors released? Did you perform a hyper-param search? Since this is the only baseline model that LIMA beats, and the only one that you trained in-house, it is important to make sure that it receives at least equal attention in training as your model. I’d be happy if you could elaborate exactly on how you trained both LIMA and your version of Alpaca.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper addresses an important and timely question aiming to shed light on the mechanisms behind the huge success of instruction tuned LLMs.  The paper is very well written and easy to follow. I really like the straightforward approach of the paper: it clearly defines the crisp question regarding the role of instruction tuning, and opts to answer it directly by creating a small, clean, and diverse training set. The resulting model is clearly a very good instruct model, and achieving this so simply is a very impressive feat. Judging this paper from an academic standpoint, I do have some concerns, detailed below.\n\n", "weaknesses": "If I understand correctly, table 1 implies that over 75% of the test set is written by the paper authors (group B) which received the same general instructions as the paper authors who wrote 20% of the training set and the entire dev set questions (group A). I see this as weakening the paper’s messages, in particular the comparisons to other models in figures 1 and 2 could be unclean. \n\nOne way to put it, as I think the paper is trying to do, is that group B are different people than group A with different interests, friends, etc, and therefore the test questions of group B can be considered an unrelated ground truth test. Another way to put it, which I think is valid, is that LIMA enjoys a significant amount of training on data that is qualitatively closer to the test distribution than any of the data used for training any of the other models. This is concerning since LIMA can be said to enjoy significant training on the tets distribution while other models don’t; this is known to have dramatic effects.\n\nAn ablation that slices the results of figure 1 according to 230 test examples written by group B and 70 test examples from Pushshift r/AskReddit can ease my concern here; if the two groups exhibit similar trends, then training on data from group A did not bias the results. If performance on the  230 test examples written by group B is way better, then I think the messages need to be revised. Another experiment that could ease my concern is training “clean-LIMA”  without the data from group A, and supplementing it with 200 examples from the other sources of the training set. A third alternative, perhaps better than the former, is to devise a \"clean test set\" without examples written by group B. I realize that the last two require further annotation and may be harder, but the authors did show that automatic evaluation with GPT-4 correlates with human judgement and this can be a place to leverage that. \n\nIf the authors convince me that I either misunderstood their setup or that training from group A did not bias the test results,I’d be happy to raise my soundness score and overall evaluation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for pointing out this potential issue!\n\nWe acknowledge the reviewer’s concern regarding potential similarities between Group A and Group B data, and provide the test set results (LIMA win / tie / LIMA lose), split 230 / 70 by prompt source, demonstrating that the trends are not over-specific to underlying similarities between Group A and Group B:\n\n|                   | Pushshift r/AskReddit (70) | Authors (230) | ALL (300)      |\n|-------------------|-----------------------------|---------------|----------------|\n| LIMA vs Alpaca (65B)   | 64%/16%/20%               | 50%/22%/28%   | 53%/21%/26%    |\n| LIMA vs DaVinci003     | 57%/13%/30%               | 40%/23%/36%   | 44%/21%/35%    |\n| LIMA vs BARD (April)   | 30%/29%/41%               | 33%/24%/43%   | 33%/25%/42%    |\n| LIMA vs Claude (April) | 20%/27%/53%               | 25%/20%/55%   | 24%/22%/54%    |\n| LIMA vs GPT4           | 22%/17%/60%               | 17%/27%/56%   | 18%/25%/57%    |\n\nAs we can see from the table, LIMA’s performance on AskReddit prompts is indeed similar to its performance on Group B’s prompts. In fact, when compared to Alpaca or DaVinci003, performance on AskReddit is significantly better.\n\nWe would also like to emphasize that closed models (DaVinci003, GPT-4, Bard, Claude) were possibly exposed to similar data during their training; much of the data in both Group A and Group B is inspired by how people use ChatGPT.\n\nComparison to Alpaca: We follow the training protocol of Alpaca by training for 3 epochs and other regularization techniques we used for LIMA training. We also performed hyperparameter search over training steps and regularization strengths but didn’t find any significant differences.\nWe also emphasize that LIMA also outperforms DaVinci003, not only Alpaca, though to a lesser extent.\n"}, {"review_id": "J8t4iUkzZK", "reviewer": "Reviewer_s3dY", "summary": "This paper presents LIMA, a 65 billion-parameter large language model fine-tuned on a dataset of 1,000 carefully curated prompts and responses without the use of reinforcement learning or human preference modeling. The aim of this research was to measure the relative importance of pretraining and large-scale instruction tuning in the training of language models. LIMA demonstrates strong performance and generalizability across a variety of tasks, even those not included in its training data. It also excels in specific response formats, learning from just a few examples in the training data. The study includes a controlled human evaluation where LIMA's responses were either equivalent or preferred to those of GPT-4 in 43% of cases. When compared to Bard and DaVinci003, this percentage rose to 58% and 65%, respectively. The results suggest that the majority of knowledge in large language models is acquired during the pretraining stage, and only limited instruction tuning data is required to generate high-quality output. Ablation experiments underscored the diminishing returns from scaling data quantity without a concurrent increase in prompt diversity, highlighting the critical role of data quality and diversity in training these models.", "questions": "Question:\n\n1. Please address the weaknesses above.\n2. Will the 1,000 examples and LIMA model be released to the public to further promote open research?\n3. As the training process only involves 1K examples, why the experiments are only performed on 65B LLaMA? I know that the ablations are carried out on 7B model, but it would be also interesting to see a progressive analysis when model scales from 7B, 13B, 33B to 65B. \n\n\n\nOther comments: There are some typos and grammatical issues that should be corrected.\n\n1. line 2: large scale -> large-scale\n2. line 37: LLaMa -> LLaMA\n3. line 163: state of the art -> state-of-the-art\n4. line 206: Of -> of\n", "limitations": "The authors discussed their limitations in Section 7.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents an effective approach to training large language models, focusing on the importance of pretraining and using a limited number of carefully curated prompts for fine-tuning. This approach challenges the common practice of using large-scale instruction tuning and reinforcement learning.\n2. The experimental results indicate the effectiveness of the proposed training approach, which shows strong performance compared to various LLMs, including commercial ones.\n3. Detailed ablation studies are given regarding the data diversity, quality, and quantity, which allows the readers to better understand different aspects of the proposed approach (and data).", "weaknesses": "1. The proposed training scheme is only tested with subjective evaluations. I wonder if the proposed method can also give significant improvements on objective benchmarks, such as MMLU, etc.\n2. Also, the evaluation can be further improved by introducing different settings, such as CoT, few-shot, etc.\n3. It is not quite clear how the conclusions can be applied to other scenarios, such as different languages, domains, etc. The paper can be further strengthened by providing some takeaways to the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- Weakness1: We intentionally did not evaluate using MMLU and other multiple-choice or classification benchmarks since they are unrelated to the functionality that we tried to imbue in the model. The experiments in LIMA show that an assistant (chatbot) can be trained from a strong pretrained model using only 1,000 high-quality examples. We were therefore interested in evaluating LIMA on prompts that are likely to be asked by assistant users, e.g. life advice, creative writing, coding, etc.\n\n- Weakness 2: Again, both CoT and in-context learning are not aligned with how laypeople are likely to use a chatbot, and it therefore does not make sense to evaluate it in those settings. We expect the model to produce high-quality outputs directly, in a zero-shot setting, without any assistance from the prompt.\n\n- Weakness 3: The same methodology can be applied to any scenario, given a strong enough pretrained model and a set of high-quality examples in the style of the desired functionality.\n\n- Q2: The examples, including the test set, are available online.\n\n- Q3: We did experiment with smaller models (see ablations), though we do not report human evaluation compared to other SotA models for those. We can add GPT-4 based evaluation to the paper’s appendix.\n"}, {"review_id": "dRR7DDRIFe", "reviewer": "Reviewer_xtqr", "summary": "The authors investigate on instruction tuning an LLM on high-quality manually collected/created data (only 1K training examples). They provide detailed explanation on how to collect the data, tune a 65B LLaMA model and compare it with other commercial/open-source LLMs. The main experiment results show that the LIMA 65B LLaMA model is better than 65B Alpaca and slightly worse than the top commercial models (GPT, Claude and Bard), based on human and GPT-4 judgements. The authors also conduct other ablation studies including dialogue modeling ability, safety, data quality v.s. quantity.", "questions": "In line 78, \"Within each exchange, we take the questions with the highest score that are self-contained in the title (no body)\". The reviewer doesn't understand this sentence. Could you clarify the meaning of the terms here? What is the score for, what is \"no body\" and what does self-contained mean here?", "limitations": "The way of manually collecting high quality data could be expensive and hard to scale. It's still a question that given the same amount of money, which option is more effective: manually collect data and guarantee quality; or just prompt powerful closed-source LLMs to get a huge amount of data.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The presented data quality is very high. There are many recent papers about instruction tuning on a large amount of instruction data (typically by prompting GPTs), while this paper demonstrates that quality can play a bigger role. This finding is very inspiring for the community.\n2. The LIMA model can also somehow reject unsafe queries, which is difficult for Alpaca, Vicuna or other fine-tuned LLaMAs on GPT generated data. This result is surprising to the reviewer as the ability of rejection to unsafe query is more likely to gain from RLHF.\n3. The model also show promising performance on dialogue tasks. This is good but actually expected, as Vicuna showed that dialogue ability is not hard to gain.", "weaknesses": "1. The test set is kind of small. In addition, although the authors mention they put effort to make the test data diverse and distinct from the training set, the diversity of the test set may still be a issue since all the authors are likely to fall within a narrow scope (i.e., well-trained machine learning researchers). It would be better to include more testing prompts from different people, e.g., real user inputs. This might be much harder for academia folks though comparing with OpenAI and Google.\n2. As pointed out in [Gudibande et al.](https://arxiv.org/pdf/2305.15717.pdf), the SFT tuned LLaMA models can receive good ratings comparable to ChatGPT from crowd workers but actually have a huge gap, and they called it as a false promise. The evaluation of the LIMA model is very limited to human/GPT judgments. Whether it has the similar issue is unknown from the current presentation. Maybe it's better than LLaMAs tuned on GPT generated data; or it also has the same issue --- which may indicate that RLHF's importance. Current evaluation cannot provide the readers enough information for this.\n3. Many recent papers have shown that the real emergent ability, aka what makes GPTs/Bard/Claude different from the others are complex task solving. See [Yao et al.](https://yaofu.notion.site/Towards-Complex-Reasoning-the-Polaris-of-Large-Language-Models-c2b4a51355b44764975f88e6a42d4e75), and also OpenAI’s [GPT-4 release blog](https://openai.com/research/gpt-4): “In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle. The difference comes out when the complexity of the task reaches a sufficient threshold”. The reviewer would like to see the analysis on these kind of tasks (e.g., MATH/BB-Hard) as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weakness 1: The test set does indeed include naturally occurring prompts from Reddit.\n\nWeakness 2: We agree that there are limitations to evaluating models by human preferences. However, there are even more significant limitations to evaluating models by other existing benchmarks, e.g. MMLU, which are completely misaligned with the actual usage of assistants/chatbots. Please provide a constructive suggestion for evaluating this type of functionality.\n\nWeakness 3: Indeed, but these are all examples of advances in pretraining, which is not in the scope of this paper. LIMA focuses on showing that the stages following pretraining can be greatly simplified.\n\nQ1: In Stack Exchange, each question is scored (up/down-voted) by the users. We select the top questions. In addition, each question has a “title” and a “body” (detailed description); we only use the self-contained title as our prompt without the “body”.\n"}, {"review_id": "Te542EDgqq", "reviewer": "Reviewer_w2T8", "summary": "This paper presents a superficial alignment hypothesis. It claims that most knowledge and abilities of large language models (LLMs) are learned in the pre-training stage, and a few datasets are just enough to transform the styles of LLMs for alignment. \n\nTo support the argument, the authors of the paper finetune LLaMA on the few curated finetuning dataset only having 1000 instances. The curated dataset consists of community-based Q&A such as Stack Exchange, WikiHow, and Reddit, and manually authored examples.\n\nExperimental results (human evaluation on 300 test samples) show that the resulting model, LIMA outperforms baselines including Alpaca 65B, Davinci003, and Bard, and shows compatible performances with other proprietary LLMs, Claude and GPT-4. \n\nFurther analyses confirm the results qualitatively, showing generalizability on the safety or multi-turn scenarios. Moreover, the data quality and diversity are suggested to be the most crucial factor for the alignment.", "questions": "1. The superficial alignment hypothesis is still valid in the small-sized models? Although you reported the ablation study based on the LLaMA-7B, we do not know that the model generates better responses compared to other baselines.", "limitations": "- It would be good to include evaluation results on other automatic evaluation benchmarks (I suggested in the Weakness).\n- More details for the manually designed prompts (for the test) make the paper more concrete.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The hypothesis is convincing, and the experimental results support it well.\n2. The main message is clearly delivered.\n3. Evaluations and corresponding cross-agreements are well conducted.", "weaknesses": "Overall the evaluation is limited.\n  - a. More details for the manually designed test prompts are needed, e.g., category, distribution, etc.\n  - b. Human (GPT-4) judgment does not guarantee the intended behaviors of the models, according to [1].\n    - b-1. There are automatic evaluation benchmarks related to alignment learning, such as HHH evaluation in Big-Bench, TruthfulQA, and RealToxicityPrompt.\n    - b-2. Also conventional NLP tasks can also be considered as the target of measurements. Specifically, knowledge probing (MMLU or Reasoning tasks) is a critical factor in getting the model, not just mimicking the preferred styles.\n  - The number of examples for the safety or multi-turn analysis is too small. Therefore, I think the claim for the multi-turn dialogue is not convincing, and the analysis should be based on at least 50-100 examples.\n  - Ablation study is conducted on the 7B model, unlike the main experiments based on the 65B model. We could not guarantee the same tendency would be observed with the different-sized backbone.\n\n[1] The False Promise of Imitating Proprietary LLMs (Gudibande et al.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weakness A. While collecting prompts, Group B assigned each prompt in the test set to a category based on their own definitions. This is a noisy process, and the categories are not entirely well-defined, which is why we did not share this information in the paper. However, per your request, we share these categories and the distribution of prompts here:\n\n- writing: 13.7%\n- coding and stem: 9%\n- knowledge: 18.7%\n- advice: 18.7%\n- commonsense: 2.7%\n- safety: 2.4%\n- brainstorm: 3.3%\n- roleplay: 3.3%\n- fermi: 1.3%\n- counterfactual: 1.7%\n- constrained generation: 1.7%\n- opinions: 4%\n- NLP: 0.3%\n\nWeakness B. Thanks for your questions! Please check our general responses.\n\nWeakness C. It is true that the evaluation is based on a limited number of dialogue turns. However, we still find it shocking that LIMA, which was not tuned on multi-turn dialogue at all, is able to conduct some degree of dialogue. We thus feel that it is important to report this observation, even if its evaluation is limited.\n\nWeakness D. We performed ablation studies on the 30B model too (not reported) and we made similar observations as in the 7B model. We can include this in the final version’s appendix.\n\nQ1: The Superficial Alignment Hypothesis explicitly states that the knowledge and capabilities are determined by the pretrained model. Therefore, a weaker (smaller) model should be less capable, even after fine-tuning. Indeed, we observe that while smaller models were able to learn the assistant style from 1,000 examples, the content produced by these models was sometimes lower. For example, in coding prompts, we found that the 7B model was more likely to produce bugs. However, for many other prompts, such as email writing, the differences were not always discernible. These observations are based on our manual analysis of the models; we did not conduct standardized evaluation for 7B models beyond what is presented in the ablations.\n"}, {"review_id": "5j7DNWI1Mr", "reviewer": "Reviewer_fPTm", "summary": "In this paper, the authors present an LLM finetuned from 65 LLaMa with very limited 1000 samples. LIMA achieves very impressive results in the human evaluation, with competitive results against GPT4/Bard/davinci003. According to my understanding, the main contribution of this paper lies on two aspects: 1. verifying fact that carefully constructed samples enhance LLM more; 2. deriving a well-constructed SFT dataset as well as a strong LLM.", "questions": "The claim of this paper is very strong and impressive. I think such a strong claim should be supported by aa more thorough experiments.\n1. The test set is from pushshift r/AskReddut and authors' questions. I would be helpful to see its performance under a finegrained evaluation on its capacability, such as knowledge QA (literature, science), reasoning, toxicity, coding and etc.\n2. Human evaluation is subjective. I wonder if the authors can provide results with objective metrics.\n3. Out-of-istriution analysis is interesting. Is it possible to evaluate LIMA on questions from a third party (e.g. questions from sharegpt)?\n4. LIMA is tuned with 1000 samples. Do you try other number of sampels? I wonder how LLM performance changes against increasing quantity of well curated samples. How do you determine the number of well curated samples set for a certain model? And is the conclusion consistent across base models?\n\nIf my concerns are addressed, I am happy to raise my score.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper verifies a fact that carefully constructed samples enhances LLM more\n2. The presented model is a very strong competitor to existing non open-source LLM\n3. The extensive analysis on data diversity, quality and quantity is useful.", "weaknesses": "1. Quality experiments are limited, where its empirical results are mostly from human evaluation on self-constructed dataset;\n2. The data construction process is hard to scale\n3. The evaluation is conducted on very limited data (~300)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Question 1: While collecting prompts, Group B assigned each prompt in the test set to a category based on their own definitions. Here are the categories and distribution of the prompts in the test set:\n\n- writing: 13.7%\n- coding and stem: 9%\n- knowledge: 18.7%\n- advice: 18.7%\n- commonsense: 2.7%\n- safety: 2.4%\n- brainstorm: 3.3%\n- roleplay: 3.3%\n- fermi: 1.3%\n- counterfactual: 1.7%\n- constrained generation: 1.7%\n- opinions: 4%\n- NLP: 0.3%\n\nAs can be seen, our test set is diverse in terms of tasks. We can add a fine-grained analysis of LIMA’s performance in the appendix with respect to each category in the final version.\n\nQuestion 2: Please check our general responses. TL;DR: human evaluation is still the least-worst option, given that we do not have a better way to evaluate open-ended long-form generation.\nTo the point of subjectiveness: we would also like to point out that reviewing NeurIPS papers is a subjective process, with proven inherent limitations, yet we still do it because we lack a better alternative.\n\nQuestion 3: The test set does include 70 prompts from questions on AskReddit, which is completely out-of-distribution. The model has never seen data from AskReddit.\n\nQuestion 4: Our ablations do explore what happens when the model is tuned on more data from the same distribution, demonstrating a “glass ceiling”. However, the final dataset used in LIMA is more diverse, in particular due to the manually-authored responses. Unfortunately, these are expensive to scale, but the contribution of this data is significant. We do hypothesize that alignment tuning can be scaled, but that the scaling law is a function of the dataset’s *diversity* and not only its *quantity*.\n"}]

Author rebuttal:
## General Response

A general concern among reviewers is that the human evaluation in LIMA is limited, and that the paper should also include automatic benchmark evaluations. While we agree that there are limitations to evaluating models by human preferences, we argue that there are even more significant limitations to evaluating **generative** AI models via existing benchmarks, especially when considering long-form responses to open-ended prompts. This problem stems from the fact that conversational prompts can have multiple good, correct responses that are completely different from one another. For example, if we ask a model to generate a humorous poem about the reviewing process at NeurIPS, one can imagine many excellent responses that wildly differ in style, content, and lexical choice. Therefore, human evaluation, with all its limitations, is the “least worst” option for evaluating the generative and conversational capabilities we hoped to imbue in LIMA.

In contrast, the proposed benchmarks, e.g. MMLU, focus on testing knowledge. They are completely misaligned with the actual usage of generative AI assistants, in both format (multiple choice vs open-ended generation) and in spirit (concrete knowledge vs creativity). We therefore do not use these benchmarks.

## Response to Ethics Reviewer KQa4
1. As mentioned by the reviewer, the paper acknowledges that any work on generative models includes the risk of generating toxic content. Since the model itself was not released, the risk of laypeople misusing the model is mitigated.

2. The paper does conduct some degree of safety analysis. We emphasize in the paper that the model is not product-grade in terms of safety.

3. Crowd workers were compensated at 15 USD / hour, via Amazon Mechanical Turk, after a training and vetting phase where annotators were also paid at a rate of 15 USD / hour.

## Response to Ethics Reviewer 43fz
Crowd workers were recruited on Amazon Mechanical Turk following standard protocols of paid vetting (15 USD / hour). We do not know and cannot provide any identifying information of crowdworkers, since it is not ethical.

Meta-review: This paper addresss the importance and effectiveness of pretraining and fine-tuning for large language models, which is an important topic. 

All reviewers initially gave acceptance scores and some reviewers raised the score during the rebuttal period.

AC also agrees with the reviewers, so decides to recommend accepting this paper.
Decision: Accept (poster)