KIWI: A Dataset of Knowledge-Intensive Writing Instructions for Answering Research Questions (2024)

Abstract: Large language models (LLMs) adapted to follow user instructions are now widely deployed as conversational agents. In this work, we examine one increasingly common instructionfollowing task: providing writing assistance to compose a long-form answer. To evaluate the capabilities of current LLMs on this task, we construct KIWI, a dataset of knowledgeintensive writing instructions in the NLP research domain. Given a question, an initial model-generated answer and a set of relevant papers, an expert annotator iteratively issues instructions for the model to revise and improve its answer. We collect 1,260 interaction turns from 234 interaction sessions with three stateof-the-art LLMs. Each turn includes a user instruction, a model response, and a human evaluation of the model response. Through a detailed analysis of the collected responses, we find that all models struggle to incorporate new information into an existing answer, and to perform precise and unambiguous edits. Further, we find that models struggle to judge whether their outputs successfully followed user instructions, with accuracy at least 10 points short of human agreement. Our findings indicate that KIWI will be a valuable resource to measure progress and improve LLMs’ instruction-following capabilities for knowledge intensive writing tasks.1

heading: 1 Introduction
text: As LLM-powered conversational agents (OpenAI, 2023; Touvron et al., 2023; Google, 2023) have gained widespread adoption, users have turned to these systems for assistance on a wide range of real-world tasks. In particular, recent works studying LLM interactions “in the wild” (Ouyang et al., 2023; Zheng et al., 2023a) have discovered that 10-20% of user queries contain requests for writing assistance, i.e. using an LLM to create, revise,
∗Work performed during an internship at AI2. 1Our dataset and code is released at https://www.cs.u
texas.edu/~fxu/kiwi/.
or organize a piece of written text. However, we currently lack an understanding of LLMs’ capabilities as writing assistants, especially in knowledgeintensive settings. What types of writing instructions do users issue to LLMs, and how well can LLMs follow different types of instructions?
We present KIWI, a dataset of expert-written Knowledge-Intensive Writing Instructions to better understand LLMs’ instruction-following abilities as writing assistants. To collect KIWI, we set up an interactive interface between a researcher and an LLM (Figure 1). We first prompt an LLM to generate a long-form answer (Fan et al., 2019) to an NLP research question, based on a set of passages from relevant research papers. A researcher then iteratively issues instructions for the model to revise the answer and evaluates the model-generated revision. The interaction session continues until the user is satisfied with the final answer or a maximum number of turns is reached. KIWI contains instructions collected from 234 interaction sessions with three state-of-the-art language models (GPT4,GPT-3.5-turbo and LLaMA-2-70b-chat), consisting of a total of 1,260 unique instruction instances. Each instance contains (1) a user instruction, (2) the model’s previous and (3) revised answers, and (4) a human annotator’s judgment (both categorical rating and free-form explanation) on whether the revised answer followed the instruction.
Using KIWI, we conduct an in-depth analysis to characterize the types of instructions issued by researchers, and to measure how well models can follow different types of instructions. We find that LLMs do not excel at this task yet, with the best model (GPT-4) achieving success for only 59% of the instructions. Specifically, LLMs fail to precisely follow user’s instructions (such as satisfying location and length constraints), often cause answer quality to degrade when integrating new information into a previous answer, and struggle to avoid making changes to answers that are not requested.
12969
Finally, we examine the ability of the strongest LLM (GPT-4) to evaluate whether a response follows an instruction, comparing its judgement to the user ratings in KIWI. Unlike prior works which found that LLMs can judge outputs as reliably as humans for open-ended instruction following (Zheng et al., 2023b; Li et al., 2023), our experiments show that GPT-4 cannot reliably evaluate responses for instructions in KIWI, which are often specific and precise. While the best performing model which retrieves and prepends 10-shot examples improves upon zero-shot performance by 5% accuracy, it still lags behind human agreement by 12% accuracy. Given these findings, we believe that KIWI will be a valuable resource for future research in instruction following and evaluation.

heading: 2 Interaction Design

heading: 2.1 Design goals
text: While prior works (Kopf et al., 2023; Zhao et al., 2023; Zheng et al., 2023a) have deployed an LLM in the wild to collect diverse types of user interac-
tion data, we focus on the use case of seeking writing assistance. We design a knowledge-intensive interaction setting with two key considerations. First, the information in the answer should be grounded to a set of relevant documents. This allows us to examine model’s ability to leverage information from multiple documents, motivated by writing tasks such as literature review (Shen et al., 2023). Second, the user interacts with the model for multiple turns to iteratively revise the answer. Compared to collecting a single editing instruction for (question, answer) pairs, this setting adheres to the iterative nature of the text editing process (Collins and Gentner, 1980; Du et al., 2022), and allows us to collect diverse and fine-grained instructions across different editing stages, as we will see in §4.1.

heading: 2.2 Interaction process
text: We provide an overview of the interaction process (Figure 1) and describe the data collection procedure in §3. The system inputs are an NLP research question q and a collection D of research papers,
which collectively contain information sufficient to write a high-quality, paragraph-length answer to q. To begin the interaction process, an LLM generates a “first draft” answer a0 to q based on D. Each interaction turn t involves four steps with actions taken by the user ( ) or the model ( ): (1) The user reads the LLM-generated answer and issues an instruction it to the model to revise the answer. (2) The model then generates an updated answer at given q, at−1 and it. (3) The user provides a rating rt and a comment ct evaluating how well at follows it. (4) The user can choose to edit the answer at if it did not follow their instruction, producing a′t. Afterwards, the user can start a new turn by issuing another instruction for at (or a′t). The interaction continues until either the user is satisfied with the answer, or a total of 10 turns is reached. We describe each step below with additional details (prompts, hyperparameters) in Appendix A.1.
Generating the initial answer Given the question q, we first retrieve a set of five passages from D with a retrieval model R, denoted as R(q,D). We prepend the retrieved passages to the question q to generate the initial answer a0 with a language model M , that is, a0 = M(q,R(q,D)).2
  Issuing instructions To start a turn t, the user issues an instruction it, specifying how they would like the LLM to revise its answer in the previous turn at−1, and categorizes its type It. We identify two types of instructions. Information-seeking instructions require the model to incorporate new information into the answer; e.g. Turn 3 of Figure 1 asks the model to acquire additional information from the source documents D. Stylistic instructions ask the model to transform content that is already present; e.g. Turn 4 in Figure 1 asks the model to compress its answer. For each interaction turn, the user indicates the type of the instruction It by selecting a checkbox in the annotation interface.
Answer revisions Next, the model M is prompted to revise its previous answer at−1 to satisfy user instruction it. We use one prompt for information-seeking instructions (which require passage retrieval from D), and another for stylistic questions (which do not).
For information-seeking instructions, we first leverage a retrieval model R to retrieves five pas-
2We perform retrieval because prepending the full text of multiple papers (each with around 6,000 tokens) exceeds the context window of the LLMs (with at most 8,000 tokens).
sages pt from D, using the instruction it as the query. Next, we prompt M to summarize the information in pt that is likely to be relevant given (at−1, it); we denote this summary as st. Finally, we generate an updated answer by prompting M with (q, at−1, it, st).3 For stylistic instructions, we generate the updated answer directly by providing (q, at−1, it) as input. We only include a single previous answer in the prompt to generate the updated answer; in the pilot study, we found that this approach yielded comparable performance to including the entire interaction history.
  Measuring answer quality After a revised answer at is generated, the user provides a rating indicating whether M successfully followed their instruction: rt = r(it, q, at−1, at). We instruct the annotators to rate at as one of the following:
• good: Successfully followed the instruction. • neutral: Partially followed the instruction. • bad: Didn’t follow the instruction at all, or modi-
fied the answer to an undesirable state.
Since our goal is to measure the helpfulness of the model response as determined by the annotator, we ask annotators to use their own judgement in selecting a rating rather than providing a detailed guideline.4 Nonetheless, we find in §5 that annotators have high agreement on this categorization. We also collect a mandatory free-form comment ct from the annotator explaining their rating rt. Example turns are in Table 9 in the Appendix A.2.
  Handling incorrect instruction-following When rt ̸= good, answer at does not fully follow instruction it and needs to be corrected. While the user might issue instructions in the subsequent turns to correct what has gone wrong with at, the success of the user’s correction attempt depends heavily on the model’s instruction following ability. The user might fall into a loop of continuing to prompt a model which is not able to perform the action requested in the instruction. To avoid such scenarios, we allow the user to edit at when rt ̸= good, which induces a′t, such that r(it, q, at−1, a′t) = good. If the user chooses to not edit the answer and rt = bad, we remove the
3We found that this “summarize-then-answer” workflow yielded better results than directly prompting M with the retrieved passages pt to produce at in preliminary experiments.
4To alleviate annotator burden, annotators are not required, but welcomed, to check the factual correctness of modelgenerated answers. Assessing faithfulness of LLM generations represents an important but orthogonal research direction.
answer from the answer history when generating answers in the subsequent turns.

heading: 3 Dataset Collection
text: We first describe our approach to curate highquality (q,D) pairs (§3.1), followed by our annotation protocol for interaction collection (§3.2).

heading: 3.1 (Question, Document set) creation
text: The process proposed in §2.2 requires high-quality (q,D) pairs to serve as a starting point for interaction collection. We aim to create (q,D) pairs which satisfy the following desiderata: (1) the questions should be realistic and likely to be asked by a researcher, (2) the questions should be challenging, requiring information from multiple documents and (3) each document set D should contain sufficient information to answer its corresponding question. We observe that the related work section of a research paper often answers a set of implicit research questions, and supplies the documents answering each question as citations. Thus, we collect high-quality (q,D) pairs by writing questions implied by related work sections, and pairing each question with its cited papers.
Source article selection We select a set of roughly 100 papers published in ACL 2023 across 11 different tracks listed in the ACL handbook.5 This set of papers covers a wide range of NLP research topics and was published after the training data cutoff time of the language models used for interaction collection in this work. The model thus has to answer the question based on the documents, instead of memorizing from its training data.
Question creation and filtering Three of the authors with prior NLP research experience manually annotate 88 questions from 75 papers. Given a source paper, the annotator first reads through its related work section and decides if a good question can be derived. A good question should fulfill the following criteria: (1) it should “stand alone”, i.e. it is not anchored in the source paper; and (2) there should be at least four articles cited in the related work which contain relevant information for answering the question. If a good question can be derived, the annotator writes a question q, together with the list of evidence papers D and the section of the related work r, from which the question is
5We choose to focus on NLP research questions, since the authors of this paper have expertise in this domain.
derived. We query the S2ORC (Lo et al., 2020) corpus to retrieve the full text of each paper and extract paragraphs. To ensure sufficient coverage, we filter out questions with either (1) more than one evidence paper missing in S2ORC or (2) fewer than 4 evidence papers in total. We obtain a total of 78 questions with an average of 6 cited papers per question.
Analysis of collected questions Our annotated questions consist of 12 words on average, with the majority of questions starting with "How" (47%) and "What" (46%). Upon manual examination, all models are able to generate a reasonable initial answer given the question and retrieved passages. This shows that our pipeline creates high quality data: the document set indeed contains sufficient information to answer the question.6 See example questions in Table 8 in the Appendix A.2.

heading: 3.2 Interaction data collection
text: At the start of an interaction session, the annotator is shown the question, initial answer, and the titles of relevant papers for the question. During the interaction, the annotator is shown the full interaction history, and the retrieved passages from the last info-seeking instruction. We release annotation instruction7 and provide annotation interface screenshot as Figure 4 in Appendix A.1.
6In the initial phase of the project, we explored using GPT4 to generate questions given related work paragraphs, but found that the generated questions were often not answerable by the cited papers. We include analysis on question generation in the Appendix A.6.
7https://docs.google.com/document/u/1/d/e/2PA CX-1vTrsM5r-p5kjy6Ue7AkbLqwOmJ-GoK9kwToBEnW-Nwk 00yd4tKSFkWu9p63j7rH-PvNwwNXLzTyRWhi/pub
Language models We conduct annotations with three models M : GPT-3.5-turbo, GPT-4 and LLaMA-2-chat(70B) (Touvron et al., 2023), competitive commercial and public LLMs available at the time of data collection, respectively. For all models, we decode with temperature 0.7. During data collection, the identity of model M is not revealed to the annotator. For each question, we collect one interaction session with each model. We ensure that annotators do not answer the same question more than once.
Retrieval models We use the state-of-the-art dense retriever Contriever (Izacard et al., 2021) finetuned on MSMARCO (Campos et al., 2016) as our retriever R. In preliminary experiments, we found that Contriever outperformed sparse methods like BM25 (Robertson and Zaragoza, 2009), which struggle at handling synonyms.
Annotators To ensure our annotators have expertise in answering research questions, we recruit NLP researchers through Upwork8 and professional networks. We pay the annotators USD $25 per hour on average. The annotators first participate in a paid pilot study, which involves reading the instructions and completing an annotation example. A total of 22 annotators participated in our pilot annotations and 15 of them performed final annotations for KIWI. The final group of annotators consists of four people with a Ph.D. degree, seven NLP Ph.D. students, one person with a Master’s degree and three undergraduate students. All annotators have at least one year of NLP research experience. On average, each interaction session required 15 minutes to complete. To ensure that the annotators are familiar with the subject matter of the question, we first collect their preferences on a set of topics derived from the track information of each question’s source paper. We assign questions to annotators based on their preferences. We randomly assign models to each annotator and hide the model information from the annotator. The annotations were collected from July to December 2023. In total, our annotations cost $2,240 USD.
4 Analysis of KIWI
We first present results on overall model performance (§4.1). Then we conduct a fine-grained analysis to understand the different types of instructions (§4.2) and errors made by LLMs (§4.3).
8https://www.upwork.com/
Model # turn (edited) avg. # t/s % info/style

heading: 4.1 Dataset overview
text: We present statistics of the interaction sessions in Table 2. We observe that the number of turns required to reach a satisfactory answer negatively correlates with model capability, with gpt-4 requiring the fewest turns and llama2 requiring the most. Comparing instruction types, around 60% of instructions are information-seeking for gpt-4 , and 45% for llama2 ; this suggests that the gpt-4 is able to more quickly generate stylistically acceptable text, allowing users to focus on information content. We also observe that users tend to issue information-seeking instructions early in the interaction session followed by stylistic instructions (Figure 2). This confirms that collecting instructions over multiple turns allows us to cover diverse instructions occurring in different writing stages.
Type % info/style % good Definition Example Instruction
More information (43%)
88/12 gpt-4 : 57 Asking for auxiliary information
related to the question or the answer (e.g. example, results, limitation).
How well do state-of-the-art models perform on these datasets?
gpt-3.5 : 45 llama2 : 34
Expand coverage (9%)
86/14 gpt-4 : 40 Asking for more information directly
answering the question, usually from multiple papers. Keep the text content as it is, but extract examples of applications of contrastive explanations to NLP from papers 0,1, and 2. gpt-3.5 : 31 llama2 : 12
Clarification (5%) 69/31 gpt-4 : 52
Asking for clarification of a concept already in the answer.
Do all of the quantization methods you mention require additional training?
gpt-3.5 : 50 llama2 : 50
Remove (13%) 4/96 gpt-4 : 77
Requests to remove certain part of the answer. Remove the repeated techniques listed in the last paragraph.
gpt-3.5 : 28 llama2 : 16
Specific edit (6%) 23/77 gpt-4 : 59 Requests to directly add information
that are provided in the instruction verbatim to the answer.
At the end of the final paragraph, add the sentence "Evaluation of factuality for natural text remain active research lines." gpt-3.5 : 35 llama2 : 19
Condense (10%) 4/96 gpt-4 : 63
Requests to shorten or summarize the answer. Please shorten the answer to a maximum of 6 paragraphs.
gpt-3.5 : 27 llama2 : 23
Reorganize (11%) 13/87 gpt-4 : 74
Requests to reorganize the answer (e.g. move things around) Reorganize the answer so that positive results are described first, followed by challenges and open problems. gpt-3.5 : 27 llama2 : 34
Others (3%) 53/47 gpt-4 : 62
Edits that don’t belong to any of the category above.
Don’t change anything further about this answer but copy it as-is.
gpt-3.5 : 23 llama2 : 29
Table 3: Definitions and example instructions for the fine-grained instruction types (§4.2). We report the distribution classified by GPT-4 and % good rating (color-coded based on the % good) across the three models.
KIWI presents a challenge for existing LLMs: The distribution over user ratings of model responses is shown in Figure 2. gpt-4 exhibits the strongest instruction-following performance (59% good responses), while llama2 exhibits the weakest (27% good responses). Interestingly, trends for the two types of instruction differ—while gpt-4 provides good responses to stylistic instructions around 70% of the time, it struggles at following information seeking instructions (i.e. adding new information to an answer). In contrast, llama2 performs extremely poorly at following stylistic feedback, with only 15% of responses rated as good. All models fail to follow user instructions in more than 40% of responses, leaving significant headroom for LLMs’ instruction-following ability.

heading: 4.2 Fine-grained instruction analysis
text: Based on our findings that LLMs struggle at following the instructions in KIWI, we conduct a finegrained analysis to categorize the different types of instructions found in the dataset, and measure model performance on each. We manually label a sampled set of instructions, finding eight categories, and then scale up the analysis using GPT-4. We construct a prompt containing an example of [(orig-
inal question, instruction) → fine-grained instruction type] for each instruction type, and prompt GPT-4 to label the entire dataset. Details about the prompt and human analysis can be found in the Appendix A.3. We validate our approach by confirming that GPT-4 achieves an agreement of 0.96 Cohen’s kappa (Cohen, 1960) with humans on a held-out set of 40 instructions.
KIWI contains diverse instruction types: The results of our analysis are shown in Table 3. Although we did not reference coarse-grained category labels when defining the fine-grained categories, we observe that our resulting fine-grained categories can be divided based on our distinction between information-seeking and stylistic instructions. Information-seeking instructions include requests to provide auxiliary information, expand answer coverage, or provide clarification. Stylistic instructions range from lower-level requests such as performing a specific edit, to more complicated operations such as condensing and reorganizing the answer. We report additional statistics for each category in Table 6 in the Appendix A.3.
Models struggle at integrating new information and precise editing: Among the information
seeking instructions, “expand coverage” is the most difficult, as this type of instruction requires both successfully retrieving relevant passages from multiple papers and coherently integrating these new information into the previous answer. For stylistic instructions, gpt-4 performs well at removal and reorganization, achieving more than 70% good ratings, while gpt-3.5 and llama2 perform poorly. Interestingly, all three models struggle to make specific edits (e.g. add a requested sentence verbatim), which are usually trivial for humans. While prior work has found that LLMs excel at single document text summarization (Goyal et al., 2022), we observe poor performance for the “condense” instructions (which request to shorten previous answer), with gpt-4 ’s response rated as good for only 63% of the time. We hypothesize that this is because answers in our task involve information from multiple papers, and multi-document summarization remains a challenge for current LLMs (Shaib et al., 2023; Martin-Boyle et al., 2024).

heading: 4.3 Error analysis
text: To understand how models fail to follow instructions, we analyze the free-form comments ct written by the annotators. As in §4.2, we manually label a small set of comments and scale up with GPT-4 to all 760 turns which received a neutral or bad rating. We construct a prompt containing [(instruction, comment → target category)] demonstrations for each fine-grained error category; see Appendix A.4 for details of human analysis and the exact prompt. On a validation set with 22 examples, GPT-4 achieves achieves an agreement of 0.72 Cohen’s kappa with humans.
We find five major categories (Table 4). The most common error type is Unrequested change, indicating that models fail to maintain answer consistency across turns. Models also fail to satisfy hard constraints, such as the location (“At the beginning of the answer...”) and length of the information (“Add one sentence about...”) specified in the instruction. This suggests that current models struggle to perform precise actions, in agreement with findings from recent work (Sun et al., 2023; Yao et al., 2024) on controlled generation. Finally, models struggle to coherently integrate new information into the existing answer text, often leading to imbalanced structure or awkward answer flow.
Retrieval analysis The updated answer could fail to follow the instruction due to failure of the
retriever instead of of the LM. We randomly sample 20 turns with info-seeking instructions that are rated as neutral or bad and manually examine retrieval performance. We find that for 50% of the turns, relevant passages are retrieved which contain sufficient information to follow the instruction, yet the language models fail to integrate the information into the previous answer. For 20% of turns, some (but not all) relevant passages are retrieved. For the remaining 30%, the retrieval system fails to retrieve any relevant information. This demonstrates room for improvement in both components.

heading: 5 Experiments: Automatically evaluating instruction following
text: Our analysis so far demonstrates that current LLMs often fail to follow users’ instructions to revise long-form answers. Next, we examine whether they can serve as an evaluator to assess whether a model-generated answer followed the instructions (Dubois et al., 2023; Chiang and Lee, 2023; Zeng et al., 2023; Zheng et al., 2023b; Liu et al., 2023).
Setting and data Formally, the task is to predict the user rating rt = r(q, it, at−1, at), introduced in §2.2. To simplify the task, we collapse {neutral,
bad} into a single label.9 We randomly split the 1,260 turns into train/dev/test set in a 70%, 15%, 15% ratio. We report aggregated results on three different random splits.
Metrics and baselines We evaluate using Accuracy, Precision, Recall and F1 against the collected human labels rt, with the good label as the positive class. We report two baselines: a Random baseline which randomly assigns a label according to the training data distribution, and a Majority baseline which always chooses the majority class in the training data. We measure Human agreement by collecting a second set of human ratings for 65 randomly-sampled instructions, which reaches a moderate to substantial agreement (Cohen’s Kappa of 0.59) with the original rating.10 We report the average and standard deviation for human performance obtained through bootstrapping.
Model We experiment with zero-shot and fewshot prompting with gpt-4 and a fine-tuned T5large (770M) (Raffel et al., 2019) model. For zeroshot prompting, we construct an instruction which specifies the criteria for the two ratings. Inspired by prior work on retrieving in-context examples (Rubin et al., 2022), we retrieve the k turns from the training data whose instructions have the highest BM25 similarity with the test instruction. See Appendix A.5 for details.
Results We report results on the test set in Table 5. GPT-4 zero-shot performs slightly worse than choosing the majority label; we found that it is biased heavily towards judging answers as good. Adding 10 in-context examples improves perfor-
9We report performance on three-way prediction in §A.5.3; the same conclusions hold as for two-way prediction.
10Two authors and two annotators from the interaction annotation performed the annotation. We ensure the interaction annotators do not re-annotate their own previous interactions.
mance, while leaving a substantial gap relative to human agreement. Finetuned T5 only achieves similar accuracy with zero-shot gpt-4 . While recent work found that LLMs can rate responses as reliably as human (Chiang and Lee, 2023; Zheng et al., 2023b), our experiments show that they are not reliable for judging responses for specific instructions (Zeng et al., 2023; Liu et al., 2023).

heading: 6 Related Work
text: Instruction following A number of recent efforts have collected instruction following datasets to train and evaluate LLMs. Some datasets contain crowd-sourced interactions in the wild (Taori et al., 2023; Conover et al., 2023; Kopf et al., 2023; Zhao et al., 2023; Zheng et al., 2023a), while others target specific tasks such as summarization (Liu et al., 2023) and controllable generation (Zhou et al., 2023). Our work focuses specifically on instruction-following for writing assistance and text editing. A number of prior works have curated instruction-based editing benchmarks (Dwivedi-Yu et al., 2022; Shu et al., 2023; Raheja et al., 2023; Zhang et al., 2023). However, these efforts consist of post-hoc synthetic instructions derived from previous datasets, consisting of a limited set of edit instructions. Our data consists of diverse instructions issued by expert users interacting with LLMs.
LLM-based Evaluation Recent works have investigated using an LLM as an evaluator in place of human evaluation (Chiang and Lee, 2023; Zheng et al., 2023b; Li et al., 2023) for instruction following. A line of work shows that LLMs can judge responses as reliably as human annotators (Zheng et al., 2023b; Li et al., 2023). Recent work (Zeng et al., 2023) points out that evaluating open-ended instruction following exhibits high subjectivity and thus low human performance. The authors curate a set of instructions that are objective and craft adversarial responses to stress test LLMs, finding that the models struggle at evaluating responses for such instructions while humans exhibit high agreement. Our study also shows that LLMs struggle at evaluating responses for precise instructions collected in a realistic setting, complementing prior work.
Domain-specific datasets Prior work has introduced datasets to evaluate language models in specific domains, including law (Shen et al., 2022; Niklaus et al., 2023; Guha et al., 2023), medicine (Jin et al., 2019; Fleming et al., 2023), and com-
puter science research papers (Dasigi et al., 2021; Lee et al., 2023). While most prior work focuses on addressing a specific information need (e.g. through task formulations such as question answering or summarization), our setup more generally evaluates a model’s ability to closely follow user instructions, which can encompass many of these information needs.

heading: 7 Conclusion
text: We present KIWI, a dataset with expert-issued writing instructions to revise a long-form answer to a research question using relevant documents. Our analysis and experiments show that current LLMs (including GPT-4) cannot perform this task reliably yet and identify common failure patterns. We are optimistic that KIWI will be a useful resource for several research directions. First, the model revisions that were judged as successful by humans, and answers edited manually by humans when models failed, can be used as training data to improve models’ instruction following abilities. Second, human judgments of model revision quality can be used to develop more accurate reward models for writing assistance. Finally, the instructions in KIWI can be used as inputs to evaluate the performance of future models; either through human judgments, or by future reward models.

heading: Acknowledgments
text: We thank Nicholas Behrje, Sergey Feldman, Sebastian Joseph, Yoonjoo Lee, Clara Na, Matthias Rémy, Prasann Singhal, Zayne Sprague, Irina Temnikova, Albert Yu and other annotators for participation in creating the dataset. We thank the Semantic Scholar team and UT Austin NLP lab for helpful discussion and piloting the annotations. We thank Pradeep Dasigi, Sachin Kumar, and Leo Zeyu Liu for feedback to improve the paper draft.
Author Contributions: Fangyuan Xu led the project and performed all the technical contributions, including designing and building the annotation interface, dataset collection, running the experiments and performing the data analysis. Fangyuan also contributed to project scoping and ideation and led the writing of the paper. Bailey Kuehl helped with collecting the interaction data, including piloting the task, giving feedback and hiring and managing the Upwork annotators. David Wadden, Kyle Lo and Luca Soldaini were mentors of
the project during and after Fangyuan’s internship, and contributed to project scoping, experiments design, ideation and direction throughout the project. David Wadden and Kyle Lo contributed to paper writing and Luca Soldaini contributed to annotation interface development. Eunsol Choi provided mentorship throughout the project, including project scoping, ideation, experiement design and paper writing. Fangyuan, Bailey and David contributed to the question annotation.
Ethics Statement
We study the instruction following capabilities of commercial and open-sourced large language models. We recruit domain experts to interact with the language models and collect the instructions issued, the language model’s response and annotator’s evaluation of the response. Our annotation protocol has been determined to be exempted from review by an IRB board at our institution. We carefully examine the collected data to remove any personal identifiers and make sure there is no harmful content. Overall, we do not foresee any major risks or negative societal impacts of our work. We hope that our work can provide insight on the strengths and weaknesses of widely-adopted large language models in the domain that we study. We open-source our data collected to facilitate future research.
Limitations
Our study focus on the domain of writing answers for scientific questions and more specifically, NLP research questions. While we believe some of our findings could generalize to other domains such as creative writing, our data might only represent a subset of instructions that occur in other writing tasks or domain. Our annotators are NLP practitioners, who might be more familiar at interacting with the language model. While this setting allows us to better focus on the actual ability gap in current models (instead of annotators’ improficiency), our data thus might not reflect the full spectrum of interactions between different users and an LLM. Our human rating focuses on whether the revised answer follows the instruction to alleviate annotation burden, and future work should look into the factuality of the revised answer. Finally, We only cover the English language, and future work might collect similar data for other languages or look into leveraging our data for cross-lingual transfer.

heading: A Appendix
text: A.1 Implementation details for the interaction system
We implement the annotation interface with gradio11. Figure 4 presents a screenshot of our annotation interface. We use the OpenAI API12 for gpt-family model inference and the Together AI API 13 for llama2. Below are the prompt templates we use.
Prompt for generating initial answer We include the top 5 retrieved passages from the document set to generate the initial answer. title_i and passage_i refers to the title and text of the i-th retrieved passages using question as the query.
SYSTEM MESSAGE: You are a helpful assistant which answers a question based on a given set of documents. Please add reference (e.g. [0]) to the document in the answer.
USER MESSAGE: Documents:
[0] Paper title: title_0
Passage: passage_0
[1] Paper title: title_1
Passage: passage_1
[2] Paper title: title_2
Passage: passage_2
[3] Paper title: title_3
Passage: passage_3
[4] Paper title: title_4
Passage: passage_4
Question: question Answer:
Prompt for answer revision of information seeking instructions We first prompt the model to generate a summary st given the previous answer (at−1), the retrieved passages and the instruction it.
SYSTEM MESSAGE: You are a helpful assistant which generates an
11https://www.gradio.app/ 12https://platform.openai.com/docs/api-referen
ce 13https://docs.together.ai/reference/inference
intermediate answer given a feedback to the previous answer. Please add reference (e.g. [0]) to the document in the answer.
USER MESSAGE: Documents:
[0] Paper title: title_0
Passage: passage_0
[1] Paper title: title_1
Passage: passage_1
[2] Paper title: title_2
Passage: passage_2
[3] Paper title: title_3
Passage: passage_3
[4] Paper title: title_4
Passage: passage_4
Previous answer:at−1
Feedback:it
Generate an answer to the feedback.
We then prompt the model to generate an updated answer, given the original question, the answer in the previous turn at−1, the instruction it and the summary st. For llama2 , we add an additional instruction at the end “Do not generate “Sure”, directly generate the updated answer. Updated answer: ”.
SYSTEM MESSAGE: You are a helpful assistant which generates an intermediate answer given a feedback to the previous answer. Please add reference (e.g. [0]) to the document in the answer.
USER MESSAGE: Original question: question
Previous answer: at−1
Feedback: it
Extra information related to the feedback: st
Answer the original question and incorporate the feedback to the previous answer. Preserve all relevant information from previous answer.
Prompt for answer revision of stylistic instructions We include the original question, answer in the previous turn (at−1) and the instruction it in the prompt. For llama2 , we add an additional instruction at the end “Do not generate “Sure”, directly generate the updated answer. Updated answer: ”.
SYSTEM MESSAGE: You are a helpful assistant which answers a question based on a given set of documents. Please add reference (e.g. [0]) to the document in the answer.
USER MESSAGE: Original question: question
Previous answer: at−1
Feedback: it
Answer the original question and incorporate the feedback to the previous answer. Preserve all relevant information from previous answer.
Prompt length On average, the prompt for generating an answer revision for information seeking instructions (with question, previous answer, instruction and retrieved passages) consists of 607 words. The retrieved passages consists of 560 words on average. The prompt for answer revision for stylistic instructions (with question, previous answer and instruction) consists of 458 words on average.
Retrieval unit We use S2ORC to parse the paper into paragraphs and retrieve from the set of paragraphs in the relevant papers D.
A.2 Examples
Examples turns with good, neutral and bad rating can be found in Table 9. Example of questions annotated (§3.1) can be found in Table 8.
A.3 Implementation details for automatic instruction analysis with GPT-4
Manual analysis We randomly sample 50 interaction turns and manually group the instructions into fine-grained categories. We discover 8 finegrained categories: requesting for more information (46%); expanding answer coverage (6%); asking for clarification (6%), categorization (6%), removal (10%), specific edits (2%), condense (12%) and reorganization (10%).
GPT-4 analysis We construct a prompt with definition and in-context examples for each of the category. The prompt we use to perform automatic instruction analysis in in Table 11. We prompt gpt-4 with temperature of 0 and top_p=1. The distribution classified by GPT-4 is presented in 3. We see a slight distribution differences between the large-scale GPT-4 analysis and small-scale manual analysis, with GPT-4 identifying more instructions as “specific edits” and less as “categorization”. We merge categories with fewer than 5% of examples into “Others”.
More statistics We report additional statistics for each instruction type in Table 6. We filter out turns where the response is rated as neutral or bad with an edited answer, resulting in a total of 815 turns. Among the coarse-grained instruction types, we see that stylistic instructions are longer, yet with less edit ratio compared to information seeking ones. Specific edit instructions are the longest, while instructions asking for more information requires the most amount of edits to the previous answer (highest edit ratio).
A.4 Implementation details for error analysis Manual analysis We randomly sample 10 interaction turns per model which are rated as neutral or bad, resulting in 30 total turns. We then group them into 5 fine-grained categories based on the question and comment.
GPT-4 analysis The prompt we use to perform automatic instruction analysis is in Table 12. We prompt gpt-4 with temperature of 0 and top_p=1.
A.5 Implementation details for automatic evaluation of instruction following
A.5.1 GPT-4 We prompt gpt-4-1106-preview 14 for this task, whose context window can fit the ten-shot examples. We decode with temperature of 0 and top_p=1. Below are the prompt templates for the different baselines.
Zero-shot prompt template The model is given an instruction, the question, the answer in the previous turn at−1, the instruction it and the answer in the current turn at.
SYSTEM MESSAGE: You are a helpful assistant in evaluating the quality of
14https://platform.openai.com/docs/models
the outputs for a given instruction to update an answer for an question. Your goal is to score a given updated answer for the given instruction.
USER MESSAGE: Score the updated answer for the given instruction by comparing it with the original answer.You should give one of the two ratings: good, or bad.
Give a bad rating if the updated answer either (1) only partially followed the instruction (for example, the instruction specify an edit of a certain length or at a certain location but the updated answer didn’t follow the constraint) or (2) introduced changes that are not mentioned in the instruction compared to the original answer or (3) provided a vague answer (e.g. without naming actual method) or (4) added the requested information but made the answer less coherent/correct.
Only give a good rating if the answer COMPLETELY followed the instruction and didn’t have ANY issues mentioned above. Your response should be ONLY the ratings.
# Question:
# question
# Original answer:
# at−1
# Instruction:
# it
# Updated answer:
# at
# Rating of the Updated answer:
#
Few-shot prompt template Each few-shot example j contains the question, the answer in the previous turn, the instruction, the answer in the current turn and the rating of the answer.
SYSTEM MESSAGE: You are a helpful assistant in evaluating the quality of
Type # words Len ratio Edit distance Edit ratioinst src target
More information 17 227 340 1.60 162 0.85 Expand coverage 39 295 398 1.48 195 0.78 Clarification 27 287 324 1.23 137 0.58 Remove 24 396 339 0.87 91.31 0.24 Specific edit 75 278 325 1.22 93 0.38 Condense 15 388 234 0.64 213 0.55 Reorganize 26 391 339 0.87 91.31 0.24 Others 23 327 342 1.36 212 0.88
Stylistic 29 361 316 0.94 141 0.42 Info seeking 22 259 355 1.60 176 0.87
GPT-4 24 260 286 1.28 145 0.70 GPT-3.5 23 289 332 1.34 145 0.68 Llama 29 379 387 1.20 173 0.56
Total 26 310 336 1.27 158 0.64
tion for instruction following edits. stands for gpt-4 . We report the % of good, neutral and bad ratings in the model predictions in the last column.
A.5.2 T5
We finetune the model to output target sequence “Rating : ri” with input sequence “Original question: q Previous answer: at−1 Instruction: it Updated answer: at”, where ri is the traget binary rating, q is the question, at−1 is the answer in the previous turn, it is the instruction and at is the answer in the current turn. We use a batch size of 16 and an initial learning rate of 1e-4 with Adam optimizer and a linear learning rate schedule. We train the model for 5 epochs and choose the checkpoint with the best validation accuracy. The hyperparameters are manually searched by the authors.
A.5.3 Additional experiments on three-way rating prediction
We report results from experiments on three-way rating prediction. The experimental setup (i.e. data splits, hyperparameters, etc.) is the same as §5, except that the task is to predict among {good, neutral, and bad}, instead of predicting between {good and bad}. We report macro precision, recall, F1 among the three classes.
Results Results are reported in Table 7. We observe a similar trend as the results for binary rating prediction (Table 5). While gpt-4 zeroshot is biased towards judging answers as good, adding ten-shot in-context examples boosts the F1 score from 40 to 48. Fine-tuned T5 achieves better F1 compared to zero-shot prompting but underperforms ten-shot prompting. Here again, we see that there is a substantial gap between human performance and model performance, suggesting that LLMs are not yet a reliable evaluator for this task.
A.6 Analysis on question generation with GPT-4
During the initial phase of the project, we explored an automatic version of the pipeline described in §3.1 by using GPT-4 to generate questions. We describe the pipelines below:
Step 1: Extracting and filtering related works We extract related work paragraphs from papers published in NLP venues (ACL, EMNLP, NAACL, etc.) using S2ORC (Lo et al., 2020). We extract the papers cited and filter out paragraphs with more than 30% of the papers missing from S2ORC or with less than 4 cited papers available. This gives us an initial set of (q, D).
Step 2: Prompting LMs to generate questions For each related work paragraph, we prompt GPT-4 to generate five questions using the below prompt. We set the temperature to be 1 and top_p=1. paper_title is the title of the paper from which the related work paragraph is extracted.
SYSTEM MESSAGE: You are a helpful assistant which generates five questions that the paragraph is addressing. The question should require multiple sentences to answer. Don’t ask multiple sub questions in a single question. Don’t refer to specific paper in the question. USER MESSAGE: Paper title: paper_title Passage: related work paragraph
Step 3: Question filtering We employ two filtering steps to filter out questions that do not fulfill the desiterata that we described in §3.1. First, we use a heuristic rule to filter out questions that contain keywords which make the question anchored to specific papers (e.g. “this paper”, “mentioned”, “author”). This process filters out 23% of the questions generated.
We then prompt GPT-4 on whether the generated question (question) can be answered by the related work paragraph using the following prompt. This process filters out 12% of the question which are rated as No or Paratially by GPT-4.
SYSTEM MESSAGE: You are a helpful assistant. USER MESSAGE: Check if the
paragraph answers the question. Reply Yes, No or Partially.
Question: question
Paragraph: related work paragraph
Manual examination We conduct a manual examination on the quality of the (q,D) pairs generated through this automatic pipeline. We look at both the question and an initial answer generated by GPT-4 given q,D, using the prompt described in §A.1. Two of the authors randomly sample 50 (question, initial answer) pairs to examine both the quality of the question and the initial answer.
We found that only 60% of the questions fulfill our desiderata described in §3.1. 28% of the question do not make sense to a researcher (e.g. “How do previous approaches to multi-task learning and domain adaptation try to create a universal representation space using encoders and architectures?”), 6% of the questions are too specific or niche (e.g. “What are the primary components used by the VGVAE model to represent the semantics and syntax of a sentence in a monolingual setting?”) and 6% of the questions do not stand alone (e.g. “How do labels like "other" or "information providing" act indicate issues with these models?”). For the questions that fulfill our desiderata, GPT-4 is unable to generate a good initial answer for 26% of them, often due to retrieval failure (e.g. “The documents provided do not discuss the potential issues with using word segmentation information and dependency trees for Chinese Named Entity Recognition (NER) as extra features.”). We note that this is not necessarily an issue with the retrieval system, rather a potential indication that the question is irrelevant to the papers cited.
Overall, we find that this pipeline does not produce high quality (q, D) pairs needed for KIWI. Hence we proceed with human annotations as described in §3.1.
Long-form QA Interaction Study Imagine you are an NLP researcher and your colleague has asked you for a paragraph-length answer about a research topic of interest. You will work with a large language model to generate an answer to send to the colleague. You will first see the question, an initial answer generated by the model, and a set of relevant papers, based on which the model generates answers. Your task is to provide feedback for improving the generated answer through several rounds of interaction. Detailed instructions can be found here
This process can last up to 10 interactions, or you can end it early by clicking “✉ Ready to Send” if you find the answer satisfactory.
Question: How are features of text and images fused in multimodal summarization models?
Answer:
⏳Current turn: 1
You are given an instruction to improve an answer to a question. Your job is to assign the instruction into one of the category below.
Example 1 # Original question: Are there any datasets available fro table-to-text generation tasks in NLP? # Instruction: How well do state-of-the-art models perform on these datasets? # Category: asking for more information # Explanation: This category of instruction ask for more information related to the question (e.g. example, limitation, evaluation results, background information, citations, definition).
Example 2 # Original question: How has prior work study and evaluate robustness of multimodal models? # Instruction: As first sentence of the text, add a very short one sentence definition of what are "multimodal models" # Category: asking for more information # Explanation: This category of instruction ask for more information related to the question (e.g. example, limitation, evaluation results, background information, citations, definition).
Example 3 # Original question: How did prior work explore learning from human feedback for different NLP tasks? # Instruction: Try to add very short information also from papers 0,1,3,5,6,7,8. Keep the rest of the text exactly as it is. # Category: expand answer coverage # Explanation: This category of instruction asks for more information directly answering the question, usually from multiple papers.
Example 4 # Original question: What are the existing approaches for the lip-reading task? # Instruction: You mention that some methods use a network to predict phoneme probabilities and were trained using CTC loss and that it was trained using YouTube videos. Does this mean that such methods require phoneme level labels for the data to be trained, or does word level transcriptions work? # Category: asking for clarification # Explanation: This category of instruction asks for clarification of a concept already in the answer.
Example 5 # Original question: What are the prior efforts that develop RL environments with language-informed tasks? # Instruction: Organize the papers better in categories of what aspect of language-conditioned RL tasks they are tackling. # Category: categorization # Explanation: This category of instruction requests to categorize the content in the answer.
Example 6 # Original question: How can information stored in knowledge bases be injected into language models task-specific model fine-tuning? # Instruction: Please shorten the answer to a maximum of 6 paragraphs. # Category: condense # Explanation: This category of instruction requests to shorten/summarize the answer.
Example 7 # Original question: How can I quantize the weights of a generative language model and still achieve reasonable model performance? # Instruction: Without changing anything else, delete every instance of the string "(Feedback)", and delete the string "[1] “Compression of Generative Pre-trained Language Models via Quantization”" at the end of the answer. # Category: remove # Explanation: This category of instruction requests to remove/delete certain part of the answer.
Example 8 # Original question: What techniques have been proposed to measure and evaluate the effectiveness of task-oriented dialogue systems? # Instruction: Move the paragraph starting with "Finally, there are also various task-specific evaluation metrics" to after the discussion of user satisfaction modeling, and make sure to cite paper [2] in that paragraph. # Category: reorganize # Explanation: This category of instruction requests to reorganize the answer(e.g. move things around, etc.).
Example 9 # Original question: What methods have been proposed to categorize the kinds of editing operations that occur during text simplification? # Instruction: Immediately after this sentence: "In the process of text simplification, different text-editing operations are used. " add this sentence "The editing operations that occur during text simplification can be usually categorized after doing literature review and reading previous publications on text simplification." # Category: direct verbatim edit # Explanation: This category of instruction requests to directly add information that are provided in the instruction verbatim to the answer.
Example 10 # Original question: How have contrastive learning techniques been applied to learn dense sentence representations? # Instruction: paragraph 3 does not describe another approach, # Category: others # Explanation: This category of instruction doesn’t belong to any of the category above.
Now, assign a category for the example below. ONLY ASSIGN ONE OF THE CATEGORY ABOVE. DONOT INVENT YOUR OWN CATEGORY. DON’T PROVIDE AN EXPLANATION.
Table 11: Prompt for automatic instruction analysis.
You are given an instruction for a model to improve an answer to a question and a comment on how the updated answer followed the instruction. Your job is to assign the comment into one of the category below.
Example 1 # Instruction: Now add this information: ""[2] introduce a controllable summarization model that provides a mechanism for users to specify high level attributes such as length, style, or entities of interest. This enables personalized generation."" # Comment: It added the information, but in the wrong place, so it looks as an extension of another method. # Category: added information but made answer less coherent/correct # Explanation: This category of comment will mention that the instruction is followed, but the updated answer is less coherent (for example, become too long; has a weird structure, or is not correct). In the example, the comment mentioned that the updated answer has the information added but the updated answer is confusing.
Example 2 # Instruction: Remove the tenth sentence. # Comment: The sentence was not removed as requested. # Category: instruction is ignored # Explanation: This category of comment will mention that the instruction is not followed at all. In this example, the comment mentioned that requested change was not performed.
Example 3 # Instruction: As a next sentence after this one "These parameters include the weights and biases of the model’s layers, such as the embedding layer, encoder, decoder, and attention mechanisms [0]." add a short sentence, which explains why parameters sharing is necessary in multilingual models used for machine translation # Comment: It added an explanation, but deleted a whole paragraph # Category: introduced unrequested changes # Explanation: This category of comment will mention that the instruction is followed but also there are unrequested changes performed. In this example, the comment mentioned that the requested change (explanation) was incorporated but it also introduced unrequested change (a whole paragraph was deleted).
Example 4 # Instruction: Great answer! Without changing anything else, in the last paragraph, mention some tasks that benefit from using explanations, and other tasks where it impedes performance. # Comment: The answer is overall still good but the new material is too long, and the model disregarded the part of the prompt where it was asked to put this material in the last paragraph. I will try to re-prompt. # Category: failed to follow hard constraint # Explanation: This category of comment will mention one ore more hard constraint in the instruction is not followed by the model. The hard constraint can be a specific location (at the beginning of the answer), length of the added/final answer (a very short definition), or an action to avoid ("do not..."). In this example, the comment mentioned that the updated answer didn’t follow the constraint in the instruction (which mention the added information should be in the last paragraph).
Example 5 # Instruction: I notice a lot of redundant sentences, such as repeated mention of something like "compress the model while maintaining performance." Can you make your answer more concise? # Comment: model mostly just added paragraphs together without cutting actual content # Category: others # Explanation: This category of comment will mention that the model tried to follow the instruction (thus not completely ignore the instruction) but the updated answer is not satisfactory. The updated answer doesn’t has issue mentioned above (unrequested change or failed to follow hard constraint), but just in general has subpar quality.
Now, assign a category for the example below. ONLY ASSIGN ONE OF THE CATEGORY ABOVE. DONOT INVENT YOUR OWN CATEGORY. DON’T PROVIDE AN EXPLANATION.
Table 12: Prompt for automatic comment analysis.

heading: Future Work
text: We open-source our data collected to facilitate future research. Limitations
Our study focus on the domain of writing answers for scientific questions and more specifically, NLP research questions. While we believe some of our findings could generalize to other domains such as creative writing, our data might only represent a subset of instructions that occur in other writing tasks or domain. Our annotators are NLP practitioners, who might be more familiar at interacting with the language model. While this setting allows us to better focus on the actual ability gap in current models (instead of annotators’ improficiency), our data thus might not reflect the full spectrum of interactions between different users and an LLM. Our human rating focuses on whether the revised answer follows the instruction to alleviate annotation burden, and future work should look into the factuality of the revised answer. Finally, We only cover the English language, and future work might collect similar data for other languages or look into leveraging our data for cross-lingual transfer