A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
Authors: Benjamin Newman
Luca Soldaini
Raymond Fok
Arman Cohan
Kyle Lo
Venue: EMNLP 2023 Main (2023)
Keywords: decontextualization
snippets
text-simplification
TL;DR: We propose a question-answering framework for decontextualizing snippets from scientific documents and investigate the performance of state-of-the-art LLMs under this framework.

Abstract: Many real-world applications (e.g., note taking, search) require extracting a sentence or paragraph from a document and showing that snippet to a human outside of the source document. Yet, users may find snippets difficult to understand as they lack context from the original document. In this work, we use language models to rewrite snippets from scientific documents to be read on their own. First, we define the requirements and challenges for this user-facing decontextualization task, such as clarifying where edits occur and handling references to other documents. Second, we propose a framework that decomposes the task into three stages: question generation, question answering, and rewriting. Using this framework, we collect gold decontextualizations from experienced scientific article readers. We then conduct a range of experiments across state-of-the-art commercial and open-source language models to identify how to best provide missing-but-relevant information to models for our task. Finally, we develop QaDecontext, a simple prompting strategy inspired by our framework that improves over end-to-end prompting. We conclude with analysis that finds, while rewriting is easy, question generation and answering remain challenging for today’s models.

A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
Benjamin Newman
Luca Soldaini Raymond Fok
Arman Cohan
Kyle Lo
Allen Institute for AI
University of Washington
Yale University kyl.mannckkllllelleenai..c
(benjaminn, rayfok)8cs. wawpkk/jsssswsssnnttoooooo wwwhington.eddee
Abstract
Citation Graph Explorer
Deep Recurrent Models with Fast... ot TTC. 2016
Many real-world applications (c.g,, note tak- ing, search) require extracting sentence paragraph from a document and showing that snippet to a human outside of the source doc- ument. Yet, users may find snippets difficult to understand as they lack context from the original document. this work, we use lan- guage models to rewrite snippets from scien- tific documents to read on their own. First, we define the requirements and challenges for this user-facing decontextualizatioon task, such as clarifying where edits occur and handling references to other documents. Second, pro- pose a framework that decomposes the task into three stages: question generation, question an- swering, and rewriting. Using this framework, we collect gold decontextualizations from ex- perienced scientific article readers. We then conduct a range of experiments across state-of- the-art commercial and open-source language models identify how best provide missing- but-relevant information to models for our task. Finally, we develop QADECON,,,,, simple prompting strategy inspired by our framework that improves over end-to-end prompting. We conclude with analysis that finds, while rewrit- ing is easy, question generation and answering remain challenging for today's 's models.
Our [Transformer model] outperfoms prior state- af-the-art (Zhou al., 2016) [which used LSTMs for machine translation]...
from Attention
Al Research Assistant
Describe the features used in Bag What... by Handler et al., 2016
Bag words and part-of-speech features.
NPFST (a methad for enriching bag of words (BOW) with finito state transducer (FST) uses POS (part-f-speachh) tagger to extract..."
Figure I: Illustration of two user-facing scenarios requir- ing snippet decontextualizztton.. (Top) A citation graph explorer surfacing citation context snippets to explain relationships between papers. (Bottom) Al research assistant providing snippets attributions. Highlighted spans are added during decontextualization.
puts (Bohnet al., 2022) and provide means for user error recovery.
However, extracted snippets are not meant to be read outside their original document: they may include terms that were defined earlier, contain anaphora whose antecedents lie in previous para- graphs, and generally lack context that is needed for comprehension. At best, these issues make ex- tracted snippets difficult to read, and worst, they render the snippets misleading outside their orig- inal context (Lin et al., 2003; Cohan et al., 2015; Cohan and Goharian, 2017; Zhang et al., 2023).
gi thub. http://bwwennee////aaaeeeeeeeeee
1 Introduction
Tools to support research activities often rely on extracting text snippets from long, technical doc- uments and showing them to users. For example, snippets can help readers efficiently understand documents (August et al., 2023; Fok et al., 2023b) scaffold exploration of document collections (e.g. conducting literature review) (Kang al., 2022; Palani al., 2023). As more applications use language models, developers use extracted snip- pets to protect against generated inaccuracies; snip- pets can help users verify model-generatee out-
In this work, consider the potential for mak- ing extracted snippets more readily-understood in user-facing settings through decontextualiza- tion (Choi et al., 2021)-the task of rewriting snip- pets to incorporate information from their originat- ing contexts, thereby making them "stand alone".

---

We focus our attention on scenarios in which users read snippets from technical documents (e.g.. sci- entific articles). For example, consider citation graph explorer that allows users to preview cita- tion contexts to explain the relationship between pupers (Luu et al., 2021). Also, consider an AI research assistant that surfaces extracted attribution snippets alongside generated answers. Figure 1 il- lustrates these two motivating applications. How do language models fare when performing snippet decontextualivation over complex scientific text? Our contributions are:
edited snippet s' is valid decontextual- ization of if s' is interpretable without any additional context, and s' preserves the truth cruth-conditional meaning of s in c.
where the context i a representation of the source document, such as the full text of a scientific article.
Multi-sentence Passages. While Choi et al. (2021) restrict the scope of their work to single- sentence snippets, they recommend future work on longer snippets. Indeed, real-world applications should be equipped handle multi-sentence snip- pets as they are ubiquitous the datasets used develop such systems. For example, 41 % of evi- dence snippets in Dasigi et al.'s (2021) dataset and 17% of citation contexts in Lauscher et al.'s (2022) dataset are longer than a single sentence. To con- strain the scope of valid decontextualizations, we preserve (I) the same number of sentences in the snippet and (2) each constituent sentence's core informational content and discourse role within the larger snippet before and after editing.
First, we introduce requirements that extend prior decontextualization work (Choi et al., 2021) to handle user-facing scenarios (e.g.. delineation of model-generated edits). We characterize additional challenges posed by decontextualizing scientific documents (e.g., longer text, citations and refer- ences) and describe methods address them ($2). Second, we propose framework for snippet decontextualization that decomposes the task into three stages: question generation, question answer- ing, and rewriting ($3). This decomposition is mo- tivated by formative study in which our frame- work makes deconntetuullization less challenging and creates higher-quality annotations. We use this framework collect gold decontextualization data from experienced readers of scientific articles ($4).
Transparency of Edits. Prior work did not re- quire that decontextualization edits were transpar- ent. We argue that the clear delincation of machine- edited versus original text is requirement in user- facing scenarios such ours. Users must be able to determine the provenance (Han et al., 2022) and authenticity (Gchrmann al., 2019; Verma et al., 2023) of statements they read, especially the context of scientific research, and prior work has shown that humans have difficulty identifying machine- -eabbine-geerrteed text (Clark et al., 2021). In this work, we require the final decontextualized snip- pet s' to make transparent to users what text came from the original snippet and what text added, removed, modified. We ask tools for decontex- tualization follow well-established guidelines in writing around how to modify quotations!. Such guidelines include using square brackets ([J) to de- note resolved coreferences or newly incorporated information.
Finally, with this data, we operationalize our framework by implementing QADECONTEXT, a strategy for snippet decontextuallization ($5). Our best experimental configuration demonstrates 11.7% relative improvement over end-to-end model prompting ($5.2). We find that of-te- art language models perform poorly on our task, indicating significant opportunity for further NLP research. We perform extensive analysis to identify task bottlenecks to guide future investigation ($6).
2 Deconfextualization for User-facing Snippets from Scientific Documents
this section, we define decontextualization and motivate some additional task requirements when considering user-facing scenarios. Then, we de- scribe additional task challenges that arise when operating on scientific documents.
2.2 Challenges in Scientific Documents
We characterize challenges for decontextualization that arise when working with scientific papers.
2.1 Requirements for User-facing Snippets Task Definition. As introduced in Choi et al. (2021), decontextualization is defined as:
Long, Complex Documents. We present quan- titative and qualitative evidence of task difficulty compared to prior work Wikipedia snippets.
APA style guide: https://ppastyl.......n.stt.. quiddiineskiiaaii///uuattiaoonn
Given a snippet-context pair (s, c), an

---

First, Choi et al. (2021) found between 80-90% of the Wikipedia sentences can be decontextual- ized using only the paragraph with the snippet, and section and article titles. However, we find in our data collection ($4) that only 20% of snippets from scientific articles can be decontextualizee with this information alone (and still only 50% when also including the abstract; see Table 5).
What corpus did Bansal et al. use?
"We test our system on the CALL-
HOME Spanish-English speech transla- tion corpus ($3)."
One method of decontextualizatioon can be:
"[Bansal et al., 2017] test [their] system on the CALLHOME Spanish- English speech translation corpus [42] [ ["Improved speech-to-text translation with the Fisher and Callhome Spanish- English speech translation corpus" at IWSLT 2013] (83)."
Second, we conduct a formative study with five computer science researchers, asking them to manually decontextualize snippets taken from Wikipedia and scientific papers. Participants took between 30- 10-160 seconds (w=88) for Wikipedia sen- tences from Choi et al. (2021) and between 220- 390 seconds (μ=2999 for scientific snippets from our work. In qualitative feedback, all participants expressed the ease of decontextualizing Wikipedia snippets. For scientific paper snippets, all partici- pants verbally expressed difficulty of the task de- spite familiarity with the subject material; 3/5 par- ticipants began taking notes to keep track of rel- evant information; 4/5 participants felt they had read the paper title, abstract and introduction before approaching the snippet: and 4/5 partici- pants encountered cases of chaining in which the paper context relevant to an unfamiliar entity con- tained other unfamiliar entities that required fur- ther resolving. None of these challenges arose for Wikipedia snippets.
incorporating the title of cited paper "[42]". But in the case of a citation graph explorer, a typical interface likely already surfaces the titles of both citing and cited papers (recall Figure 1), in which case the addition of title isn't useful. Possibly preferred is an alternative decontextualization that describes the dataset:
D Baaasa et al., 2017] test [their] sys- the CALLHOME Spanish-Englist speech translation corpus [42] I, noisy multi sulti--peakkr corpus of telephone calls in a variety of Spanish dialects] ($3)."
2.3 Addressing Challenges
To address the increased task difficulty that comes with working with long, complex scientific doe- uments, we introduce a framework in (S3) and describe how it helps humans tackling this task manually. We also opt to remove all references to in-document tables and figures from snippets, and leave handling them to future work..
Within and Cross-Document References. Tech- nical documents contain references to within- document artifacts (e.g., figures, tables, sections) and to other documents (e.g., web pages, cited works). Within-document references are typically to tables, figures, or entire sections, which are dif- ficult to properly incorporate into a rewritten snip- pet without changing it substantially. With cross- document references, there is no single best way to handle these when performing decontextullization; in fact, the ideal decomtextualizzationn likely more dependent on the specific user- serr-fccng application's ss design rather than on intrinsic qualities of the snip- pet. For example, consider interacting with an AI research assistant that provides extracted snippets:
Finally, to handle cross-document references, assume in the AI research assistant application setting that user would have access basic infor- mation about the current document of interest but no knowledge about any referenced documents that may appear in the snippet text. Similarly, we as- sume in the citation context preview setting, that a user would have access to basic information about
"While numeric citations henefit substantially from inter- face assistance in surfacing information about the cited pa- per (Chang et al., 2023a), researchers may be able to recall details of cited papers from name-year citations like "(Post et al., 2013)", We define our target decontextualization behavior under the least-informative citation format.
"Participants were rescarchers with at least five pub- lished research papers at an NLP, ML or HCI venue. Four had com-
completed PhDs while one was a PhD program, puter science. All were familiar with the decontextualization task, unfamiliar with the goals of our study.
'Real-world systems currently only surface scientific paper snippets in text-only interfaces, though one can imagine future multimodal interfaces might have different task requirements for snippet decomtextualization.
'Each participant saw at least two snippets. To control for learning effects, we randomized assignment order of Wikipedia or Science.

---

the current (citing, cited) document pair but no knowledge about any other referenced documents that may appear in the snippet text.
to the framework results in higher-quality anno- tations in the scientific document setting, 3/5 of participants who were assigned QA first said that they preferred to follow the framework even in the NoQA setting". Two of them additionally noted this framework is similar to their existing note- taking practices. The remaining 2/5 of participants who were assigned NoQA first struggled initially; both left their snippets with unresolved acronyms or coreferences. When asked why they left them as-is, they both expressed that they lost track of all the aspects that needed decontextualizatton.. These annotation issues disappeared after these partici- pants transitioned to the QA setting, Overall, all participants agreed the framework was sensible to follow for scientific documents.
3 QA for Decontextualizatioon
Decontextualization requires resolving what addi- tional information a person would like to be in- corporated and how such information should be incorporated when rewriting (Choi et al., 2021). If we view "what" as addressed in our guidelines ($2), then we address "how" through this proposal:
3.1 Our Proposed Framework
We decompose deconicxtualization into three steps:
1. Question generation. Ask clarifying questions about the snippet.
2. Question answering. For each question, find an answer (and supporting evidence) within the source document.
Data Collection
Following the results of our formative study, we implemented an annotation protocol to collect de- contextualized snippets from scientific documents.
3. Rewriting. Rewrite the snippet by incorporat- ing information from these QA pairs.
We present arguments favor of this framework:
4.1 Sources of Snippets
QA and Discourse. Questions and answers are a natural articulation of the requisite context that ex- tracted snippets lack. The relationship between questions and discourse relations between doc- ument passages can be traced to Questions Un- der Discussion (QUD) (Onea, 2016; Velleman and Beaver, 2016; De Kuthy et al., 2018; Riester, 2019). Recent work has leveraged this idea cu- rate datasets for discourse coherence (Ko et al., 2020, 2022). We view dccontextualization as a task that aims to recover missing discourse infor- mation through the resolution of question-answer pairs that connect portions of the snippet to the source document.
We choose two tific documents as our source of snippets, one for each motivating application setting (Figure 1):
datasets scien-
Citation Graph Explorer. We obtain citation context snippets used in citation graph explorer from scientific papers S2ORC (Lo et al., 2020). We restrict to contexts containing a single ci- tation mention to simplify the annotation task, though we note that prior work has pointed out the prevalence of contexts containing multiple cita- tions? (Lauscher al., 2022).
AI Research Assistant. We use QASPER (Dasigi et al., 2021), a dataset for scientific document understanding that includes QA pairs along with document--roundded attributions exttrtttee pas- sages that support a given answer. We use these supporting passages user-facing snippets that require decontextualization..
Improved Annotation. In our formative study ($2.2), also presented participants with two dif- ferent annotation guidelines. Both defined decon- textualization, but one (QA) described the stages question generation and question answering as prerequisite hefore rewriting the snippet, while the other (NoQA) showed before-and-afie examples of snippets. All participants tried both guidelines; we randomized assignment order to control for learning effects.
4.2 Annotation Process
Following our proposed framework:
FFor these participants, their main complaint was the act of writing down questions and answers explicitly seemed slow. hut they agreed with the framework overall as intuitive. FFuture work could investigate decontextualization amid multiple outward references in the same snippet.
While we find adhering to the framework slows down annotation and does not impact annotation quality in the Wikipedia setting ($A.4), adhering

---

Writing Questions. Given a snippet, we ask an- notators write questions that clarify seek addi- tional information needed to fully understand the the annotation
5.1 Is end-to-end LLM prompting sufficient? Naively, one can approach this task by prompting a commercially-avaiaate language model with the instructions for the task, the snippet, and the entire contents of the source paper. We experiment with text-davinci-003 and S gpt-4-0314, For gpt-4, most papers entirely fit in the context window (for a small number of papers, we truncate them fit). For davinci, we represent the pa- per with the title, abstract, the paragraph containing the snippet, and the section header of section con- taining the snippet (if available). This choice was inspired by Choi et al.'s (2021) use of analogous information for decontextualizing Wikipedia text, and we empirically validated this configuration in our setting as well (see $A.3). We provide our prompts for both models in $A.6.4 and $A.6.5.
snippet. Given the complexity task we used Upwork to hire four domain experts with experience reading scientific articles. Annota- tors were paid $20 USD per hour".
Answering Questions. We hired a separate set of annotators to answer questions from the previous stage using the source document(s)) We addition- ally asked annotators to mark what evidence from the source document(s) supports their answer. We used the Prolific annotation platform as high- quality source for a larger number of annotators. Annotators were recruited from the US and UK and were paid $17 USD per hour. To ensure data quality, we manually filtered a total of 719 initial answers down to 487 by eliminating ones that an- swered the question incorrectly or found that the question could not be answered using the informa- tion in the paper(s) (taking ~0 hours).
For automated evaluation, we follow Choi et al. (2021) and use SART (Xu et al., 2016). Originally developed for text simplification, SARI is suitably repurposed for decomlextualization as it computes the F1 score between unigram edits to the snip- pet performed by the gold reference versus edits performed by the model. As are interested in whether the systems add the right clarifying in- formation during decontextualivattion, we report SARI-add as our performance metric. We addition- ally report BERTScore (Zhang et al.. 2020) which captures semantic similarity between gold refer- ence and model prediction, though it is only used as diagnostic tool and does not inform our evalua- tive decisions; due to the nature of the task, as long as model generations are reasonable, BERTScore will be high due to significant overlap between the source snippet, prediction and gold reference.
Rewriting Snippets. Given the original snippet and all QA pairs, we ask another set of annotators from Prolific to rewrite the snippet incorporating all information in the QA pairs.
4.3 Dataset Statisties
In total, we obtained 289 snippets (avg, 44.2 tokens long), 487 questions (avg. 7.8 tokens long), and 487 answers (avg. 20.7 tokens long). On average, the snippets from the Citation Graph Explorer set have 1.9 questions per snippet while the AT Research Assistant snippets have 1.3 questions per snippet. Questions were approximately evenly split between seeking definitions of terms, resolving coreferences, and generally seeking more context feel informed. See $A.2 for breakdown of question types asked by annotators.
We report these results Table 1. Overall, we find that naively prompting LLMs end-to-end performs poorly on this task.
5.2
Can our QA framework inform an improved prompting strategy?
5 Experimenting with LLMs for Decontextualization
To improve upon end-to-end prompting, we imple- ment QADECONTEXT, a strategy for snippet de- contextualization inspired by our framework. This approach is easy to adopt, making use of widely- available LLMs as well as off-the-shelf passage retrieval models. See Figure 2 for a schematic. All prompts for each component are in $A.6.
We study the extent to which current LLMs can perform scientific decontextulaization, and how our QA framework might inform design of methods.
"https: hhttps://www. upwork. com
"We use the minimum wage Washington, DC, USA ($16.10 USD at time annotation) as reference deter- mining fair compensation.
Question Generation. We prompt an LLM SS davinci) to generate questions with a one-shot prompt with instructions. We found more in-
10 "https: /www. http://wwwwwpww.e/e/e/e prolific.co

---

2
Question Answering
Retrieve Q. undance
Document
Document
Answer
Snippet
Generate di aBsReY
Document
Question Generation
Rewriting
Questian
Snippet
Question
dic Generato srever
Evidence
Decontorlualzedd
Snippet
Snippot
Answer
Question
Evidence
Figure 2: The three modules used for QADECONTEXT. Question generation formulates clarification questions given a snippet and (optionally) the source document. Question answering @ returns an answer and (optionally) supporting evidence for given a question, snippet, and (optionally) the source document. Rewriting @ receives the snippet and (one of more elements in) the context produced by previous modules to perform decontextualizzatin.. For examples of the outputs of these steps, see Table
context examples allowed for better control of the number of questions, but decreased their quality.
Models QG QA R
Metries
Strategy
SARI-add
BERTScore
QADECONTEXT
0.140
0.483
QADECONTEXT
0.146
0.472
Question Answering. Given a question, we can approach answering in two ways. In retrieve-then- answer, we first retrieve the top relevant para- graphs from the union of the source document and any document cited in the snippet, and then use an LLM to obtain concise answer from these k = 3 and Con-
End-to-end
only
0.135
0.499
End-to-end
only
0.103
0.536
Table 1: Comparison between our QADECONTEXT strategy versus prompting the model end-to-end. The davinci and @ gpt-4 icons represent the models used for each of the Question Generation (QG), Ques- tion Answering (QA), Rewriting (R) components our strategy: end-to-end prompting only uses single model. Results from higher performance strategy are bold.
paragraphs. Specifically, triever (Izacard al., 2021) for the retrieval step. and davinci or S gpt-4 as the LLM.
Alternatively, the full document setting, we di- rectly prompt an LLM that supports longer context windows gpt-4) to answer the question given the entire source document as input. This avoids the introduction of potential errors from performing within-document passage retrieval.
Input D Q A E
Rewriter Module (SARI-add) davinci
claude
tūlu 1lama2
X
0.120
0.135
0.077
/ V X
0.167
0.198
0.087
0.090
0.418
0.413
0.090
0.238
XXX 7
0.142
0.177
0.069
0.073
Finally, prompt LLM ($ davinci) with the snippet, generated questions, generated answers, and any relevant context (c.g.. retrieved evidence snippets if using retrieve-then- answer and/or text from the source document) ob- tained from the previous modules. This module is similar to end-to-end prompting of LLMs from $5.1 but prompts are slightly modified to accommodate output from previous steps.
Rewriting.
X X
0.216
0.217
0.107
0.173
X
0.433
0,422
0.130
0.330
TXX7
0.144
0.174
0.069
X
0.199
0.224
0.101
1/VV 0.378
0.427
0.205
X
0.095
0.097
0.042
0.041
X X
0.547
0.527
0.252
0.312
Table 2: Oracle performance of QADECONTEXT when using gold (Q)uestions, Jswwes,,, answer (E)vidence obtained from annotators or source (D)ocument. Across models, extra input hurts performance (e.g., QA outper- forms DQA and DQAE). Results from best two input configs are bold. Entries for @ tülu are missing as inputs don't tt fit context window.
We report results also in Table 1. We find our QADECONTEXT strategy achieves a 41.7% relative improvement over the @ gpt-4 end-to-end bascline, but given the low SARI-add scores, there remains much room for improvement.
Results.

---

5.3 Human Evaluation
binations using two commercial (A) claude-v1, text-davinci -093) and two open source (#2 tülu-30b, 0 11ama2-caat-70b) models. Our prompts are $A.6.
We conduct small-scale human evaluation (n 60 samples) comparing decontextualized snippets with our best end-to-end ($ davinci) and QADE- CONTEXT approaches. Snippets were evaluated whether they clarified the points that the reader needed help understanding, System outputs for given snippet were presented in randomized order and ranked from best to worst. The evaluation was performed by two coauthors who were familiar with the task, but not how systems were imple- mented. The coauthors annotated 30 of the same snippets, and achieved binary agreement of 70%. This is quite high given the challenging and sub- jective nature of the task; Choi et al. (2021) report agreements 80% for snippets from Wikipedia. Our QADECONTEXT strategy produces convinc- ing decontextualized snippets in 38% of cases against 33% for the end-to-end approach. We note that decontexualization remains somewhat subjec- tive (Choi et al., 2021), with only 42% of the gold decontextualizzttion judged acceptable. We con- duct two-sample Binomial test and find that the difference between the two results not statisti- cally significant (p = 0.57). See Table for quali- tative examples of QADECONTEXT errors.
We report results in Table 2. First, we observe that, average, the performance ranking of dif- ferent input configurations to the rewriter is consistent across models: (1) Including the gold evidence (E) is better than including larger docu- ment context (D), (2) including the gold answer (A) results in the largest improvement in all settings, and (3) performance is often best when the rewriter receives only the questions (Q) and answers (A).
Second, we find that overall performance of the best oracle configuration of QADECONTEXT ($ davinci) achieves 261 higher performance over the best QADECONTEXT result in Table As we did not change the rewriter for these oracle experiments, we conclude significant errors are being introduced in the question generation and answering modules, rather than in the rewriter.
6.2 Are question generation or question answering the performance bottleneck?
We continue this investigation using similar ora- cle experiments to assess performance bottlenecks in the question generation and question answering modules. To scope these evaluations, we only con- sider input configurations to the rewriting module based the top two oracle results for @ from Table 2 2-QA and DQAE. We report these
6 Analyzing Performance Bottlenecks through QADECONTEXT
davinci
Modularity of our framework for decontextualiza- tion allows us to study performance bottlenecks; Which subtask (question generation, question swering, rewriting) do LLMs struggle with the most? We conduct ablation experiments to bet- ter understand the performance and errors of each module in QADECONTEXT. We refer the reader to Table 4 for qualitative error examples.
new results Table 3).
Question Generation. QADECONTEXT we replace generated questions with gold ones? From Table 3, we see a relative lift ranging from 48.2% to 72.7% % by switching to gold questions (see rows 5 vs 8, 6 vs 9, 7 vs 10). Question generation is a major source of error.
First, how much better is
6.1 Is rewriting the performance bottleneck?
To study if the rewriting module is the bottle- neck, we run oracle experiments to provide an upper bound on the performance of our strat- egy. We perform these experiments assuming that the LLM-based rewriting module receives gold
Question Answering. How much better is retrieve-then-answer in QADECONTEXT if we used gold evidence instead of relying on retrieval? Just ablating the reirieve step. from Table we only see a modest improvement ranging from 14.2 % to 17.8% % by replacing the retrieved evi- dence with gold evidence (see rows vs 5, 4 vs 6). Within-documen passage retrieval is not a major source of error.
Questions, Answers, and answer Evidence paragraphs. We also investi- gate various combinations of this gold data with the source Document itself (i.e., title, abstract, paragraph containing the snippet, and section header). To ensure our best configuration ap- plies generally across models, we study all com-
How much better QADECONTEXT we used gold answers? We ablate the full document ap- proach by replacing generated answers with gold

---

QG
QA
Rewrite
vate future work pursuing methods that can adapt to different document types, such as Wikipedia scientific documents, and user scenarios, such as snippets being user- user-facing versus intermediate ar- tifacts in a larger NLP systems. These situations require personalizing decomlextuullzzatons to di- verse information needs.
Question Evidence Answer DQAE SARI-ade add BERTScore
X/Vx 0.527
0.625
gold
//V 0.427
0.580
gold
X/VX 0.274
0,541
gold
gpt-4
VVV 0.256
0,523
X/VX 0.240
0,530
retrieve
gpt-4
//VV 0.217
0.515
Related Work
full doc
g
X/VX 0.209
0,525
X/VX 0.139
0,481
retrieve
gpt-4
7.1
Decontextualization: Uses and Challenges
//V 0.146
0.472
davinci
full doc
gpt-4 X/VX
0.141
0.495
Our work is based on Choi et al. 's (2021) semi- nal work on decontextualizztion. They show de- contextualized snippets can improve passage re- trieval. Potluri et al. (2023) show an extract-then- decontextualize approach can help summarization.
Table 3: Ablating modules in our decontextualization pipeline that affect the input the final Rewriter mod- ule. We ablate (1) source of question and (2) use of the full document retrieving passages as evidence. We investigate including the evidence (H) and source docu- ment information (D) in the Rewriter prompt in addition the questions (Q) and answers (A). Last three rows are fully predictive, while others use gold data.
Despite its utility, decontextualizzaton remains a challenging task. Eisenstein et al. (2022) noticed similar failures to those found in $5.1 when dealing with longer input contexts. Beyond models, decontextualization is challenging even for humans. Choi et al. (2021) note issues related to subjectivity resulting in low annotator agreement. Literature in human-compute interaction on the struggles humans have with note-taking (judging what infor- mation to include or omit when highlighting) are similar to those observed in our formative study and data annotation (Chang et al., 2016).
ones, and see the largest relative improvement: 66.8% % to 92.3% (see rows vs 3, 2 vs 4). Ques- tion answering is a major source of error.
Overall. While the relative performance improve- ment from using gold data is large in both the ques- tion generation and question answering modules, the absolute values of the scores are quite differ- ent. On average, using gold questions provides a 0,080 increase in absolute SARI-add (rows vs 8, vs 9, 7 vs 10), while using gold answers pro- vides a 0.212 absolute increase (rows 1 vs 3, vs 4). We identify question answering as the main performance bottleneck in QADECONTEXT.
7.2 Bridging QA and other NLP Tasks
In this work, we establish a bridge between de- contextualization and QA. A similar bridge be- tween QA and discourse analysis has been well- studied in prior NLP literature. In addition to the relevant works discussed in $3.1, we also draw attention to works that incorporate QA to anno- tate discourse relations, including Ko al. (2020, 2022); Pyatkin et al. (2020). In particular, Pyatkin et al. (2020) show that complex relations between clauses can be recognized by non-experts using a QA formulation of the task, which is reminiscent of the lowered cognitive load observed during our formative study ($3.1). 1). Beyond discourse analysis, prior work has used QA approach to down- stream NLP tasks, including elaborative simplifica- tion (Wu al., 2023), identifying points of confu- sion in summaries (Chang et al., 2023b), evaluating summary faithfulness (Durmus et al., 2020), and paraphrase detection (Brook Weiss et al., 2021).
6.3 Does QADECONTEXT generalize beyond scientific documents?
We compare our approach the one used by Choi et al. (2021) by applying our QADECONTEXT strategy to their Wikipedia data. In these exper- iments, we find that QADECONTEXT performs slightly worse than end-to-end LLM prompting (~1 percentage point SARI-add absolute differ- ence). These results match our intuitions about the QA approach from our formative study ($3.1 and $A.44 in which study participants found that fol- lowing the QA framework for Wikipedia was cum- bersome, was unhelpful, or hindered their ability to perform decontextualization. The results also moti-

---

Question Generation
Instruction following
davinci might fail to follow requirements specified in the instructions. For example, our prompt explicitly required avoiding questions about figures. which weren't part of the source document.
Realistic questions
davinci might generate questions that a human wouldn't It need to ask as the information is already provided in the snippet. For example, for the snippet "In addition, our system is independent of any external resources, such as MT systems dictionaries, Samiotou (2004).". @ davinci generated "What kind of external resources were used by Kranias and Samiotou (2004)?" even though the information is already in the snippet (see highlighted text).
opposed to the work by Kranias and
User
background
davinci generates questions whose appropriateness depends on user background knowledge, For example, "What is ROUGE score?" is not good question for a user with expertise in summarization.
Question Answering
Retrieval errors
davinci or @ gpt-4 fails to abstain and hallucinates answer despite irrelvant retrieved passages.
The question is answerable from retrieved context, hut @ davinci or @ gpt-4 4 either unnecessarily abstains or hallucinates a wrong answer. For example, given question: "What does "each instance' refer to?" and the retrieved passage: "The main difference was that (Komiya and Okunura, 2011 I) deternined the optimal DA method for each triple of the target word type of WSD, source data, and get data, but this paper determined the method for each instance." ". the model outputs "Each instance refers to each word token of the target data." The correct answer is highlighted.
Answer errors
Rewriting
Format errors
davinci might fail to enclose snippet edits in brackets. During human evaluation ($5.3), annotators found that 24% of generations had these errors (compared to 5% of gold annotations).
Missing info
Overall, annotators found that 45% of decontextualized snippets through QA QADeCONTEnT were still missing relevant information raised additional questions (compared to 34% for the gold suippets).
Table 4: Most common error types different stages of QADECONTEXT. Question generation and question answering errors identified through qualitative coding of 30 oracle outputs from $6.2. Rewriting errors identified during human evaluation ($5.3).
7.3
QA for User Information Needs
models. Our QADECONTEXT strategy relies on the latter and falls naturally from human workflows as found in our formative study.
Like in user-facing decontextualization, prior work has used questions to represent follow-up (Meng et al., 2023), curiosity-driven (Ko et al., 2020), or confusion-drine driven (Chang et al., 2023b) informa- tion needs. QA is a well-established interaction paradigm, allowing users to forage for information within documents through the use of natural lan- guage (Wang et al., 2022; ter Hoeve et al., 2020; Jahanbakhsh et al., 2022; Fok al., 2023a).
Conclusion
In this work, we present a framework and a strategy to perform decontextualization for snippets from scientific documents. We introduce task require- ments that extend prior work to handle user-facing scenarios and the handle the challenging nature of scientific text. Motivated by formative study into how humans perform this task, we propose a -asedd framework for decontextualization that decomposes the task into question generation, an- swering, and rewriting. We then collect gold de- contextualizations and use them to identify how to best provide missing context so that state-of-ther art language models can perform the task. Finally, implement QADECONTEXT, simple prompt- ing strategy for decontextualizattin, though ulti- mately we find that there is room for improvement on this task, and we point to question generation and answering in these settings as important future directions.
7.4 Prompting and Chaining LLMs
Motivated by recent advancement in instruction tun- ing of LLMs (Ouyang et al.. 2022), several works have proposed techniques to compose LLMs to per- form complex tasks (Mialon et al., 2023). These approaches often rely on a pipeline of LLMs to generate to complete task (Huang et al., 2022; Sun al., 2023; Khot et al., 2023), while giving model access to modules with different capabili- ties (Lu et al., 2023; Paranjape al., 2023; Schick et al., 2023). While the former typically seen as an extension of ofuin-of-thoughh (Wei et al., 2022), the latter enables flexible "soft interfaces" between

---

Limitations
Prompting does not guarantee stable output, limiting downstream applicability of the decon- textualization approach. As highlighted in Ta- ble 9, all approaches described in this work do not reliably produce outputs that precisely follow the guidelines described in $2. Thus, current systems are likely not suitable to be used in critical appli- cations, and care should be taken when deploying them in user-facing applications.
Automated evaluation metries may not corre- late with human judgment. In this work, we make extensive use of SARI (Xu et al., 2016) to es- timate the effectiveness of our decontextualizztton pipeline. While Choi et al. (2021) has successfully applied this metric to evaluate decontextualization systems, text simplification metrics present key bi- ases, for example preferring systems that perform fewer modifications (Choshen and Abend, 2018). While this work includes human evaluation on subset of our datasets, the majority of experiments rely on aforementioned metrics.
Deconfextualization is only studied for English and for specific scientific fields. In this work, we limit the study of decontextualization to natural language processing papers written in English. The reason for this is two-fold: first, most scientific manuscripts are written in English; second, current instruction-tuned LLMs, particularly those that are open, are predominantly monolingual English mod- els.
Collecting and evaluating decontextuallzations of scientific snippets is expensive. The cost of collecting scientific decontextualions limited the baselines could consider. For example, Choi al. (2021) approach the decomtextualization task by fine-tuning
sertence-to-sqquence model. While training such a model on our task would be an interesting baseline to compare to, it is not feasible because collecting enough supervised sam- ples is too costly. In our formative study, we found that it took experienced scientists five times longer to decontextualize snippets from scientific papers compared to ones from Wikipedia. Instead, we are left to compare our method to Choi al.'s (2021) by running our pipeline in their Wikipedia setting.
Ethical Considerations & Broader Impact
Reformulation of snippets may inadvertently introduce factual errors alter claims. Scien- tific documents are mean disseminate precise and verifiable research findings and observations. Because LLMs are prone to hallucination and may inadvertently modify the semantics of a claim, their use in scientific applications should be carefully scrutinized. Our decontextualization approach is essentially motivated by the need to make snip- pets portable and understandable away from their source; however, this property makes verification of their content more challenging. While this work does not discuss safeguards be used to mitigate this risk, these factor must be considered if this research contribution were to be implemented in user facing applications.
The high cost of collecting data in this domain also limited our human evaluation due to the time and expertise required for annotating model gen- erations. For example, a power analysis using 0.05 and power= 0.8, and assuming true ef- feet size of 5 percentage points absolute difference. estimates that the sample size would be n 1211 judgements per condition for our evaluation $5.3. Evaluating model generations is difficult for many tasks that require reading large amounts of text or require domain-specinn expertise to evaluate. Our work motivates more investment in these areas.
Availability of decontextualization tools may dis- courage users from seeking original sources. Because decontextualization systems are not gen- erally available to the public, users today may be more likely to seek the original content of a snip- pet. Progress in decontexualization systems might change that, as snippets may offer a credible re- placement for the full document. We recognize that, while this functionality might offer improvements in scientific workflows, it would also encourage bad scholarly practices. Even more broadly, more general-domain decontextualization systems might lead to users not visiting sources, thus depriving content creators of revenue.
Closed-source commercial LLMS are more effec- tive than open models. While we experimented with open models for writing decontextualized snip- pets (6 tülu-30b, 0% 1lama2-chat-27 - 70b), results indicate large gap in performance between their closed-source counterparts, such Al claude and s davinci. Since these systems are not available everywhere and are expensive, their use makes difficult for other researches to compare with our work, and use our approach.

---

Acknowledgements
context. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
The authors would like to thank Doug Downey, Eu- nsol Choi, Marti Hearst, Jessy Li, and the Semantic Scholar team at A12 for useful conversations, partic- ipation in user studies, and feedback on our paper draft. We would also like to thank the reviewers for their helpful suggestions and actionable feedback.
Yapei Chang, Kyle Lo. Tanya Goyal, and Mohit lyyer. 2023b. BooookScore: A systematic exploration of book-length summarization in the era of LLMs.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sen- tences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461.
Author Contributions
Benjamin Newman led the project, collected the annotations, implemented all methods, and ran ex- periments. Luca Soldaini, Arman Cohan, and Kyle Lo were project advisors and provided mentorship. Luca Soldaini also contributed to the code, Kyle Lo conducted the formative study, and the two of them helped with human evaluation. Raymond Fok con- tributed HCI expertise to the framing of the paper. All authors were involved with writing the paper.
Leshem Choshen and Omri Abend. 2018. Inherent bi- ases in reference- evaluation for grammatical error correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 632-642. Melbourne, Australia, Association for Computational Linguistics.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan, and Noah A. Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Voiume I: Long Papers), ), pages 7282-7296, Online. Association for Computational Linguisties.
References
Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2023. Paper plain: Making medical research papers approachable to healthcare consumers with natural language pro- cessing. ACM Trans, Comput. Interact. Just Accepted.
Arman Cohan and Nazli Goharian. 2017, Contextualiz- ing citations for scientific summarization using word embeddings and domain knowledge. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17. page 1 1133-1136, 136, New York, NY, USA. Association for Computing Machinery.
Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roce Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisen- stein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni. Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. ArXiv,
Arman Cohan, Luca Soldaini, and Nazli Goharian. 2015. Matching citation text and cited spans in biomedi- cal literature: search-oriented approach. In Pro- ceedings of the 2015 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1042-1048, Denver, Colorado. Association for Com- putational Linguistics.
Daniela Brook Weiss, Paul Roit, Ayal Klein, Ori Ernst, and Ido Dagan. 2021. QA-Align: Representing Cross-Text Content Overlap by Aligning Question- Answer Propositions. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 9879-9894, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seekinn chored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1599-4610 On- line. Association for Computational Linguistics.
questions and answers an-
Joseph Chee Chang, Nathan Hahn, and Aniket Kittur. 2016. Supporting mobile sensemaking through inten- tionally uncertain highlighting. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST 16, page 61-68, New York, NY, USA. Association for Computing Machinery.
Kordula De Kuthy, Nils Reiter, and Amdt Riester. 2018. QUD-based annotation of discourse structure and information structure: Tool and evaluation. In Pro- ceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources As- sociation (ELRA).
Joseph Chee Chang, Amy X. Zhang, Jonathan Bragg, Andrew Head, Kyle Lo, Doug Downey, and Daniel S. Weld. 2023a. Citesee: Augmenting citations in scien- tific papers with persistent and personalized historical

---

Esin Durmus, He He, and Mona Diab. 2020. FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summariza- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguisties.
organization of scientific literature. In Proceedings of the 35th Anmual ACM Symposium on User Interface Software and Technology, UIST '22, New York, NY, USA. Association for Computing Machinery.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In Proceedings of the 10th International Conference on Learning Representations. ICLR.
Jacob Eisenstein, Daniel Andor, Bernd Bohnet, Michael Collins, and David Mimno. 2022. Honest students from untrusted teachers: Learning interpretable question-am wattioonnnsweiing pipeline from a pretrained lan- guage model. In Workshop on Trustworthy and So- cially Responsible Machine Learning, NeurlPS 2022.
Wei-Jen Ko, yu- Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. 2020. Inquisitive question gener- ation for high level text comprehension. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6544-6555, Online. Association for Computational Linguistics.
Raymond Fok, Joseph Chee Chang. Tal August, Amy X. Zhang, and Daniel S. Weld. 2023a. Qlarify: Bridg- ing scholarly abstracts and papers with recursively expandable summaries.
Raymond Fok, Hita Kambhamettu, Luca Soldaini, Jonathan Bragg, Kyle Lo. Marti Hearst, Andrew Head, and Daniel Weld. 2023b. Scim: Intelligent skimming support for scientific papers. In Proceed- ings of the 28th International Conference on Intelli- gent User Interfaces. IUI '23, page 7764490, New York, NY, USA. Association for Computing Machin- ery.
Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, and Junyi Jessy Li. 2022. Dis- course comprehension: A question answering frame- work to represent sentence connections. ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11752-11 11764, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Proceed-
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. GLTR: Statistical detection and visual- ization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 11 11,111, Florence. Italy. Association for Computational Lin- guistics.
Anne Lauscher, Brandon Ko, Bailey Kuchl, Sophie Johnson, Arman Cohan, David Jurgens, and Kyle Lo. 2022. MultiCite: Modeling realistic citations requires moving beyond the single-sentence single- label setting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, pages 1875-1889, Seattle, United States. Association for Computational Linguistics.
Han L. Han, Junhang Yu, Raphael Bournet, Alexandre Ciorascu, Wendy E. Mackay, and Michel Beaudouin- Lafon. 2022. Passages: Interacting with text across documents. Proceedings of the 2022 CHI Confer- ence on Human Factors in Computing Systems, CHI '22, New York, NY, USA. Association for Computing Machinery.
Jimmy Lin, Dennis Quan, Vineet Sinha. Karun Bakshi. David Huynh, Boris Katz, and David R. Karger. 2003.
The role of context in question answering systems. In CHI '03 Extended Abstracts on Human Factors in Computing Systems, CHI EA '03, page 1006-1007, New York, NY, USA. Association for Computing Machinery.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 91 18-9147. PMLR.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4969 4983, Online. Asso- ciation for Computational Linguistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense infor- mation retrieval with contrastive learning.
Pan Lu, Baolin Peng. Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023, Chameleon: Plug-and-play compo- sitional reasoning with large language models. ArXiv, abs/2304.09%. 099842.
Farnaz Jahanbakhsh, Elnaz Nouri, Robert Sim, Ryen W. White, and Adam Fourney. 2022. Understanding questions that arise when working with business documents. Proc. ACM Hum.-Comput. Interact., 6(CSCW2).
Kelvin Luu, Xinyi Wu, Rik Koncel-Keezione Kedziorski, Kyle Lo, Isabel Cachola, and Noah A. Smith. 2021. Ex- plaining relationships between scientific documents. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
Hyeonsu Kang, Joseph Chee Chang, Yongsung Kim, and Aniket Kittur. 2022. Threddy: An interactive system for personalized thsead-based exploration and

---

11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 21 2130-2144, Online. Association for Computational Linguisties.
Simeng Sun, Y. Liu, Shuo Wang. Chenguang Zhu, and Mohit lyyer. 2023. Pearl: Prompting large language models to plan and execute actions over long docu- ments. ArXiv, abs/2305. 14564.
Yan Meng. Liangming Pan, Yixin Cao, and Min- Yen Kan. 2023. Followupgg: Towards information- seeking follow-up question generation. ArXiv, abs/2309.05007.
Maartje ter Hoeve, Robert Sim, Elnaz Nouri, Adam Fourney, Maarten de Rijke, and Ryen W. White. 2020. Conversations with documents: An exploration of document-centen assistance. Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, CHIIR '20, page 43-52, New York, NY, USA. Association for Computing Machinery.
Grégoire Mialon. Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-YY Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023, Augmented lan- guage models: a survey. Transactions on Machine Learning Research. Survey Certification.
Leah Velleman and David Beaver. 2016. Question- based Models of Information Structure. In The Ox- ford Handbook of Information Structure. Oxford Uni- versity Press.
Edgar Onea. 2016. Potential questions at the semantics- pragmatics interface.
Vivek Kumar Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2023. Ghostbuster: Detecting text ghostwritten by large language models. ArXiv, abs/2305.15047.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022, Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 01-27730-27744
Sheng-Fu Wang. Shu-Hang Liu, Tian-Yi Che, Yi-Fan Lu, Song-Xiao Yang. Heyan Huang, and Xian-Ling Mao. 2022. Hammer pdf: An intelligent pdf reader for scientific papers. In Proceedings of the 31st ACM International Conference on Information & Knowl- edge Management, CIKM '22, page 5019-500 --220, New York, NY. USA. Association for Computing Machin- cry.
Srishti Palani, Aakanksha Naik, Doug Downey, Amy X. Zhang. Jonathan Bragg, and Joseph Chee Chang. 2023. Relatedly: Scaffolding literature reviews with existing related work sections. In Proceedings of the 2023 CHI Conference on Human Factors in Comput- ing Systems, CHI '23, New York, NY, USA. Associa- tion for Computing Machinery.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Yating Wu, William Sheffield, Kyle Mahowald, and Junyi Jessy Li. 2023. Elaborative simplification as implicit questions under discussion.
Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh, Hanna Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi- step reasoning and tool-use for large language mod- els. ArXiv, abs/2303.09014.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing sta- tistical machine tramslation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
Abhilash Potluri, Fangyuan Xu, and Eunsol Choi. 2023. Concise answers to complex questions: Summariza- tion of long-form answers.
Shiyue Zhang, David Wan, and Mohit Bansal. 2023, Extractive is not faithful: An investigation of broad unfaithfulness problems in extractive summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume
Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020. QADiscourse Discourse Relations QA Pairs: Representation, Crowdsourcing and Baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2804-2819, Online. Association for Computational Linguistics.
1: Long Pupers), pages 2153-2174, Toronto, Canada. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu. Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with BERT. In International Conference on Learning Representations.
Amdt Riester. 2019. Constructing qud trees. Questions in Discourse.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves use tools. ArXiv, 14471.0..04611

---

A Appendix
TSP. Title, Section header, and the Paragraph containing the snippet. This is the same con- dition as (Choi et al., 2021)
A.1 Location of necessary context for decontextualizing snippets
See Table 5:
TASP and TAISP. These add the Abstract and Introduction respectively both of these contain much of the background context that might need to be incorporated into the snip- pets.
Source
Location of necessury context
of snippets
Wikipedia
Ouly Context Paragraph
Only Context Paragraph
Science
Conteal Paragraph or Absiract
51.1%
Cited Paper
14.8%
We found that TASP performed best, 0.03 SARI-add points better than TSP, and 0.01 points better than TAISP. Not including the introductions is potentially helpful because they might include too much distracting information).
Table 5: Illustration of to decontextualize text from scientific papers compared to Wikipedia. Very often context needed decon- textualize scientific snippets comes from outside the paragraph with the snippet, including cited papers. This is compared Choi et al.'s (2021) estimate that the full-text was needed for only 10-20% of the decontextu- alizations.
it
difficult
A.4 Additional findings from formative study
In our formative study, we found that stepping through the full framework slows down manual decontextualization. Participants averaged 110 sec- onds ( (Wikipedia) and 555 secons (science) per snip- pet when following QA and instead averaged 66 seconds (Wikipedia) and 313 seconds (science) per snippet in the NoQA condition. Second, we find noticeable difference in annotation quality in ei- ther setting when operating on Wikipedia snippets. of participants complained that writing down each question and answer was awkward given the simplicity of the task.
A.2 Types of questions asked about scientific document snippets
We additionally ask annotators to label their ques- tions based on categories we developed while pi- loting the writing process. We determined that the questions that people ask fall into three categories: (1) Definitions of terms expansions of acronyms, (2) Coreference resolution, (3) Simply seeking more context to feel more informed. The annota- tors' labels are in Table 6:
A.5
davinci vs S gpt-4 on QA
Question Type Coref
Define
More Context
We compare davinci to gpt-4 on our ques- tion answering step, finding that @ gpt-4 outper- forms S davinci in all cases. The results are visible in Table 7.
RA CGE
50 (31 (6)
59 (37%)
52 (32%)
102 (31 (6)
142 (44%)
82 (25%)
Table 6: Number of tokens and counts in dataset, separating out Research Assistant (RA) and Citation Graph Explorer (CGE) examples.
A.6 LLM prompts
The following prompts are for the different stages of the pipeline. They the prompts for the best- performing models. For prompts for Al claude, # tülu and 09 1lama2, please see the github reposi- tory linked on the first page.
A.3 TASP: Selecting important sub-regions of document when prompting
For models with context windows too small to fit entire papers like S davinci, we need a condensed representation of the paper to use in prompts. (Choi et al., 2021) find that most of the sentences they decontextualize only require the Title, Section Header of the section the sentence is in, and the paragraph surrounding the snippet. For our snippets from scientific documents. this is likely not
A.6.1 Question Generation
The following text is from a scientific paper, but might include language that requires more context to understand. The language might be vague (like "their results") or might be too specific (like acronyms or jargon). Write questions that ask for clarifications. If the language is clear, write "No questions.".
when paper- specific terms need to be defined. As such, we explore a number different options:
Guidelines:

---

QG
QA
Rewrite
paper, answer the question about "text snippet" below.
Question
Evidence
Answer
DQAE SARI-add BERTScore
X/ / 0,527
0.625
Information from the paper: Title: "((title))"
gold
//V 0.427
0.580
gold
X/VX 0.233
0,536
Abstract: "((abstract)))
gold
davinci
/VV 0.236
0.527
Paragraph with potentially helpful information:
"[[ evidence }}"
X/Vx 0.274
0,541
Paragraph with potentially helpful information:
gpt-4
/VV 0.256
0.523
"[[ evidence #2 }}"
Paragraph with potentially helpful information: "[[ evidence #3 }}"
X/VX 0.193
0,524
davinci
/VV 0.190
0.518
retrieve
Section of the paper the snippet comes from: "(( section header))" Paragraph with the snippet: "[(paragraph with snippet)]"
X/Vx 0.240
0.530
gpt-4
//V 0.217
0.515
full doc
gpt-4
X/Vx 0.209
0.525
Text snippet: "{(snippet))"
X/VX 0,117
0.504
davinci
//V 0.140
Given the above information, please answer the following question. Keep your answer concise and informative. It should be at most sentence long. If you cannat find the answer then write "No answer.": Question: ([question))
davinci
retrieve
X/VX 0.139
0.481
gpt-4
/VV 0.146
0,472
full doc
gpt-4
x/Vx
0.141
0.495
Table 7: Ablating modules in decontextualizationn pipeline that affect the input to the final Rewriter module, We ablate (1) source question, (2) use of the full document retrieving passages evidence, (3) choice the QA model for obtaining the answer, and (4) the amount of context provided the rewriter module. Last three rows are fully predictive, while others use gold data. Rewriter module is identical to that from last row of Table 2.
A.6.3 Rewriting
The following "text snippet" will be quoted in an article using the Chicago Manual of Style The following questions were answered using information from the paper. Rewrite the "text snippet" into quote format by adding the answers in-between square brackets. Write as if you were an expert scientist in the field of natural language processing.
Write the two, three most important questions. Do not write unimportant questions.
Information from the paper:
Question: {{ question #1 }} Answer: {{ answer #1 }
not ask about people citations. Sometimes citations show "BIBREF" Do not ask questions whose answer in the snippet.
Question: question #2 })
Answer: {[ answer
Do not ask about Tables ("TABREF"), Figures ("
FIGREF"), Sections ("SECREF") Formulas (" INL INEFORM").
Text snippet: "[[sentence))"
Instructions:
Example:
Snippet: spirit, CaRE (Gupta al., 2019) comes closest our model; however, they not address the problem type compatibility
Using the given information, please rewrite the
text snippet by adding additional information into square brackets. example: snippet "Our approach performs well" becomes "[REFØ's] approach bidirectional language modeling] performs well".
link prediction task BIBREF3 (See Figure FIGREF2 for details)."
Questions:
what is "CaRE*?
what is the authors' approach?
For example: the snippet "Our task becomes "[REFe's] s] task is MT [machine translation]."
what is type compatibility?
Snippet: "([snippet))"
Questions:
After adding clarifying information:
Replace first-person pronouns with placeholder. Replace "we" with "[REFØ]" and "our" with "[REFO's]"
A.6.2 Question Answering
Remove discourse markers (like "in conclusion",
Using the given information from the scientific
"in this section", "far instance", etc.)

---

Citations are marked as BIBREF or (Author Name, Year). Keep these the same, Do not add any additional citations.
If needed, additional short clarifications that necessary for an expert reader understand the broader context of the quote. Only up to single sentence and put the sentence between square brackets.
Remove any references to Figures ("FIGREF") and Tables ("TABREF") Fix the granmar
Please rewrite the snippet according to the instructions and the given information.
After adding clarifying information: Replace first-person pronouns with a placeholder. Replace "we" with "[REFa I" and "our" with "[REFO's]". Remove discourse markers (like "in conclusion", "in this section", "for instance", etc.)
Rewrite:
A.6.4 End-to-End Model @@gpa ggt-4)
system:
You scientist in the field of natural language processing. Using the given information from scientific paper, rewrite given text snippet so it stands alone. To do this:
Citations are marked as BIBREF or (Author Name, Year). Keep these the same. Do not add any additional citations.
Remove any references to Figures ("FIGREF
") and Tables ("TABREF") Fix the grammar
Remove discourse markers (like conclusion", this section", "for instance", etc.)
Reminders:
Replace first-person pronouns with placeholders. Replace "we" with "[REFe 1" and "our" with "TREFe']]" Remove time-specific words like "current" Make other sur surface-leve changes to fix grammar
Follow the Chicago Manual of Style for quotes by putting all added text between square brackets.
The rewritten snippet is quote, the word order should closely match the original snippet's. Ignore irrelevant information.
Resolve any vague or unclear references in the snippet (e.g. "our approach" or "our method")
Please rewrite this snippet according the instructions and the given information
Define any specific terminalagy or acronyms that other scientists will not be familiar with.
Text snippet: "[[sentence))"
user:
Using the following scientific paper, rewrite the "text snippet" that follows so it stands alone. The "text snippet" will be quoted in an article using the Chicago Manual of Style. Rewrite the "text snippet" into quote format by adding the answers in- between square brackets.
A.6.5 End-to-End Model (g davinci)
The following "text snippet" will be quoted in an article using the Chicago Manual of Style. Using the given information from scientific paper, rewrite the "text snippet" into quote format by adding in any clarifying information in square brackets. Write if you scientist in the field of natural language processing.
expert
Paper:
([full_text]]
Information from the paper:
Text Snippet: "[(sentence))"
Title: "[[title)]"
Instructions:
Abstract: "[[abstract))" {% if secttttssetioo_hheaeeeee
Using only given information, please snippet into quote format. Specifically add the following clarifying information square brackets following Chicago Manual of Style:
rewrite
Header section with the snippet: "[E context_section_heed.
(% endif x)
Paragraph with the snippet: "[fcontext_paragran) })"
Resolve any vague or unclear references in the snippet (e.g. "our approach" or "our method"). Put any clarifying text between brackets. For example Our approach performs well" becomes "[ REF0's] s] approach [bidirectional language modeling] performs well". Define any specific terminology or acronyms that other scientists will not be familiar with. For example "Our task is MT" becomes "[REFe's] 's task is MT [machine translation]."
Text snippet: "[[sentence))"
Instructions:
Using the given information, please rewrite the text snippet into quote format. Specifically add the following clarifying information in square brackets following the Chicago Manual of Style:
Resolve any vague or unclear references in the snippet (e.g. "our approach" or "our method

---

"). any clarifying text between brackets For example "Our approach performs well" becomes "[REFe's] approach [bidirectional language modeling] performs well".
Define any specific terminology or acronyms that other scientists will not familiar with. For example "Our task "[REFO"s] task is MT [machine translation]. If needed, add additional short clarifications that are necessary for an expert reader to understand the broader context of the quote. Only add up to a single sentence and put the sentence in between square brackets.
becomes
After adding clarifying information:
* Replace first-person pronouns with a placeholder Replace "we" with "[REFØ]" and "our" with "[REFO's]" s]".
Remove discourse markers (like "in conclusion",
"in this section", "for instance", etc.) Citations are marked as BIBREF or (Author Name, Year). Keep these the same. Do not add any additional citations.
Remove references to Figures ("FIGREF") and Tables ("TABREF")
Fix the grammar
Reminders:
Follow the Chicago Manual of Style for quotes putting all added text between square brackets.
The rewritten snippet is quote, the word order should closely match the original snippet's.
Please rewrite this snippet according the instructions and given information. Text snippet: "[[sentence))" Rewrite:
A.7 Sample QA Pairs

---

Tite: "DOLORES: Deep Contextualized Knowledge Graph Embeddings"
User query: "Is fine-tuning required to incorporate these embeddings into existing models?" The only requirement is that the model accepts input, embedding layer (for entities and relations). If model fulfills this requirement (which large number of neural models knowledge graphs do), can just use Dolores embeddings as drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings.
Original Snippet
Question: "What is an embedding layer?"
Answer: "An embedding layer is a layer in neural network model that accepts as input representations of entities and relations in the form of embeddings."
Question: "What Dolores embeddings?"
QA-Pairs
Answer: "Dolores embeddings are deep representations of entities and relations in knowledge graphs learned using Bi- Dirrectiionl LSTMs from entity-relation chains." Question: "How do we initialize the corresponding embedding layer?"
Answer: "We initialize the corresponding embedding layer with Dolores embeddings." [REFO's] model accepts input representations for entities and relations in the form of dense continuous vector embeddings [i.c.. an embedding layer]. Dolores embeddings, which are deep contextualized knowledge graph embeddings learned using deep neural sequential model, can be used as drop-in replacement for the embedding layer in existing knowledge graph prediction models. To initialize the corresponding embedding layer, [REFO] simply uses Dolores embeddings.
Decontextualized
Snippet
(SARI-add: 0.176)
Citing puper: "Extructing Social Networks from Literary Text with Word Embedding
Tools") (Wohlgenannt et al., 2016)
Cited paper:
Extracting Social Networks from Literary Fiction "iroom""(Fia "(Elson et al., 2010)
In contrast to our work, (Elson et al., 2010) are solely focus on length and number of dialogues between persons to measure relatedness, whereas our approach looks at gen- eral co-occurrence similarity as measured by LT tools which use word embeddings. Question: "What are LT tools?"
Original Snipper
Answer: "LT tools are language technology tools that use word embeddings for measuring similarity and co-oceurrence text."
Question: "How do LT tools measure co- co-occurrence similarity?"
QA-Puirs
Answer: "LT tools measure co co-occurrence or similarity using word embeddings."
Question: "What word embeddings?"
Answer: "Word embeddings are numerical representations of words in a multi- dimensional space, which capture semantic and syntactic information about the words and their relationships with one another."
[REPO's] approach looks at general co-occurrence or similarity as measured by LT (lan- guage technology] tools, which use word embeddings [language modeling techniques that transform the vocabulary of an input corpus into continuous and low low-dimensional vector representation, capturing semantic and contextual information of words).
Decontextualized
Snippet
(SARI-add: 0. 14148)
Table 8: Two examples of the outputs of the different stages our best decontextualization pipeline. The questions, answers, and decontextualized snippet are all model generated. The first example is from the QASPER dataset (Dasigi et al., 2021), }); the snippet is an evidence passage containing the answer the user question. The second a text span extracted from Wohlgenannt al. (2016) citing Elson al. (2010). Note that the questions not all natural and are sometimes redundant, but the information they query is only included once in the decontextualized snippet.

---

Tite: "DOLORES: Deep Contextualized Knowledge Graph Embeddings"
User query: "Is fine-tuning required to incorporate these embeddings into existing models?"
The only requirement is that the model accepts input, an embedding layer (for entities and relations). If model fulfills this requirement (which a large number of neural models on knowledge graphs do), we can just use Dolores erubeddings as drop-in replacement. We just initialize the corresponding embedding layer with Dolores embeddings.
Original Snippet
The only requirement is that the model accepts as input an embedding layer (for entities and relations). If model fulfills this requirement (which a large numher of neural models on knowledge graphs do). the authors can just use Dolores embeddings [which are created using connections between language models and random walks on knowledge graphs] as drop-in replacement. The authors just need to initialize the corresponding embedding layer with Dolores embeddings.
Gold
The only requirement is that the model accepts as input, an embedding layer [for entities and relations]. If model fulfills this requirement (which a large number of neural models on knowledge graphs do), [REFO] can just use Dolores embeddings as a drop-in replacement. [REFO] just has to initialize the corresponding embedding layer with Dolores embeddings. which can be achieved by creating connections between language models and random walks on knowledge graphs.
Best Oracle Pipeline (SARI-add: 0.522)
[REFO's] model accepts input representations for entities and relations the form of dense continuous vector emheddings [i.c., an embodding layer]. Dolores embeddings. which are deep contextualized knowledge graph embeddings learned using a deep neural sequential model, can be used as drop-in replacement for the embodding layer in existing knowledge graph prediction models. To initialize the corresponding embedding layer, [REFO] simply uses Dolores embeddings.
Best Pipeline (SARI-add: 0.176)
Citing paper: "Extracting Social Networks from Literary Text with Word Embedding Tools" (Wohlgenannt et al., 2016)
Cited paper:
"Extracting Social Networks from Literary Fiction fiction"[Elsen "[Esson et al., 2010)
contrast to our work. (Elson et al., 2010) are solely focus on length and number of dialogues between persons to measure relatedness, whereas our approach looks at gen- eral co-occurrence or similarity as measured by LT tools which use word embeddings. In contrast the authors' work [hased en-necurence statistics and cosine similarityl, (Elson 2010) focus solely length and number of dialogues between persons measure relatedness, whereas the authors' approach looks at general co-occurrence or similarity as measured by [Language Technologyl (LT) tools which use word embeddings.
Original Snippet
Gold
[REPO's] approach [experimentation based co-occurence statistics and cosine similarity| looks at general co-occurrence similarity measured by LT tools [state of the art word embedding tools] which use word embeddings.
Best Oracle Pipeline (SARI-add: 0.5)
[REOOs) s] approach looks at general co occurrerence or similarity as measured by LT [lan- guage technology] tools, which use word embeddings [language modeling techniques that transform the vocabulary of an input corpus into continuous and low- low-dimensional vector representation, capturing semantic and contextual information of words).
Best Pipeline (SARI-add: 0.148)
Table 9: Two examples of our decontextualization pipeline compared with gold annotations and end-to-end output from GPT-3. The first example is from the QASPER dataset (Dasigi al., 2021); the snippet is an evidence passage containing the answer the user question. The second is a text span extracted from Wohlgenanot et al. (2016) citing Elson et al. (2010) Together, they demonstrate how an effective decontextualization system can improve consumption of text outside the originating document. Text in blue has been added by the systems.


Reviews:
[{"review_id": "pbS6UykwGP", "reviewer": "Reviewer_ELeZ", "summary": "", "questions": "Question A: You do not seem to mention using Upwork and Prolific in the limitations yet there are potential risks, mostly of technical (malpractice by workers, use of LLMs to perform the tasks, English proficiency) and ethical (e.g. worker farms scattered in low-income countries) nature. Did you account for such shortcomings?\n\nQuestion B: Between the time of submission and the present state of the art open models have made significant progress so that your assertion line 626 could be modified (\"Closed-source commercial LLMs make for effective pipelines; open models do not\"). Can you comment on this point? Did you measure progress by open models other than tülu?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ELeZ for recognizing our work on this “comparatively new field of research” as “sound” and “comprehensive”. We agree that our “open discussion” of task definition and evaluation is important, especially for engaging with future researchers to explore relatively nascent areas of NLP such as ours. Below, we summarize and address reviewer ELeZ’s questions and concerns:\n\n***Concern 1:*** Use of commercial systems in evaluation\n\n***Response 1***: We agree that it is difficult to interpret and reproduce evaluation results from commercial systems, especially as they change over time. Commercial systems are used by many people, which is why we felt it was important to include them in our evaluation. **We demonstrate that our setting is challenging even for these commercial systems, which helps raise awareness and recalibrate public perception around the strengths and limitations of these systems.** This recalibration is especially important as many of the scientific applications cited in our introduction are built using commercial models. Testing on multiple commercial systems also gives us confidence that our protocol will work on open models as their performance improves.\n\nNevertheless, for scientific reproducibility and our findings’ utility for the broader research community, we agree with the importance of evaluating using open models as well. This is why we also **performed evaluation using the open model Tülu** as it was state-of-the-art on a wide range of tasks at the time of submission. After our paper submission, we also **evaluated the newly released Llama 2 model** (70B chat) and found it to outperform Tülu, but still underperform commercial systems. Specifically, compared to the “best setting” results in Table 1 (bottom row), the SARI-add scores for Llama 2 are 0.342 compared to Tülu (0.252). Furthermore, Llama 2’s increased context length gives us the ability to add results that previously weren’t feasible for Tülu. For example, including the full document + QA pairs + evidence results in a 0.205 SARI-add score compared to 0.378 (Claude) or 0.427 (davinci). We are excited to add the full set of Llama 2 results to Table 1 to the camera-ready, as they indicate rapid advancement in available open models, while still leaving much room for improvement compared to commercial systems.\n\n**_Concern 2_**: How did we select annotators who created our dataset? \n\n**_Response 2_**: We agree that, as with any data collection, one must safeguard against worker exploitation and low-quality annotations. **We took care to avoid potential technical and ethical malpractice and ensure high-quality annotations**. First, **the size of our dataset allowed us to hand-verify every example we collected. This verification took the authors over 20 hrs to conduct.** To incentivize workers, we also paid a reasonably high rate for annotator work:\n\n- On Upwork, we hired four contractors and paid them each $20/hr USD to create the questions. These annotators were first given a set of pilot annotations after which we clarified their questions about the annotation protocol.\n- On Prolific, we avoided the issue of “worker farms” by limiting participants to participants in the USA or the UK and paying them $17/hr USD.\n\nWe will make this information clearer in Section 4.2.\n\nWe’d again like to thank Reviewer ELeZ for their thoughtful questions; the constructive feedback has helped us improve our manuscript. We hope our responses have increased their confidence in our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is scientifically sound, the state of the art comprehensive, the goals are clearly stated and the evaluation is convincing.\n\nThere are a number of open questions regarding the definition and the evaluation of the task but as this field of research is relatively new and as the authors openly discuss the matter it is yet another reason to accept the submission.", "reasons_to_reject": "As the dataset described in 4.3 is fairly small and the methods used are quite complex there might be unknown biases affecting the evaluation. The use of commercial models is subject to change over time which make the evaluation impossible to reproduce or results difficult to interpret.", "questions_to_authors": "Question A: You do not seem to mention using Upwork and Prolific in the limitations yet there are potential risks, mostly of technical (malpractice by workers, use of LLMs to perform the tasks, English proficiency) and ethical (e.g. worker farms scattered in low-income countries) nature. Did you account for such shortcomings?\n\nQuestion B: Between the time of submission and the present state of the art open models have made significant progress so that your assertion line 626 could be modified (\"Closed-source commercial LLMs make for effective pipelines; open models do not\"). Can you comment on this point? Did you measure progress by open models other than tülu?", "paper_topic_and_main_contributions": "The article tackles decontextualization, a comparatively new field of research whose success is currently closely tied to the advances of LLMs. It describes and operationalizes the task of rewriting snippets to incorporate information from their originating contexts. It also evaluates several methods on a specially created dataset.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 410: For the sake of clarity you could define the SARI method or at least reveal what's behind the acronym."}, "extra_text": {}}, {"review_id": "AvLxWsCIr1", "reviewer": "Reviewer_uGwc", "summary": "", "questions": "I'm curious how InstructGPT (davinci) vs. GPT4 looks like in the end2end setting? From the human evaluations, it seems that GPT-4 is selected the best end2end system, but it wasn't clear from Table 2. \n\nOn a related note, it seems that CLF scores don't have a strong correlation with other metrics. It might be worth adding a bit more discussion and analysis to help explain why that is.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to Reviewer uGwc for their insightful questions and suggestions. We are glad that the reviewer found our “in-depth” evaluation of our proposed decontextualization method “conceptually very interesting”. Below, we summarize and address Reviewer uGwc’s questions and concerns:\n\n****\n\n**_Concern 1.1_**_:_ Additional baselines would make our work more comparable to prior work. \n\nOption 1: Train a supervised model on our data like Choi et al., 2021 did on theirs.\n\n**_Response 1.1:_** We carefully considered our evaluation approach and experimental setting, and we appreciate the opportunity to dive deeper into our decision-making. \n\nWe did not pursue a supervised baseline because **collecting large-scale data for scientific decontextualization is too costly**. During our formative study investigating the difference between Choi, et. al. (2021)’s and our scientific decontextualization setting, we found that our setting takes **five times longer** to annotate compared to Wikipedia snippets (details in Appendix A.4). Annotation time and difficulty in recruiting and fairly compensating qualified experts prevented us from collecting enough data for supervised finetuning. \n\nRecent decontextualization work also prefers prompting over fine-tuning. For example, Eisenstein et al., 2022 (cited in the text) find that a seq2seq model fine-tuned on \\~11k examples performs worse at decontextualizing Wikipedia sentences than a prompted LLM (PaLM). Overall, we recognize the limitation of not having a supervised baseline, but we hope the Reviewer finds this is a reasonable tradeoff given the other experiments we’ve invested in. We will make this clearer in our Limitations section.\n\n****\n\n**_Concern 1.2:_** Additional baselines would make our work more comparable to prior work.\n\nOption 2: Test our pipeline on Wikipedia data from Choi et al., 2021.\n\n**_Response 1.2:_** This option is feasible, and we agree such an experiment would help the reader better understand our setting vs the simpler Wikipedia setting in a quantitative manner. In such experiments, we observe that **our QA-based approach performs slightly worse than end-to-end LLM prompting on Wikipedia** (\\~1 percentage point SARI-add absolute difference). This matches our intuitions about the QA approach from our formative study (Section 3 & Appendix A.4) in which **our study participants found that following the QA framework for Wikipedia was cumbersome** and either was unimportant or hindered their ability to perform decontextualization. **Our formative study and experiments illustrate that the QA framework’s benefit over end-to-end baselines is a consequence of our more challenging setting**.\n\nOur quantitative analysis in Table 6 of Appendix 1a also enriches the comparison between the Wikipedia and scientific settings. There we analyzed the question: _how much full text is needed to perform our task?_ We found that **only 20% of our snippets can be decontextualized using only the surrounding paragraph; in contrast, Choi et al. report being able to decontextualize 80-90% using just the surrounding paragraph**. We hope together these results paint a better quantitative picture of the Scientific setting vs the Wikipedia setting. We thank the Reviewer for this suggestion and will include it in the camera-ready using our additional page.\n\n***\n\n**_Concern 2:_** Because we need to fit the entire paper into the context window, we can’t provide in-context examples, which could disadvantage the model.\n\n**_Response 2:_** We agree with the reviewer that not using in-context examples could potentially disadvantage the end-to-end models. In fact, **we do include examples as part of model instructions in the end-to-end prompting baselines**. As one example, in our end-to-end prompt, we include the following instructions:\n\n_Resolve any vague or unclear references in the snippet (e.g. \"our approach\" or \"our method\"). Put any clarifying text between brackets. For example \"Our approach performs well\" becomes \"\\[REF0's] approach \\[bidirectional language modeling] performs well\"._\n\nThis question made us realize that the prompts in the Appendix A.5 were from an older version of the system, before we iterated on the prompts. Thank you for pointing this out - it has been corrected for the camera-ready version. The full prompt is available at our anonymous code link in the paper, at the path `configs/templates/full_paper_chat`.\n\n****\n\n**_Concern 3:_**  Lack of significance testing reported for human evaluation.\n\n**_Response 3:_** To address this feedback, **we have performed more human evaluation.** \n\n- First, we performed double-annotation on the previous human annotation data; based on these, we estimated binary agreement between two annotators of **0.7**. We believe this is reasonably high given our task is quite challenging and subjective; Choi et al reported agreement for Wikipedia snippets around 0.8 and also discussed inherent subjectivity of this task. \n- Next, we had both annotators annotate **60 new generations using GPT3-davinci** for end-to-end instead of GPT4, based on the Reviewer’s suggestion. This was an oversight on our part, and we should have used the one with the higher SARI score. Thanks to the Reviewer for catching this.\n\nWe find the **same human eval result (Table 5) holds from before: Gold > Pipeline > End-to-End.**  We also find the Pipeline and End-to-End acceptability scores are still close (**5-7 percentage points difference**). \n\nWe conducted a **two-sample Binomial test** using just the new **n=60** judgements as well as aggregating all judgments **(n=90)**. In both cases, the difference between Pipeline and End-to-End was **not statistically significant** (p=0.57 under n=60; p=0.45 under n=90). We conducted a power analysis using $\\alpha$=0.05 and power=0.80, and assuming a true effect size of 5 percentage point absolute difference, **estimated sample size would’ve been n=1211 judgments per condition**. Unfortunately, this would’ve been prohibitively costly for us to collect. \n\nIn conclusion, we believe it’s important to be transparent with our findings. **We will add the following to our paper**:\n\n- The additional human evaluation results (increased sample size, agreement, and switch to davinci end-to-end) to Table 5\n- The statistical analysis in Section 5.5 when interpreting results\n- A more clear discussion of the sample size issue in the Limitations\n- Add discussion about scalable human evaluation as an important future challenge that NLP research must tackle (similar scalable issues for human evaluation are being seen in other generative NLP areas, like summarization). Our task can hopefully be yet another motivating example for more investment in this important area.\n\n****\n\n**_Concern 4:_** Reviewer uGwc wonders why CLF seems not to be correlated with the other metrics.\n\n**_Response 4:_** We would not necessarily expect the CLF metric to be the same as the others. CLF measures the precision, recall, and F1 of the added clarifications, rather than added tokens. For example, if a decontextualization has two clarifications, one short (e.g. expanding an acronym) and one long (e.g. describing the method of a cited paper) and the prediction includes tokens in the longer one, but misses the shorter one, the SARI score would be high, but the CLF score would be low. Cases like these lead to the discrepancies. We will add this clarification to our discussion about CLF.\n\n****\n\nThe authors would like to thank Reviewer uGwc for their detailed feedback, questions, and corrections. We hope these updates address the main concerns with the paper, and if so, we hope the reviewer updates their scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an in-depth evaluation and ablation studies for using LLMs to do QA-QG-rewrite based decontextualization. The authors show the performance gap between using gold annotated question, answer, evidence vs. the model generated ones, where the later setting is more realistic. The author shows that a QA-QG-rewrite pipeline system can perform better than end-to-end system in 0-shot settings. The conceptual idea of the method is very interesting.", "reasons_to_reject": "Since decontextualization is an established task, IMO the evaluation in the paper could be greatly strengthened by comparing to an established baseline, e.g. a finetuned seq2seq system with dataset from Choi et. al. 2021. This will --\n1. Help explain the conceptual differences between the decontextualization task for Wikipedia vs. \"user-facing\" scenarios, e.g. scientific documents, in a more quantitative way.\n2. Help readers/users understand the capability of using LLM for decontextualization vs. supervised baselines.\n\nIMO there are two ways to go about this, either --\n1. Test the QA-QG-rewrite pipeline on the Choi et. al. 2021 test set (i.e. Wikipedia sentences), and compare to the numbers there, or\n2. Train a supervised baseline from Choi et. al. 2021, and test on your data.\n\nApart from baseline choices, here are some additional comments on evaluation settings\n- It seems that all LLM prompts are used in zero-shot setting without providing in-context examples. I understand that this is needed for fitting an entire paper in the context window of a single prompt forward. But this seems really disadvantageous for the end2end model, as it's a relatively complex and procedural task. IMO using even a single demonstration could make the performance much better. It's worth dedicating some discussion + analysis to compare the performance in zero- vs. few-shot settings (even if this can only be done on a subset of the dataset)\n-  Since the human evaluation is done only on 30 snippets, it's worth checking if the improvements are statistically significant.", "questions_to_authors": "I'm curious how InstructGPT (davinci) vs. GPT4 looks like in the end2end setting? From the human evaluations, it seems that GPT-4 is selected the best end2end system, but it wasn't clear from Table 2. \n\nOn a related note, it seems that CLF scores don't have a strong correlation with other metrics. It might be worth adding a bit more discussion and analysis to help explain why that is.", "paper_topic_and_main_contributions": "The paper introduces a question-answering based framework for de-contextualization -- the task of adding necessary context to a sentence from the document it appears in to make the sentence standalone. The proposed framework features a three-stage pipeline -- Given a sentence, the framework (1) generates clarifying questions, (2) Extract answers to the questions from other sentences in a document, and (3) De-contextualize and rewrite the sentence based on the answers. \n\nThe paper studies the problem in the domain of scientific documents, and conducts a small human study to analyze the needs and challenges of decontextualization from a user perspective. The proposed system uses different combinations of LLMs for the three parts of the pipeline. The authors provide evaluation and ablation for the overall system.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The overall presentation of the paper was clear and easy to follow."}, "extra_text": {}}, {"review_id": "rhyUl0XXl8", "reviewer": "Reviewer_BSQ8", "summary": "", "questions": "\n\n* Authors need to put substantial efforts in rewriting the paper in order to enhance readability. I ended up reading the abstract very confused, for example. There are also some grammar error across the manuscript.\n\na) \"Many real-world applications (e.g., note taking, search) require extracting a sentence or paragraph from a document and showing that snippet to a human outside of the source document.\" This phrase needs a little of rephrasing as a means of enhancing readability.\n\n\nb) \"Yet, snippets divorced from their origin are often missing context and can be in comprehensible when presented to users\" Same here, maybe a better crafting is as follows: \"However, these snippets can be hard to understand [for a human reader] when presented outside from their context\"\n\n\nc) Many sentences are very long thus take some extra effort to read and understand. Consider splitting and linking properly.\n\nd) \"Second, we propose a framework that decomposes the task into question generation, question answering, and rewriting stages.\"\nAfter reading this sentence, I ask myself \"what are the authors decomposing?\"\n\n\ne) \"Using this framework, we collect gold decontextualizations from experienced readers of scientific articles and conduct a range of experiments across state-of-the-art commercial and open-source language models to identify how best to provide missing-but-relevant in formation to models for our task.\" This sentence needs rephrasing and splitting.\n\nf) In the definition (lines 115-119), it is explicitly defined \"c\".\nIt looks like that snippet is wrongly \"decontextualized\".\n\ng) \"Finally, to handle cross-document references, we assume in the QA application setting that a user would have access to basic information about the current document of interest but no knowledge  about any referenced documents that may appear in the snippet text.\" This is not realistic, especially if a ground-breaking work was cited like figure 1.\n\nh) Some examples of each of the three steps should be added to figure 2 for the sake of clarity\n\ni) \"A clarification matches if at least 75% of the added tokens in the prediction match one of the targets\" .. What about highly frequent terms such as play, experiment, and results. Please answer as it relates to the CLF metrics.\n\nj) \"Surprisingly, the QA module of the pipeline is the weakest.\"\nWhy was it a surprise? I think it is the hardest component and its gold standard provides lot of new pieces of information. I would say \"as a conclusion\" instead of \"Surprisingly\"\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors would like to thank Reviewer BSQ8 for their thorough and careful review. We appreciate that the reviewer believes the experiments are “clean” and that “the problem is challenging and interesting”. \n\n****\n\n**_Concern 1:_**  Lack of significance testing reported for human evaluation.\n\n**_Response 1:_** We have performed these tests on an expanded human evaluation. To avoid too much overlap in our rebuttals, please see our **Response 3 to Reviewer uGwc** for details.  \n\n****\n\n**_Concern 2:_**  Concerns about our clarity of writing.\n\n**_Response 2:_** We appreciate the suggestions to improve our writing clarity. **We edited the sentences surfaced by the Reviewer,** as well as others that are long and difficult to read. (Addressing the reviewer’s questions “a-f” and “j”)\n\nAdditionally, the reviewer suggests adding examples of the outputs of each step of the pipeline to Figure 2 (h). We agree that adding examples will clarify the paper. **We will use our additional camera ready page to add a new Table with an example generation from each step to the main paper.** This Table will also serve for illustrating error analysis (see Reviewer’s concern #4 below). Additionally, **we will add more to the Appendix of the paper**. \n\n****\n\n**_Concern 3:_** Reviewer BSQ8 asks how common words might skew the CLF metric.\n\n**_Response 3:_** Like many n-gram based scores (such as SARI), we do not explicitly weigh the token types by frequency when calculating CLF. In preliminary evaluations, we found that removing stop words were sufficient for using CLF to distinguish between systems.\n\n****\n\n**_Concern 4:_** More In-depth error analysis for the pipeline.\n\n**_Response 4:_** We have performed this analysis and agree the results would be informative for readers. We plan to include it in a new Table in the main body as part of our additional camera ready page. We iterated on model prompts to minimize these errors, but still some occurred. We describe our findings for each pipeline step below:\n\n**Question Generation:**\n\n- Asking about Citations\n\n  - E.g. Snippet: “While Shen et al. (2019) makes RNN decoder as a MoE…”\n  - Question: “Who are Shen et al. (2019)?\"\n\n- Asking questions that are answered in the text.\n\n  - E.g. Snippet: “In addition, our system is independent of any external resources, such as MT systems or dictionaries, as opposed to the work by Kranias and Samiotou (2004).”\n  - Question: “​​What kind of external resources were used by Kranias and Samiotou (2004)?”\n\n- Asking questions that an expert likely wouldn’t ask:\n\n  - Snippet: “Concretely, we apply the … model trained on the full Reddit dataset to millions of new unannotated posts, labeling these posts with a probability of dogmatism according to the classifier.”\n   - Question: \"How is the classifier labeling posts with a probability of dogmatism?\"\n\n\n**Question Answering** \n\nThere were two main sources of errors in the QA portion of the pipeline that can be broken down into two categories:\n\n- Retrieval Errors: The answer **was not** in the retrieved context, so the model just made something up rather than abstaining.\n- QA Errors: The answer **was** in the retrieved context, but the model still answered the question incorrectly or unnecessarily abstained.\n\n\n**Rewriting:**\n\n- In fact, we have some of this analysis already in Table 5, when we report on human evaluation. To make this more prominent, we will move out this content from Table 5, leaving it to focus solely on the accept/reject judgment component of the human evaluation. The error analysis will go into the new table with the other pipeline errors.\n- Our findings are still what we report in our submission: Most rewriting errors are **formatting** (usually not putting the added information between brackets, which is a requirement of user-facing snippets) or **missing information** (the system missing important information that is in the gold decontextualization).\n\n****\n\nThank you again for your close reading and thoughtful reviews of our paper. We hope that the changes we have made sufficiently address the reviewers’ comments, that the reviewer increases their scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n* The experimental section is nice and clean. They provide nice features in their analysis. \n\n* The problem is challenging and interesting. \n\n*The paper can be understood by a wider audience.\n\n* Results provides insight into how to move forward in this task.\n\n\n\n\n\n\n", "reasons_to_reject": "* Error analysis and significance tests are missing.\n\n* The studied problem might be of interest of a narrow audience.\n\n* The English needs improvements.\n\n", "questions_to_authors": "\n\n* Authors need to put substantial efforts in rewriting the paper in order to enhance readability. I ended up reading the abstract very confused, for example. There are also some grammar error across the manuscript.\n\na) \"Many real-world applications (e.g., note taking, search) require extracting a sentence or paragraph from a document and showing that snippet to a human outside of the source document.\" This phrase needs a little of rephrasing as a means of enhancing readability.\n\n\nb) \"Yet, snippets divorced from their origin are often missing context and can be in comprehensible when presented to users\" Same here, maybe a better crafting is as follows: \"However, these snippets can be hard to understand [for a human reader] when presented outside from their context\"\n\n\nc) Many sentences are very long thus take some extra effort to read and understand. Consider splitting and linking properly.\n\nd) \"Second, we propose a framework that decomposes the task into question generation, question answering, and rewriting stages.\"\nAfter reading this sentence, I ask myself \"what are the authors decomposing?\"\n\n\ne) \"Using this framework, we collect gold decontextualizations from experienced readers of scientific articles and conduct a range of experiments across state-of-the-art commercial and open-source language models to identify how best to provide missing-but-relevant in formation to models for our task.\" This sentence needs rephrasing and splitting.\n\nf) In the definition (lines 115-119), it is explicitly defined \"c\".\nIt looks like that snippet is wrongly \"decontextualized\".\n\ng) \"Finally, to handle cross-document references, we assume in the QA application setting that a user would have access to basic information about the current document of interest but no knowledge  about any referenced documents that may appear in the snippet text.\" This is not realistic, especially if a ground-breaking work was cited like figure 1.\n\nh) Some examples of each of the three steps should be added to figure 2 for the sake of clarity\n\ni) \"A clarification matches if at least 75% of the added tokens in the prediction match one of the targets\" .. What about highly frequent terms such as play, experiment, and results. Please answer as it relates to the CLF metrics.\n\nj) \"Surprisingly, the QA module of the pipeline is the weakest.\"\nWhy was it a surprise? I think it is the hardest component and its gold standard provides lot of new pieces of information. I would say \"as a conclusion\" instead of \"Surprisingly\"\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "paper_topic_and_main_contributions": "this paper proposes a strategy to decontextualize scientific documents. In so doing, their framework divide the challenge into question generation, answering and rewriting. The authors also designed a gold standard for their study. \n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Justification_for_Ethical_Concerns": "The same listed on page 9. In particular, any kind of false/error in the inferences might lead to some legal/ethical issues such as attributing false claims to some authors, or imprecise conclusions that can lead other studies to something wrong.\n"}}]

Author rebuttal:

Meta-review: This paper tackles decontextualization through the lens of question generation and question answering. Decontextualization is a relatively new but important task; the goal is to rewrite an extractive (short) text snippet such that it "stands alone". This work builds on prior work (a seq2seq system from Choi et al), and introduces a 3-step process with question generation, question answering, and rewriting. This framework, inspired by prior work in QUD, is compelling and useful in this task. This work involves an annotated dataset and sound evaluation of LLMs.

The reviewers pointed out several issues to be addressed; notably please include the answers in the author response to reviewer questions, such as new results re LLAMA2, additional human evaluation, additional error analysis, and other clarifications.
Decision: Accept-Main