Concise Answers to Complex Questions: Summarization of Long-form Answers
Year: 2023

Abstract:
Long-form question answering systems provide rich information by presenting paragraph-level answers, often containing optional background or auxiliary information. While such comprehensive answers are helpful, not all information is required to answer the question (e.g. users with domain knowledge do not need an explanation of background). Can we provide a concise version of the answer by summarizing it, while still addressing the question? We conduct a user study on summarized answers generated from state-of-the-art models and our newly proposed extract-and-decontextualize approach. We find a large proportion of long-form answers (over 90%) in the ELI5 domain can be adequately summarized by at least one system, while complex and implicit answers are challenging to compress. We observe that decontextualization improves the quality of the extractive summary, exemplifying its potential in the summarization task. To promote future work, we provide an extractive summarization dataset covering 1K long-form answers and our user study annotations. Together, we present the first study on summarizing long-form answers, taking a step forward for QA agents that can provide answers at multiple granularities.

heading: 1 Introduction
text: Long-form answers (Fan et al., 2019), as compared to span-based short answers (Rajpurkar et al., 2016), can provide comprehensive answers to a broader set of questions (Cao and Wang, 2021; Fan et al., 2019). While providing comprehensive information in multiple sentences is helpful, users often prefer short and concise answers to their questions when possible (Choi et al., 2021). Today’s search engines already present concise answers by highlighting the most relevant parts from the passage excerpts. In this paper, we present the first study on summarizing long-form answers.
∗Equal contribution.
Summarizing long-form answers introduces a new challenge in addition to the faithfulness and fluency challenges of generic summarization which mostly focus on news articles (Nallapati et al., 2016; Narayan et al., 2018): the summary output should still provide a reasonable answer to the original question. We take inspiration from a recent study (Xu et al., 2022) that reports that up to 40% of sentences in long-form answers contain non-essential information, such as providing background information or examples (Wang et al., 2022), which demonstrates the potential for compressing long-form answer.
We first aim for an extractive summarization model and collect sentence-level annotations on long-form answers, where annotators identify sentences that address the question directly and can serve as the “summary”.1 We collect a dataset covering 1,134 examples, each consisting of a question, a long-form answer, and a set of summary sentences. To improve the extractive summaries collected, we propose a simple and yet novel summarization approach, extract-and-decontextualize, which first extracts summary sentences and rewrites them to stand-alone (Choi et al., 2021). Compared to abstractive summarization models trained on noisy distantly supervised datasets (e.g. CNN/DM (Nallapati et al., 2016) and XSum (Narayan et al., 2018)) which encourage paraphrasing but also hallucinations (Kryscinski et al., 2019; Cao et al., 2018; Kang and Hashimoto, 2020), decontextualization makes minimal edits to the original sentence, preserving its meaning while improving its fluency.
How well do summarization approaches perform in this new domain – can generated summaries provide fluent, adequate answers to the questions, while preserving the semantics of the original longform answers? We evaluate fine-tuned abstrac-
1This is a simplified annotation task compared to the original discourse study of Xu et al. (2022).
9709
systems, and human evaluation results ("summary adequacy" and "faithfulness"). We highlight the gold extractive summaries we collected.
tive summarization model (Zhang et al., 2019), prompted large language model (GPT-3) (Brown et al., 2020), and our extract-and-decontextualize approach with a user study. Table 1 shows two examples from our user study. We find vanilla extractive approach, even with gold sentences, presents inadequate summaries but decontextualizing them makes them on par with GPT-3 abstractive answers. While none of the systems consistently present high-quality summaries (GPT-3 records a 67% success rate), most questions (95%) have at least one system that can generate a valid summary, showing the potential for successful compression of long-form answers. Together, we present the first corpus and study on summarizing longform answers, opening doors for developing more flexible QA systems which provide answers with varying amounts of information. We release our data, code, and user study templates at https: //github.com/acpotluri/lfqa_summary.

heading: 2 Background and Motivation
text: The focus of our study is to find a concise answer to a complex question (Fan et al., 2019). One way to
generate a concise answer is through controllable generation, where the long-form QA model is instructed to generate an answer given a pre-specified length. However, long-form question answering remains challenging, both in terms of modeling (Krishna et al., 2021) and reliable evaluation (Xu et al., 2023). Existing models often hallucinate (Krishna et al., 2021; Liu et al., 2023) even when paired with relevant evidence documents. Instead of generating a concise answer from scratch, we summarize an existing long-form answer, leveraging a large amount of user-written long-form answers often in community-driven QA forums like ELI5 in Reddit.
How feasible would it be to summarize existing long-form answers? Xu et al. (2022) conducted an in-depth study on the structure of such longform answers, assigning one of six functional roles (answer, answer summary, organizational sentence, auxiliary information, and example) to each sentence in long-form answer. The study suggests sentences corresponding to “answer summary" captures the salient information and “often suffice by themselves as the answer to the question." Furthermore, they suggest up to 40% of sentences belongs
to roles (e.g., auxiliary information) that are not necessary to answer the question, suggesting summarizing existing answer is viable. We follow their study and collect larger-scale data focusing on the “answer summary" role to study the summarization of long-form answers.
Summarizing existing answers will support providing a consistent answer set of different granularities, where the users can expand condensed answer to see a more detailed version of the same answer. Consistent answers at multiple granularities are harder to enforce with a controllable generation approach. For instance, if we generate a five-sentence answer from the raw evidence set, the five-sentence answer can contain information absent in the tensentence answer.
Lastly, retrieval-augmented long-form QA models (Nakano et al., 2021) resemble query-focused summarization. Query-focused summarization (Xu and Lapata, 2020; Kulkarni et al., 2020) often studies challenging multi-document settings, where the input text is summarized focusing on a particular query, provided at inference time content control. A difference to our setting is that a long-form answer is written for the question q, presenting already synthesized information tailored for the question.2

heading: 3 Extractive Summary for Long-form Answers
text: We first introduce our annotation task of identifying key sentences for long-form answers, which will be used as an extractive summary. Extractive summaries allow easier data collection and evaluation but can suffer from disfluency and incoherence. Thus, we manually evaluate our collected gold extractive summaries in Section 5.

heading: 3.1 Task
text: Given a question q and its long-form answer consisting of n sentences a1, a2, ...an, the model makes a binary decision on whether each sentence ai should be included in the summary. This setup differs from general summarization in having question q as an additional input.

heading: 3.2 Source Data
text: We use long-form answer data, (question, answer) pairs, from prior study (Xu et al., 2022) which
2This is true for two out of three datasets (ELI5/WebGPT, 82% of our data) we study. In NQ, the paragraphs are written independently, representing the QFS setting.
compiled three existing LFQA datasets. ELI5 (Fan et al., 2019) consists of question answer pairs extracted from the subreddit Explain Like I’m Five. Natural Questions (NQ) (Kwiatkowski et al., 2019): NQ contains Google search queries as the questions, paired with paragraph-level answers from Wikipedia passages identified by annotators. WebGPT (Nakano et al., 2021) contains answers written by trained human annotators, with the questions sourced from ELI5. The annotator first searches for related documents using a search engine and then constructs the answers with direct references to those documents. We only take answers that passed their validity annotation, which excludes questions with false presupposition, illformed queries, and answers that do not provide valid answers. Their preprocessing step also filters answers with more than 15 sentences or less than 3 sentences.

heading: 3.3 Annotation Task
text: Given a question and its long-form answer, annotators select a set of summary sentences containing salient information addressing the question. The annotator interface and instructions are in the appendix. As saliency is somewhat subjective, we collect three-way annotations for each example. We recruited crowd workers from Amazon Mechanical Turk. We recruited workers from Englishspeaking countries, with at least a 95% acceptance rate on 1000+ HITs. Each worker was paid $0.50 per annotation, translating to an hourly rate of $15. We recorded reasonable agreement (Fleiss’ Kappa 0.53) for the annotations.3

heading: 3.4 Dataset Statistics
text: Table 2 contains our collected dataset statistics, comparing it to a popular news summarization dataset (Nallapati et al., 2016) and a query-focused summarization dataset, AQuaMuSE (Kulkarni et al., 2020). To compute the summary length in our dataset, we randomly choose one of three summary annotations. The average number of sentences chosen as summaries by a single annotator was 1.6 out of 6.2 sentences in long-form answers. The statistics show that our data handles shorter texts and compress less than existing datasets. On average,
3Xu et al. (2022) hired expert annotators (undergraduate linguistics students), as they required annotators to provide sentence-level labels among six functional roles. The expert annotators reached a similar agreement (0.52 Fleiss’ kappa) for the “summary” role.
long-form answers were compressed to about onethird of their original length, with a slightly higher compression rate for ELI5 answers. This aligns with the prior discourse study (Xu et al., 2022) which reports ELI5 contains sentences that serve other functional roles (like providing an example) more frequently (23% compared to 5% and 8% in NQ/WebGPT datasets), neither of which are likely to be included in the summary.

heading: 3.5 Automatic Extractive Summarization
text: Having collected a new dataset, we evaluate existing extractive summarization models on it. Is it easy for models to identify key sentences from long-form answers?
Setting We aggregate all data from three datasets (ELI5, NQ, WebGPT) and split them into 70% train, 15% validation, and 15% test set. We report classification metrics (precision, recall, F1 scores) with summary sentences being the positive class. For each long-form answer, metrics are computed against each of the three references, with the results from the reference with the maximum F1 score reported. We also report exact-match (EM), whether the model-predicted summary sentence set matches any of the three annotations. The training details and hyperparameters can be found in Appendix B.
PreSumm We use PreSumm (Liu and Lapata, 2019), a BERT-based extractive summarization model, which was trained on the CNN/DailyMail (Nallapati et al., 2016) dataset. It encodes the document with pre-trained BERT (Devlin et al., 2018) and outputs a score for each sentence. We select a threshold for the score at which it is considered a summary sentence to maximize
the F1 score on the validation set. We evaluate both the original model (trained on CNN/DM dataset) and the model fine-tuned on our dataset.
T5 We use a sequence-to-sequence model, T5large (Raffel et al., 2019), to classify whether a sentence belongs to the summary or not. This was the best performing model for fine-grained role classification of long-form answers in Xu et al. (2022). For question prepending input, the input sequence to the model would be: [q [1] a1 [2] a2 ... [n] an]. The output sentence would then be of the form: [[1] r1 [2] r2 ... [n] rn], where ri was a binary class label whether i-th answer sentence ai belongs to the summary or not.
Results Table 3 reports model performances on the test set. The result on the validation set can be found in Table 8 in the appendix. With in-domain fine-tuning, both models are able to accurately predict which sentences belong to the summary. Fine-tuned T5 model shows a strong performance, though underperforming human, especially in exact match. We also find all trained classifiers benefit from having questions as additional input, signifying that questions provide important signals for content selection. While there is room for improvement, results suggest that predicting key sentence sets is not a major hurdle for state-of-the-art language models. Thus, we use the gold extractive summary for our user study (Section 5).

heading: 4 Abstractive Summaries for Long form Answers
text: While we have gold extractive summaries at hand, they often suffer from disfluencies and factual errors (Zhang et al., 2022). We aim to improve this in two ways, (1) by introducing a decontextualization (Choi et al., 2021) model to edit extractive summaries and (2) by using abstractive summarization
models. We explore zero-shot transfer from an abstractive summarization model (Zhang et al., 2019) and prompting an instruction-tuned large language model (Brown et al., 2020). We experiment with two types of input sequences: (1) long-form answer only as an input (2) the question followed by a separation token and the long-form answer, whenever applicable. In the latter setting, models sometimes output the question as a part of the summary, which we remove with postprocessing.4

heading: 4.1 Editing Extractive Summary with Decontextualization
text: The disfluencies and lack of coherence of extractive summaries are well-known issues, motivating a flurry of abstractive summarization models (Rush et al., 2015; See et al., 2017). While abstractive models can provide coherent and fluent summaries, one of their major issues is hallucination (Kryscinski et al., 2019; Cao et al., 2018). Recent work explores extract-and-abstract approaches (Hsu et al., 2018; Liu et al., 2018; Pilault et al., 2020), aiming to take the best of both worlds. Most of these approaches are fine-tuned on an abstractive summarization dataset. As we don’t have an abstractive summary of long-form answers at hand, we opt to use a decontextualization model to rewrite the extractive summary.
Decontextualization (Choi et al., 2021) is a text editing task, which aims to rewrite the target sentence in a document such that the edited target sentence can be interpreted when presented alone while preserving its meaning. While its use cases in QA and text retrieval (Gao et al., 2022) have been explored, its use case in summarization has not been explored. Earlier prior work (Clarke and Lapata, 2010; Durrett et al., 2016) have studied discourse constraints for summarization – that for each pronoun included in the summary, the pronoun’s antecedent should be included or the pronoun to be rewritten as a full mention to make summary coherent and clear. Decontextualization is well-suited to prevent these common errors of pronouns/concepts being “orphaned" in extractive summary.
4For extractive models, we exclude the question if it is chosen as the summary. For abstractive models, we remove the first sentence of the summary if it has high lexical overlap (over 75% unigram overlap) with the question (which happened for roughly 38% of the dataset).
Method We use an off-the-shelf decontextualization system from recent work (Chen et al., 2021),5 which trained a T5 3B model on the original decontextualization dataset (Choi et al., 2021) on Wikipedia text. This model takes the concatenation of the Wikipedia page title and a paragraph with the sentence to be decontextualized as input. For ELI5 and WebGPT answers which lack a page title, we consider the question as the title.
If the title is t and the answer consists of k sentences [a1, a2, . . . , ak] with the i-th sentence being the target to be decontextualized, the input will be formatted as:
[CLS] t [s] a1 . . . ai−1 [s] ai [s] ai+1 . . . ak[s]
where [CLS] is a start token and [s] is a separator token. The model outputs the sequence: [CATEGORY] [SEP] y, where the category is one of DONE (if it made edits to the sentence in which case y would be the new sentence), Unnecessary (the sentence does not need an edit, already standalone), or Infeasible (the sentence is tricky to be made stand-alone with minimal edits).6 We only apply decontextualization when the first sentence in the extractive summary is not included in the summary set (56% of examples in the dataset), and only decontextualize the first summary sentence.
Decontexutalization Results Table 4 presents basic statistics of the output from decontextualization model. Somewhat surprisingly, the decontextualization model edited only 17.1% of input examples, diverging significantly from its training distribution where 60% of examples
5https://github.com/jifan-chen/ QA-Verification-Via-NLI/.
6In the case of infeasible and unnecessary cases, y would just be the same as ai).
are edited. For these edited sentences, we report the length increase (∆), or the average value of (len(decontext)-len(original)) / len(original), following the original study. While decontextualization is attempted less frequently when it is decontextualized the length of the sentence increases more substantially. More ELI5 sentences were classified as Infeasible. We hypothesize that the sentences in ELI5 could be deemed more challenging because of the narrative nature of Reddit posts. We include sample decontextualization outputs in Table 9 in the appendix.
We manually examine decontextualization outputs from ELI5 and Web-GPT to evaluate their performance on out-of-domain, non-Wikipedia texts. We (the authors of this paper) randomly sample 50 examples where the model has made changes, and 50 examples from the entire set. Out of 50 edits, 42 edits were meaning preserving (without introducing factually incorrect contents), and 44 edits successfully decontextualized the sentence (without unresolved or unclear references). On a randomly sampled set of 50 examples, we evaluate whether the category assigned is correct (infeasible, unnecessary, done), finding 45 examples were assigned the correct category. Overall, we found the zero-shot performance of the decontextualization system on the new domain was surprisingly robust. Recent work (Eisenstein et al., 2022) also showed large language model can perform decontextualization robustly when prompted carefully. We will evaluate decontextualized summaries with a user study in Section 5.

heading: 4.2 Abstractive Models
text: In this section, we explore abstractive models for summarization to improve fluency.
Pegasus (Zhang et al., 2019) shows promising performance across diverse summarization benchmarks. We examine a suite of Pegasus fine-tuned on various summarization datasets and chose a model fine-tuned on the CNN/DailyMail as it showed the most promising results upon manual inspection. We do not fine-tune it with our extractive dataset to preserve its abstract nature.
GPT-3 Recent work (Goyal et al., 2022a) has found that GPT-3 (Brown et al., 2020) exhibits strong zero-shot performance on several news summarization benchmarks. Unlike fine-tuned abstractive models, prompted language models would not
inherit issues from noisy distant supervision training datasets. Thus, we investigate its ability to perform zero-shot long-form answer summarization. Specifically, we used the text-davinci-002 model.7 We explore two settings: with and without length control in the prompt, following prior work (Goyal et al., 2022a). The prompt with length control is “Q: {question text} A: {answer text} Summarize the above answer in {length of gold summary} sentences”, and the prompt without length control is “Q: {question text} A: {answer text} Summarize the above answer.”

heading: 4.3 Automatic Evaluation
text: We first aim to perform an automatic evaluation of abstractive systems, using gold extractive summaries as references. While this would not evaluate fluency, automatic metrics measure the content selection of generated abstractive summaries.
Setting We use the same data split as in Section 3.5, and repeat lead baselines: LEAD-2 and LEAD-3. We use established automatic summarization evaluation metrics ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020).8 As our dataset is 3-way annotated, we report the highest ROUGE-L F1 score among the three reference answers and use the same reference answer to compute BERTScore F1. The Human baseline is computed by choosing one extractive summary annotation at random as the reference and doing a pairwise computation of ROUGE and BERTScore with the other two annotations for that example.
Results Table 5 reports model performances on the test set. The results on the development set are in Table 7 in the appendix. Similar to other domains, lead baselines show strong performances, outperforming models trained on out-of-domain data (Pegasus, GPT3). Yet, they are inherently limited, covering only 73% of the summary sentences. We see that the abstractive models show better performance with the BERTScore metric compared to the ROUGE-L metric, potentially due to the ROUGE-L metric punishing for paraphrasing. Having the question in addition to the answer improves the performance of the Pegasus model. Having length control also improves the zero-shot performance of GPT-3, similar to the finding from
7We set the max generation length to 512 tokens and temperature to 0. The generations were queried on October 19, 2022.
8We use the bert-base-uncased checkpoint.
prior work (Goyal et al., 2022b). This is a semioracle setting as the model is given the summary length.

heading: 5 Human Evaluation of Summary Answers
text: So far we have evaluated summarized answers against the gold extractive summary. Yet, we are aware extractive answers themselves are limited and automatic evaluation of summary is non-trivial. To properly evaluate summarizing long-form answers, we launch a user study evaluating four different types of answer summaries: a gold extractive summary, a gold extractive summary that is decontextualized, an abstract summary from Pegasus, and an abstract summary from GPT3. Can the summarized answer present a useful, concise answer that preserves the original meaning of the long-form answer, without producing incoherent discourse structure (e.g., orphaned anaphora)?

heading: 5.1 User Study Design
text: We design a two-stage interface to evaluate the summarized answer. The exact wording and interface can be found in the appendix (Figures 5, 6, 7, and 8). First, they are shown the summary answer and the question alone, and then, the original long-form answer will be shown to them.
Stage 1: The annotators first measure the quality of the summary answer itself. FLUENCY (choices: Yes/No): if the answer is grammatical and fluent. We do not distinguish coherence and fluency as prior study (Fabbri et al., 2021)
reports that annotators often confuse those two dimensions. ADEQUACY (choices: Yes/Partially/No): if the summary adequately answers the original question.
Stage 2: The annotators then measure both the summary and original long-form answer. FAITHFULNESS (choices: Yes/No): if the summary accurately captures the main idea of a long-form answer regarding the question. LONG-ANSWER ADEQUACY (choices: Yes/Partially/No): if the long-form answer addresses the question adequately. This annotation only evaluates the original long-form answer, as a control to avoid blaming the summarization system when the long answer itself is not adequate. As we filtered out invalid long answers during pre-processing, most answers should be labeled as adequate.

heading: 5.2 User Study Setting
text: Data We annotate 175 long-form answers paired with four types of summary: (1) summary generated from our best abstractive model (Pegasus), (2) gold extractive summary (GOLD), (3) gold extractive summary that is decontextualized with automatic decontextualizer system (GOLD++) and (4) GPT-3 zero shot summaries with length restriction. We sample 150 examples at random and additionally sample 25 examples where the decontextualization process made edits to the gold extractive summary.
The average length of the tokens for the four summary settings were 43.4, 40.9, 47.6, and 31.3 for Pegasus,GOLD,GOLD++,GPT3.
Annotators Human evaluation was done on the Amazon Mechanical Turk platform. We required the workers to be from English-speaking countries and have at least a 95% acceptance rate on 1000+ HITs. Each worker was paid $0.50 per annotation, translating to an hourly rate of $15. We set up the task that each annotator will see only one variant of the summary per each long-form answer. The annotators were not aware of which summarization system provided the summary. A small subset of data is annotated by the authors, following the same setup. We had 561 unique annotators for this task.

heading: 5.3 Results
text: Table 6 presents the results from the user study. We report two numbers – one on all 175 examples, and one on a subset of 63 examples where decontextualization changed the extractive summary.
We include the inter-annotator agreement for each question in the first row. We observed moderate to high agreement for all four questions. Evaluating the quality of answers (summary adequacy and long answer adequacy) was more subjective than evaluating fluency or faithfulness, revealing the challenge of open-ended long-form answer evaluation as pointed out in prior work (Krishna et al., 2021). We also see high agreement among annotators by comparing long answer adequacy distributions across four rows, which are very similar as expected.
Can a summarized version of long-form answers provide an adequate answer to the original question? We see somewhat mixed results – while the annotators said the summaries provide at least a partial answer to the question most of the time (over 90%), only about 60% of answers per system provide adequate answers. Again, we find that decontextualization helps – on about 10% examples, annotators labeled extractive answers as partially adequate, but their decontextualized versions are adequate.9 GPT-3 produces adequate summaries the most, showcasing its powerful zero-shot summarization ability (Goyal et al., 2022a). Further analysis showed that summary adequacy is highly system dependent rather than question dependent – for 90% of the questions, there is at least one system whose outputs are adequate according to the majority of the annotators.
We find fluency is not a major issue, both for extractive and abstractive systems. The large-scale language model (GPT3), in particular, provides the most fluent answers. For the extractive summaries, we see a substantial gain (about 10% on 63 examples where decontextualization changed the input) in fluency by introducing contextualization. The
9This difference was also statistically significant with a t-test where Yes/Partially/No maps to a (1.0/0.5/0.0) score.
fluency gap between Gold and Gold++ was statistically significant on McNemar’s test with p < 0.05.
We observe a slightly lower performance on faithfulness across four summary systems compared to fluency. While the weaker abstractive model (Pegasus) ranks slightly lower than the extractive model, GPT-3 somewhat surprisingly outperforms extractive approaches in meaning preservation. This mirrors findings from a recent study (Zhang et al., 2022) about how extractive summary can also introduce factual errors. Overall, faithfulness has been extensively studied in summarization literature (Fabbri et al., 2022a) but mostly in the news domain.
When can we use summarized answers? In the last column, we report the percentage of summary answers that are fluent, adequate, and faithful to the original long-form answer. Decontextualized answers (GOLD++) and GPT-3 zero-shot summary achieve more promising results than the other two approaches. Of the 168 long answers considered “adequate” by a majority of the annotators, 160 (95%) of them has at least one summary that was considered functional by a majority of the annotators. We examine error cases in the next section.

heading: 5.4 What makes it hard for models to summarize long-form answers?
text: As we have identified fluency as a minor issue, we specifically look at 60 examples that satisfy all the following conditions: (1) summary is fluent, (2) summary answer is not fully adequate nor faithful, and (3) long-form answer is adequate.
We identify a few patterns of why the summary answers fall short: (1) around 10% of them contain summarization errors (e.g. not properly resolving anaphora or hallucination). (2) for around 60% of examples, adding a few more sentences to the summary was necessary to provide a coherent answer to the question. This is particularly true in cases
Summarization error
Q: Why do most restaurants sell Pepsi instead of Coke, and yet Coke is seen to be a bigger competitor? A: Coke sells way more soda by volume than Pepsi. As a response, Pepsi offers its products to restaurants at a reduced cost, which is why many restaurants carry it. But only up to midscale places – no nice restaurant serves Pepsi, because Coke has more cachėt, and also you need it for mixed drinks. Note also that McDonald’s, the single biggest restaurant chain in the world, serves Coke. Complex Answer
Q: How is it that the human brain/body sometimes wakes up seconds before an alarm goes off?! A: Your body does have internal regulation mechanisms, I’m not a doctor and there are plenty who are who can talk more intelligently about the circadian rhythm of the body etc. The other component is psychological. What’s happening is an example of confirmation bias. You’ve woken up a few times almost on the clock (relative to the total number of days you’ve ever slept in your life). Though this number is astronomical low, you only remember the times you did wake up on the minute. You bias yourself to count those times and subconsciously ignore the other times and thus you feel as though you have an ability to wake up on time. This also happens when people think that they can catch when people are looking at them. You sometimes do and sometimes don’t, but the times you don’t are not out of the ordinary so you forget them. Thus you only remember catching them and get a false sense of confirmation. GPT-3 summary: The human brain/body sometimes wakes up seconds before an alarm goes off because of the body’s internal regulation mechanisms and the psychological phenomenon of confirmation bias.
Figure 1: Examples with inadequate summaries: In the first example, the highlighted extractive summaries needs further decontextualization. In the second example, the long-form answer is too complex.
where the answers are multifaceted (e.g., providing multiple reasons for some phenomena, and the current summary contains only one of them). We also noticed a few cases where disclaimers (e.g., “I’m talking about poverty in U.S.") or counterexamples in the long-form answer that were not included in the summary, potentially misleading the readers. (3) some long-form answers (around 25%) are tricky to summarize without massive rewriting as it is explaining a complex procedure (e.g., why the Obama administration could not pass legislation, see the full example in Table 1). Figure 1 presents two representative failure cases. Future QA models can actively identify questions that require comprehensive v.s. concise answers.

heading: 6 Related Work
text: Query/Aspect-focused summarization Our task is relevant to query-focused summarization, which studies controllable summarization with respect to a query (Xu and Lapata, 2020; Deng et al., 2020; Zhu et al., 2020; Vig et al., 2021) or aspect (Angelidis et al., 2021; Hayashi et al., 2021; Ahuja et al., 2022; Kulkarni et al., 2020). Recently proposed MASH-QA (Zhu et al., 2020) dataset on the medical domain presents a question, context document, and extractive answer sentences. Compared to these works which summarize documents written independently of the question into a summary, we aim to compress long-form answers written with respect to the question. Another line of work (Fabbri et al., 2022b; Song et al., 2017) studies generating summaries of multiple answers to the same question. Lastly, Deng et al. (2019) looks into the
same task formulation of summarizing long-form answers, but their evaluation is limited to distantly supervised data.
Decontextualization for summarization Slobodkin et al. (2022) proposes the task of controllable text reduction, which rewrites chosen sentences from a document in a coherent manner using existing summarization datasets. They cover longer documents and involve multiple sentences to be decontextualized whereas we reuse a single-sentence decontextualization model (Choi et al., 2021).

heading: 7 Conclusion and Future Work
text: We present the first study on generating concise answers to complex questions. We collect an extractive summarization dataset in the new summarization domain of long-form answers to support future research. To address this new task, we deploy diverse summarization models, including zeroshot abstractive summarization models and a new decontextualization postprocessing method, which is applied to extractive summaries. Through our comprehensive user study, we find that around 70% of the summaries can serve as functional, concise answers to the original questions. Our work shows potential for building QA systems that generate answers at different granularities, as well as using decontextualization to improve the faithfulness and fluency of extractive summaries. Future work can also look into applying controllable generation techniques (Yang and Klein, 2021; Li et al., 2022; Qin et al., 2022) to generate answers with different lengths to generate concise answers.
Limitations
Our study is limited in scope, studying only English question-answering data. We also acknowledge that the long-form answers we study are not always factually correct, as they can be outdated (Zhang and Choi, 2021) or incorrect as they are crawled from web forums (Fan et al., 2019).
Further, our user study is limited in its scale, evaluating 175 instances, and does not carefully study potentially diverging interpretations from annotators of different demographics. We also do not extensively explore all summarization models, such as the extract-and-abstract approaches mentioned in related work.
Ethics Statement
Our data collection and user study protocols do not collect identifiable private information from annotators.
The question-answering data we annotated comes from an English online forum and might contain biased information. Our annotation is done by crowd-workers recruited from an online platform. We make use of pre-trained language models to generate abstractive summaries, which could suffer from hallucinating unfactual contents (Kang and Hashimoto, 2020) and perpetuating bias (Field et al., 2021). Thus, more post-processing steps are required before presenting these contents to users. Our user study shows that our proposed method, extract-and-decontextualize, could be one effective post-processing step to reduce hallucination.

heading: Acknowledgements
text: We thank Tanya Goyal, Jessy Li, Jiacheng Xu, and members of the UT Austin NLP community for their helpful feedback on the draft. We thank Jifan Chen for sharing the decontextualization model with us. We also thank the reviewers and metareviewer of the ACL community for helpful comments and feedback on the earlier draft of the paper. Lastly, we would like to thank the crowdworkers for their help with our data annotation and user study. The work is partially supported by a gift from Google Faculty Research Award.

heading: A Appendix
text: A.1 Summary Annotation Interface
Figure 3 presents the interface for summary annotation 3 and Figure 4 is the screenshot of the instruction presented to the annotators.
A.2 User Study Interface
Figures 5 and 6 are screenshots of the interface provided to the MTurkers who participated in the user study to analyze the quality of the summaries and Figures 7 and 8 are screenshots of the instructions provided with the corresponding steps.
A.3 Dataset Compression Statistics Figure 2 plots the token-level compression ratio (% of tokens included in the summary) on the three different types of long-form answers we study.

heading: B Model Training Details
text: All models are trained/evaluated on NVIDIA Quadro RTX 8000 GPUs. We use pytorch-transformers Wolf et al. (2019) to implement our models. The hyperparameters are manually searched by the authors.
PreSumm We use the checkpoint of BertSumExt from https://github.com/ nlpyang/PreSumm. We use the same hyperparameter in the original paper, using a batch size of 16 and a learning rate of 2e − 3. On two GPUs, fine-tuning on the training set and then evaluating on the test set takes between 1 to 2 hours.
T5 We use the T5-large checkpoint with 770 million parameters and fine-tune for 30 epochs with a batch size of 16 and learning rate of 1e−4. On two GPUs, fine-tuning on the training set and then evaluating on the test set takes between 2 to 3 hours.
B.1 Validation Set Results Tables 7 and 8 show our automatic evaluation results on the validation set for the extractive and abstractive models (computed in the same way that the test set values were).
B.2 Decontextualization Sample Output Table 9 gives three examples of the modifications that the decontextualization models made to the extractive gold label summaries.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section on Page 10, right after Section 6 (Conclusion)
3 A2. Did you discuss any potential risks of your work? Ethics Statement section on Page 10, after the Limitations section
3 A3. Do the abstract and introduction summarize the paper’s main claims? The abstract is on the first page and the Introduction is the first section
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? In section 3 we refer to a dataset collected in Xu et al. (2022) which provided input to collect our annotations and in section 4, we mention all the pre-trained models used and their sources.
3 B1. Did you cite the creators of artifacts you used? Sections 3.2 and 3.3 contain citations for the datasets used and Sections 4.1 and 4.2 contain citations for the models used.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? In section 1 we mention that we plan to open source all of our models, data, and annotations at time of publication, we will distribute with the CC BY-SA 4.0 license. Our code/data can be found at https://github.com/acpotluri/lfqa_summary/tree/main and https://huggingface.co/datasets/abhilashpotluri/lfqa_summary
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We didn’t explicitly discuss it in the paper but we only use publicly available models and question answering datasets and we build our dataset for research processes so it is compatible with the intentions of the existing artifacts.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? In sections 3 and 5 we discuss the data collection process (which was done through MTurk) and we also have screenshots of the annotation pages which show that we do not collect any personal information. We also have the annotation template for both the data and the user study in the public Github repository that we released.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In section 3.2 we discuss the datasets which our data is sourced from and their domains.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In section 3.4 and Table 2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Sections 3.5 and 4.3 along with Tables 4 and 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.1/4.2 has model details and Appendix B has the remaining model details and computing infrastructure/budget descriptions.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? The experimental setup is provided in section 4.3 and hyperparameters for fine-tuned models is in Appendix section B.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Experimental statistics for the test set are in section 4.3 and results on the development set are in the appendix.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 contains details of the models used for evaluation and any parameters which were set (as well as details of which evaluation packages were used).
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3 has the dataset annotation details and section 5 has the human evaluation study
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix sections A.1, A.2, and figures 3,4,5,6 contain full details and screenshots of the annotation instructions.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Sections 3.3 and 5.2 have the details of the participants for each annotation task and the hourly pay rates.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The instructions (which are attached in the appendix) for the annotation explain that we are trying to understand multi-sentence answers to complex queries for research purposes.
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We discuss this in the ethics statement.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Sections 3.3 and 5.2 have the details of the participants for each annotation task.

heading: Future Work
text: We collect an extractive summarization dataset in the new summarization domain of long-form answers to support future research. To address this new task, we deploy diverse summarization models, including zeroshot abstractive summarization models and a new decontextualization postprocessing method, which is applied to extractive summaries. Through our comprehensive user study, we find that around 70% of the summaries can serve as functional, concise answers to the original questions. Our work shows potential for building QA systems that generate answers at different granularities, as well as using decontextualization to improve the faithfulness and fluency of extractive summaries. Future work can also look into applying controllable generation techniques (Yang and Klein, 2021; Li et al. , 2022; Qin et al. , 2022) to generate answers with different lengths to generate concise answers. Limitations
Our study is limited in scope, studying only English question-answering data. We also acknowledge that the long-form answers we study are not always factually correct, as they can be outdated (Zhang and Choi, 2021) or incorrect as they are crawled from web forums (Fan et al. , 2019). Further, our user study is limited in its scale, evaluating 175 instances, and does not carefully study potentially diverging interpretations from annotators of different demographics. We also do not extensively explore all summarization models, such as the extract-and-abstract approaches mentioned in related work