SlowGuess/NAACL_2022_OCR huggingface dataset:
Alexander R. Fabbri\* Xiaojian Wu\* Srini Iyer\* Haoran Li\* Mona Diab
† Yale University ‡ Facebook AI
afabbri@salesforce.com
{xiaojianwu,sviyer,aimeeli,mdiab}@fb.com
Community Question Answering (CQA) for such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of community-based questions. Each question thread can receive a large number of answers with different perspectives. One goal of answer summarization is to produce a summary that reflects the range of answer perspectives. A major obstacle for this task is the absence of a dataset to provide supervision for producing such summaries. Recent works propose heuristics to create such data, but these are often noisy and do not cover all answer perspectives present. This work introduces a novel dataset of 4,631 CQA threads for answer summarization curated by professional linguists. Our pipeline gathers annotations for all subtasks of answer summarization, including relevant answer sentence selection, grouping these sentences based on perspectives, summarizing each perspective, and producing an overall summary. We analyze and benchmark state-of-the-art models on these subtasks and introduce a novel unsupervised approach for multi-perspective data augmentation that boosts summarization performance according to automatic evaluation. Finally, we propose reinforcement learning rewards to improve factual consistency and answer coverage and analyze areas for improvement.
In a world of information overload and the ubiquity of discussion fora, there is a need for text summarization as a means of distilling relevant information into a concise form. The problem is even more pertinent for question answering within the context of Community Question Answering (CQA) fora, where a person poses a question and can get an abundance of answers to sift through. Ideally, an answer summary should cover the multiple perspectives found in the answers, where available. Table 1
Question: I recently relocated to USA and have no Credit Score. Is Secure Credit Card is the only option for me to start building my credit score? Also please recommend which other credit cards are available for people like me to build credit score
Answer 1: If you have an AMEX from another country, you can get an AMEX in the US. American Express has a separate system that is not as strongly country-dependent as, say, VISA and MasterCard...
Answer 2: Secured credit cards are usually not very cost effective for building credit. Find a local credit union, of medium to large size. A credit union is like a bank, but operates under slightly different rules, and is non-profit...
Answer 3: If you have had an American Express card abroad, you can try and get a US Amex...
Answer 4: If the country you came from has an HSBC, you can ask HSBC to use your credit rating from that country to give you an HSBC Mastercard in the US...
Summary:
There are a range of options available to you, although your chance of success will depend on the bank that you apply with. However, if you have previously had a card with HSBC or American Express, the process may be simpler. Other options could include borrowing from a credit union or asking a friend or family member to be an additional cardholder with you.
Table 1: An example summary from our AnswerSumm dataset, illustrating the multiple viewpoints present manually-written summaries, and a subset of the 8 user answers to which the summary can be aligned.
illustrates such an example where a person poses a question about relocating to the US and obtaining a credit score and a credit card. We present a sample of the 8 answers to that question on StackExchange and a manually-curated summary covering the answers' main perspectives. Answer summarization is a form of query-based, multi-document summarization (Ernst et al., 2020), and creating answer summaries that reflect the underlying varying perspectives entails several subtasks: selection of answer sentences relevant to the question (query sentence relevance), grouping these sentences based on perspectives (clustering), summarizing each perspective (cluster summarization), and producing an overall fused summary (fusion).
To date, most CQA fora have a notion of a 'best
*Author is currently at Salesforce AI Research.
Human Language Technologies, pages 2508 - 2520
answer,' which is either manually chosen by the person who asked the question or by a moderator, or obtained via community ratings. Work in this field typically makes use of this best answer as a proxy for summaries, i.e. the focus is on extractivelike summaries (Tomasoni and Huang, 2010; Chan et al., 2012; Pande et al., 2013; Wang et al., 2014; Song et al., 2017). Datasets such as WikiHowQA (Deng et al., 2020a), which consists of a question, a long answer, and an answer summary, focus on answer selection and the summarization of a single answer. While CQASumm (Chowdhury and Chakraborty, 2019) uses the chosen best answer as the answer summary, they also apply heuristics to ensure token overlap with the remaining answers. However, the best answer only presents one person's perspective and rarely captures the variety of perspectives discussed in a thread. Furthermore, we find that the heuristics applied in CQASumm generally promote only long answers instead of multiple perspectives. To validate our hypothesis, we examine a set of 30 summaries from CQASumm and found that only  $37\%$  of the examples contained multi-perspective answers. In contrast,  $75\%$  of our dataset requires multi-perspective summaries.
As alluded to above, although answer summarization is an important research topic with practical applications, there are no relevant datasets or techniques to address it effectively, i.e. no manually-curated dataset exists for the answer summarization problem, and no dataset decomposes the task into its constituent subtasks. This work tries to close the research gap in answer summarization; we develop an annotation pipeline for multiperspective abstractive answer summarization. We introduce the largest human-annotated dataset for answer summarization, containing components for sentence relevance, clustering, cluster summarization, and global answer summarization. We enlist ten professional linguists to contribute to our annotation efforts. We iterate over instructions and devise pre-pilot, pilot, and final annotation stages as well as re-annotation for quality assurance. We collect over 4,631 high-quality data points. For validation of our curated data set, we benchmark state-of-the-art models on the subtasks of this dataset and perform qualitative analysis to provide a clear baseline and directions for future work. We then propose a data augmentation pipeline to further boost summarization performance. To generate a silver multi-perspective summarization dataset,
we introduce a pipeline for automatically creating multi-perspective bullet-point answer summaries for data augmentation, which boosts performance. We find that a strong baseline model trained on our human-annotated data inherently outputs factually consistent summaries, and model performance is improved by adding data from our automated pipeline. Finally, we introduce entailment-based and semantic area RL rewards namely to analyze its effect on factual consistency and semantic coverage, ensuring we are capturing all factually relevant perspectives.
Extractive Answer Summarization: Much work has focused on the extractive summarization setting as an answer-ranking problem (Chan et al., 2012; Pande et al., 2013; Wang et al., 2014). Liu et al. (2008) find that only  $48\%$  of the best answers on Yahoo! Answers are unique best answers; there are multiple correct ways to answer a question. Other recent work has focused on sentence extraction using metadata (Tomasoni and Huang, 2010), sparse-coding frameworks (Song et al., 2017), or answer-aware sequential extraction (Deng et al., 2020b). Our focus is on an answer summarization pipeline which ultimately results in abstractive answer summaries.
Abstractive Answer Summarization: Another line of work has attempted abstractive answer summarization by treating the tagged best answer as the gold summary of all the other answers (Chowdhury and Chakraborty, 2019; Chowdhury et al., 2020). Recent work summarizes answers to medical questions via a medical concept graph Zhang et al. (2020) and incorporates multi-hop reasoning (Zhang et al., 2020) and answer relevance from a QA model into the summarization model (Su et al., 2021). Most related to our dataset creation, Chowdhury and Chakraborty (2019) present CQASumm, a dataset of about 100k automatically-created examples consisting of the best answer as the gold summary, which, however, contains noise due to automatic creation.
Multi-document Summarization: Answer summarization can be viewed as a query-based multi-document summarization (MDS) problem. Approaches to query-focused multi-document summa
For reproducibility of our findings, we will make our data and code publicly available at https://github.com/ Alex-Fabbri/AnswerSumm.
rization have dealt with data sparsity via data augmentation (Pasunuru et al., 2021) by restructuring the title and paragraphs of news articles to match the target task, coarse to fine-grained modeling Xu and Lapata (2020), and by converting generic summarization data into proxy queries (Xu and Lapata, 2021). Several large-scale MDS datasets have been introduced in the news domain (Fabbri et al., 2019; Gu et al., 2020; Gholipour Ghalandari et al., 2020), for creating Wikipedia lead-paragraphs (Liu et al., 2018) and for long-form question answering (Fan et al., 2019). However, Wikipedia summarization is topic-based and less granular than our setting, and the ELI5 dataset (Fan et al., 2019) summarizes web documents rather than direct query answers.
3 AnswerSumm
We introduce our annotation protocol and the characteristics of our manually-curated answer summarization dataset. Our annotation pipeline is illustrated in Figure 1.
Annotation Protocol Our annotation pipeline consists of four steps 1) Answer Sentence Selection (SentSelect), 2) Clustering (SentCluster), 3) Cluster Summarization (ClusterSumm), and 4) Cluster Summary Fusion (ClusterSummFusion). We refer to the task of taking forum answers and producing final overall summaries E2ESumm. We believe that this pipeline mirrors the process by which humans create summaries of multiple answers by narrowing and organizing information, followed by paraphrasing. Furthermore, dividing the summarization task in such a way paves the way for future work in understanding the steps by which a model creates a final summary, and recent work has similarly divided multi-document summarization into these subtasks (Ernst et al., 2020). For consistency, the same annotator completes all four steps for a given example. However, we surmise that if each subtask is performed well, then multiple annotators can be involved for a given example.
For a given question thread, we present the annotator with the question, the forum from which the question came, the title of the post, and the tags that the original poster associated with the question. The user answers are then presented, where each answer has been automatically segmented into individual sentences using SpaCy (Honnibal et al., 2020). It is worth noting that sentence-level granularity is chosen as a simplifying assumption as an appropriate level of segmentation. We are cog-
nizant that clause level might be more accurate, however, given state-of-the-art clause detection as well as the precedence for sentence-level modeling in previous work (Tomasoni and Huang, 2010; Song et al., 2017), we opted for sentence-level segmentation.
Answer Sentence Selection (SentSelect): We ask the annotators to mark each sentence as relevant or not depending on whether it provides information useful in answering the user's question. Annotators are instructed to mark as irrelevant sentences that do not function as independent units, such as those which need additional context to be understood as an answer to the question. As a result, noise from sentence segmentation may cause certain sentences to be marked as not relevant, but upon manual inspection, we found this to not be an issue.
Clustering (ClusterSumm): Annotators then cluster found relevant sentences into groups of the same topic. Sentences that are on the same topic but have different polarities are grouped together. We do not pre-define a desired number of clusters. Furthermore, clusters consisting of a single item are allowed, and a sentence can belong to multiple clusters. A sentence in multiple clusters may occur in the case of complex sentences which present multiple viewpoints.
Cluster Summarization (ClustSumm): The annotators summarize each individual cluster of relevant sentences from the previous step. Each cluster summary should typically consist of 1-4 complete sentences. To allow for abstract summaries, we instruct the annotators to try to use their own words (paraphrase) instead of copying large segments of the sentence clusters verbatim. Using the sentences' exact words is allowed, but they should not copy more than five consecutive words from a sentence. Additionally, the summary should function as an answer rather than as an analysis of the summary sentences. So, rather than stating, "Most of the answers indicate that it is highly subjective," the annotator writes directly "It is highly subjective." To ensure that the summary information can be found in the input answers, we also instruct the annotators to focus solely on the answer threads and not their external knowledge of the subject. The summary should solely (1) summarize the viewpoint present in the sentence cluster; and, (2) try to include some specific details from the assertions and anecdotes made by the answer sentences.
Figure 1: An illustration of our dataset annotation pipeline. Given a question and answers to that question, professional linguists 1) select relevant sentences, 2) cluster those selected sentences, 3) summarize each cluster's sentences, and 4) fuse clusters into a coherent, overall summary.
We leave it to the annotator's judgment to leave out details from clusters that are too minute.
Cluster Summary Fusion (ClusterSummFus i o: The annotator combines the cluster summaries from the previous step into a single, coherent summary. The annotators can apply additional paraphrasing and need not simply insert each cluster summary; they may combine some cluster summaries into a single sentence. The annotator is asked to order and insert discourse connectives as necessary to increase inter-sentential coherence in the final summary.
Data Filtering We selected question threads for annotation from the StackExchange data release<sup>2</sup>, as it is publicly available and has been shared using a Creative Commons ShareAlike license. We created a whitelist of non-technical fora which do not require domain knowledge to summarize, similar to work on non-technical email summarization (Ulrich et al., 2008). We sampled from 38 fora. Table 3 illustrates the top 20 fora and their frequency. In addition to this preliminary filtering, we further prompted annotators to discard any examples for which they felt unable to adequately assess the relevance of answer sentences to a question due to lack of required domain knowledge or context.
The filtering of question threads was motivated by heuristics detailed in Tomasoni and Huang (2010), which aims to find threads suitable for summarization. We only include answers with a non-negative community score which is determined by the number of upvotes by community members minus the number of downvotes. Moreover, they do not include comments to answers for simplicity, although future work may incorporate this into modeling. Threads were removed if 1) there were
less than four answers, 2) the sum of the length of all answers was outside of (100, 1500) words, and 3) the average length of answers was outside of the (50, 300) words interval. Questions include the subject of the post and the content of the post when available. Out of about 870k question threads, about 8k met these criteria. While this filtering may be strict, it avoids threads that contain short or single answers for which summarization may be superfluous, thus creating a higher-quality, diverse, dataset as confirmed by our analysis that  $75\%$  of our examples require multi-perspective summaries
Quality Controls Our annotators are 10 professional linguists recruited through a professional vendor. We provide the linguists with an example of an annotated question thread for clarity and discussed the instructions in-depth with the vendors to avoid ambiguities. To ensure that the linguists are well-trained and that the annotations meet our requirements, we completed our annotations in three stages. We began with a pre-pilot of 50 example question threads, followed by a pilot of 500 examples and then a final set of 5000 examples. We divide annotation files into groups of 50 examples, which are split among the annotators. We make use of the pilot and final annotation sets for our dataset release. To determine inter-annotator agreement (IAA), 250 examples were repeated across three annotation files. A Fleiss Kappa of 0.25 was achieved for sentence relevance selection, the first task. The IAA score indicates fair agreement.
Dataset Statistics and Comparison We provide statistics about the subtasks from our dataset pipeline in Table 2. There does not exist a manually-curated dataset for abstractive answer summarization. CQASumm is the closest dataset with our desired answer summarization qualities,
$^{2}$ https://archive.org/download/ stackexchange
Task	Input	Output
SentSelect	6.4 Ans
40.3 Sents	9.2 Sents
787 Words
SentCluster	9.2 Sents	2.6 Clusters
ClusterSumm	3.4 Sents	21 Words
77 Words
ClusterSummFusion	55 Words	47 Words
Table 2: Average statistics for input and output across the four Answersumm subtasks. E2ESumm's input is that of the SentSelect and the output is that of ClusterSummFusion.
Forum	Frequency	Forum	Frequency
English	662	Travel	356
Cooking	636	Music	262
Gaming	485	Bicycles	242
SciFi	408	DIY	213
ELL	378	Aviation	190
although it is created automatically based on heuristics which simply promote answers as summaries rather than truly summarizing answers. We also present a comparison of dataset statistics between our dataset AnswerSumm, and the standard XSum and CNN-Daily Mail (Nallapati et al., 2016) summarization datasets in Table 4. In general, we find our dataset to be more abstractive than CNN-DailyMail and less so than XSum. Furthermore, the average number of input tokens for the E2ESumm task, is larger than those two datasets, confirming that the input to our tasks provides reasonable grounds for requiring summarization.
4 Pipeline for Data Augmentation
Manually annotating data at the scale of other existing summarization datasets such as CNN-DailyMail is impractical. Taking advantage of the abundance of unlabeled StackExchange fora available, we develop a pipeline to automatically create data similar to that which is manually annotated above. This process provides augmented data for training summarization models.
Data Filtering Similar to filtering for manual annotation, we obtained question threads from StackExchange and applied heuristics motivated by Tomasoni and Huang (2010) to find threads suitable for summarization. Threads are removed if: 1) there are less than three answers; 2) the longest answer is at least 400 words; 3) the input token length of all answers is not between 100 and 1000 words;
Table 3: The ten most frequent forums found in the AnswerSumm dataset and their associated counts.
Dataset	Novel unigrams	Ext. Oracle	Input Len	Summ Len
AnswerSumm	21.0	40.05/18.45/35.70	787	47
XSUM	35.8	29.79/8.81/22.65	431	23
CNN	16.8	50.38/28.55/46.58	761	46
DailyMail	17.0	55.23/30.55/51.24	653	55
Table 4: Comparison between AnwerSumm and the XSum (Narayan et al., 2018) and CNN-DailyMail (Nallapati et al., 2016) datasets, with data statistics from (Narayan et al., 2018). Oracle Extractive and Length refer to the maximum ROUGE (Lin, 2004) score achievable by an extractive model, and the average length of the input and summaries, respectively.
and, 4) the average length of answers is between 50 and 300 words. Heuristics were chosen to provide enough examples for data augmentation, leaving about  $130\mathrm{k}$  question threads in total.
Pipeline Overview The input to our pipeline is a user question and its answers. We select question threads from StackExchange and operate on the sentence-level of these answers, as in our manually-created data. Our automatic dataset pipeline consists of the following components which aim to mirror the manual pipeline: 1) a relevance model to select relevant sentences and remove irrelevant ones; 2) a clustering model to cluster similar content – reflecting various perspectives; and, 3) input and abstractive summary creation from cluster centroids, resulting in bullet points for the various perspectives reflected in the answers. Figure 2 illustrates the pipeline.
Relevance model: A sentence-level relevance model trained on CQA fora is leveraged to eliminate irrelevant sentences from the input (collection of answers to a question). The output from this stage serves as input to the clustering stage. Model details are found in Section 6.
Clustering: Typical K-Means clustering for short text (Xu et al., 2017; Hadifar et al., 2019; Rakib et al., 2020) does not work for our setting as the value of  $\mathrm{K}$  is not known a priori. In fact, it varies from question to question. Accordingly, we use the sentence-transformers library (Reimers and Gurevych, 2019a) to perform clustering. Specifically, we start with a RoBERTa-based model finetuned for sentence embeddings on an entailment dataset, which is further fine-tuned for semantic similarity. Clustering parameters are chosen based on a StackOverflow clustering dataset containing labeled clusters, as provided in Rakib et al. (2020). We apply Agglomerative clustering with average
Figure 2: An illustration of our automatic dataset pipeline which mirrors the manual pipeline for data augmentation. Given a question and answers, relevant sentences are selected and clustered. Then, the cluster centroid sentence of non-singleton clusters is removed from the input to use as bullet point summaries.
linkage, cosine distance, and a maximum distance of 0.65. Parameters are empirically chosen.
To create the final summaries, we locate the centroid of clusters with at least two sentences and select these centroids as bullet-point summaries. Further, we remove the centroid sentences from the sentence-segmented input answers to create a challenging abstractive summarization dataset analogous to the XSum dataset (Narayan et al., 2018). Since each cluster contains at least two sentences, we assume that given a perfect clustering algorithm, a related sentence can help generate the removed centroid sentence. While removing sentences naturally decreases coherence, we believe that this introduces a tolerable level of noise. We also experimented with cluster centroid paraphrasing and not removing from the input, but this did not improve downstream performance, which we use to measure the value of this dataset and the level of noise.
5 RL-Based Training
Cross-entropy loss in standard sequence-to-sequence model training suffers from exposure bias and also does not directly optimize evaluation metrics (Ranzato et al., 2016). The REINFORCE algorithm (Williams, 1992), on the other hand, allows for optimizing the evaluation metrics using nondifferentiable rewards. We use an RL multi-reward objective to promote summaries with both high coverage of the input answers and faithfulness.
5.1 Multi-Reward Optimization
We follow the settings of Pasunuru and Bansal (2018) for optimizing multiple rewards. In the equations which follow,  $x = \{x_{1}, x_{2}, \ldots, x_{n'}\}$  refers to the input source tokens (e.g. a question and its answers), and  $y^{*} = \{y_{1}^{*}, y_{2}^{*}, \ldots, y_{N}^{*}\}$  refers to the gold target summary which consists of
$\{y_{1s}^{*}, y_{ss}^{*}, \ldots, y_{Ns}^{*}\}$  sentences. Standard training minimizes the negative log-likelihood (NLL) loss using teacher forcing (Williams and Zipser, 1989):