SlowGuess/NAACL_2022_OCR huggingface dataset:
Wenxuan Zhou $^{1*}$ , Qiang Ning $^{2}$ , Heba Elfardy $^{2}$ , Kevin Small $^{2}$ , Muhao Chen $^{1}$
<sup>1</sup>University of Southern California, <sup>2</sup>Amazon
{zhouwenx,muhaoche}@usc.edu {qning,helfardy,smakevin}@amazon.com
Current question answering (QA) systems primarily consider the single-answer scenario, where each question is assumed to be paired with one correct answer. However, in many real-world QA applications, multiple answer scenarios arise where consolidating answers into a comprehensive and non-redundant set of answers is a more efficient user interface. In this paper, we formulate the problem of answer consolidation, where answers are partitioned into multiple groups, each representing different aspects of the answer set. Then, given this partitioning, a comprehensive and nonredundant set of answers can be constructed by picking one answer from each group. To initiate research on answer consolidation, we construct a dataset consisting of 4,699 questions and 24,006 sentences and evaluate multiple models. Despite a promising performance achieved by the best-performing supervised models, we still believe this task has room for further improvements.
Open-domain question answering (QA) systems (Voorhees, 1999) aim to answer natural language questions using large collections of reference documents, contributing to real-world applications such as intelligent virtual assistants and search engines. Current QA systems (Zhu et al., 2021) usually adopt a three-stage pipeline consisting of: (1) a passage retriever (Yang et al., 2019; Karpukhin et al., 2020) that selects a small set of passages relevant to the question, (2) a machine reader that examines the retrieved passages and extracts (Wang et al., 2017, 2019) or abstracts (Lewis et al., 2020) the candidate answers, and (3) an answer reranker (Wang et al., 2018; Kratzwald et al.,
2019) that fuses features from previous stages to either select one final answer or return the top-ranked answer from the previous stage.
Current QA research (Joshi et al., 2017; Kwiatkowski et al., 2019) primarily examines the case where each question is assumed to have a single correct answer. However, in practice, many questions can have multiple correct answers. For example, the question "Is coffee good for your health?" can be answered with respect to different aspects (e.g., "coffee can help you with weight loss", "coffee can cause insomnia and restlessness"). To correctly identify different aspects of answers to the same question while mitigating aspect-level redundancy, it is important to consolidate the answers. Answer consolidation is particularly desirable for applications such as intelligent assistants, where responses are desired to be both comprehensive and concise. Additionally, in scenarios where QA is used for knowledge extraction (Bhutani et al., 2019; Du and Cardie, 2020) or claim verification (Yin and Roth, 2018; Zhang et al., 2020), consolidation is also an essential step to identify salient knowledge or evidence while mitigating duplication.
To effectively recognize multiple aspects of answers in QA systems, our first contribution is to introduce and formalize the answer consolidation problem. Specifically, given a question paired with multiple answer snippets, answer consolidation first partitions the snippets into groups where each group represents a single aspect within the answer space. Once partitioned, the final answer set is produced by returning a representative snippet from each group. In this formulation, the answer consolidation task is a post-processing stage that takes predicted answer-mentioning snippets (in this work, sentences) from previous QA stages and produces an answer set that maximizes answer aspect coverage while minimizing answer duplication.
To foster research on the answer consolidation problem, our second contribution is the collec
*This work was conducted when the first author was doing an internship at Amazon.
1The contributed resources and implementation are available at https://github.com/amazon-research/ question-answer-consolidation.
Human Language Technologies, pages 4314 - 4325
tion of a new dataset, namely, QUASI (Question-Answer consolidation). QUASI consists of 4,699 questions such that each question is paired with multiple answer-mentioning sentences grouped according to different aspects. Starting with a Quorabased question source (Chen et al., 2018), noting the potential for multi-aspect answers, we first retrieve 10 relevant answer sentences from the web for each question. These sentences are then examined by three crowd-sourced workers, who exclude sentences that do not contain an answer and group the remaining ones. Finally, individual sentence groupings from different workers are aggregated into a single partitioning. QUASI consists of 24,006 sentences and 19,676 groups, corresponding to an average number of 4.18 aspects per question and 1.22 sentences per group.
Our third contribution is a comprehensive benchmarking for the answer consolidation problem based on QUASI. Specifically, we consider two evaluation settings: (1) classification, where the model predicts whether two sentences are in the same group. (2) sentence grouping, where the model groups the answer sentences. We evaluate a wide selection of zero-shot and supervised methods, including various SoTA sentence embedding models (Reimers and Gurevych, 2019; Gao et al., 2021), cross-encoders (Devlin et al., 2019; Liu et al., 2019), and a newly proposed answer-aware cross-encoder model. In the supervised setting, the answer-aware cross-encoder achieves the best results based on the Matthew correlation coefficient (MCC) score of  $87.8\%$  (classification setting) and an adjusted mutual information (AMI) score of  $68.9\%$  (sentence grouping setting). As this performance is notably far from perfect, our findings indicate the need for future investigation on this meaningful, but challenging, task.
QA with multiple answers. Many QA datasets have assumed that a question has a single correct answer (Joshi et al., 2017; Kwiatkowski et al., 2019), while in real scenarios, many questions can have multiple correct answers. Fewer datasets for QA or machine reading comprehension (MRC) have been proposed with the consideration of multiple answers. In extractive MRC, MASH-QA (Zhu et al., 2020) allows one question to be answered by multiple non-consecutive text spans. In abstractive MRC/QA, MS MARCO (Campos et al., 2016) sim
ply treats different workers' answers as different answers. DuReader (He et al., 2018) merges similar answers during data construction. QReCC (Anantha et al., 2021) allows one worker to provide multiple different answers. Beyond these, WebQuestions (Berant et al., 2013) and GooAQ (Khashabi et al., 2021) include lists of diverse answers, and TREC-QA (Baudiš and Šedivý, 2015) uses regular expressions to capture multiple answers. However, none of the aforementioned efforts have investigated effective consolidation of multiple answers. In this work, we formally define and collect a dataset for the answer consolidation problem as a complement to previous work. From another perspective, AmbigQA (Min et al., 2020) focuses on the case where a question can be interpreted in different ways, leading to the question disambiguation task. This is fundamentally different from our work that partitions answers to the same question into different coherent subsets. Stance detection (Liu et al., 2021) is concerned with the focused problem of collecting approving/disapproving opinions for a yes-no question, unlike our studied problem where all multi-answer questions are considered and the answers are not limited to binary opinions.
Answer Summarization. Questions with multiple answers are common in online communities. For example, Liu et al. (2008) observe that no more than  $48\%$  of best answers on Yahoo! Answers are unique. Many efforts (Song et al., 2017; Chowdhury and Chakraborty, 2019; Fabbri et al., 2021) have been devoted to summarizing reusable answers in community QA. Particularly, AnswerSumm (Fabbri et al., 2021) proposes a dataset where different answers are rewritten to bullet points by humans. While training on summarization data may enable the model to return salient and non-redundant answers, such training only works for obstructive machine readers. A more related work is BERT-DDP (Fujita et al., 2020), which considers the problem of getting a diverse and nonredundant answer set. They construct a dataset based on Yahoo! Chiebukuro where workers are asked to provide an answer set given a question. However, the correct answer set is not unique when answers are equivalent. As they treat all but the annotated answer set as wrong, both training and inference are prone to false negatives. In this paper, we group the answers with respect to their aspects and provide a discriminant rule, such that the correct group assignment is unique.
Diverse passage retrieval. Many information retrieval efforts address the problem of retrieving diverse documents for a query (Clarke et al., 2008; Fan et al., 2018; Abdool et al., 2020). In QA, Min et al. (2021) examine answer diversity in passage retrieval and propose a self-supervised dynamic oracle training objective. However, as passages may contain irrelevant information to the question, the retriever faces the challenge of identifying and integrating answers in passages when assessing answer diversity. In this work, we consider a dedicated task of answer consolidation and leave the problem of identifying answers to previous QA stages.
3 Answer Consolidation Task
Motivation. Many questions can have multiple correct answers, including questions explicitly asking for a multi-answer list (e.g., What are the symptoms of flu?) or questions where different people have different opinions (e.g., debate questions), amongst others. To provide users with a comprehensive view of the answers, the QA system needs to actively identify different answers as opposed to only returning the most popular or top-ranked answer. Additionally, as the same answer may be repeated or paraphrased many times in the reference corpus (e.g., web), the QA system may also need to eliminate the redundant answers. We address these requirements within answer consolidation.
Basic concepts. When answering a specific question, different answers may be given regarding different perspectives, opinions, angles, or parts of the overall answer. We regard such answers as those pertaining to different aspects. Furthermore, we refer to two sentences as equivalent if they contain the same answer aspect(s) and distinct if they express different answer aspect(s). To better identify equivalent/distinct sentences, we propose the following operational discriminant rule: Given two answer-mentioning sentences  $s_1$ ,  $s_2$  for the same question  $q$ , we can rewrite the answers contained in  $s_1$  and  $s_2$  into yes-no questions  $q_1'$  and  $q_2'$ , which can be answered by yes/no/irrelevant. Then, if  $s_1$  and  $s_2$  give coherent answers of yes/no $^3$  to  $q_2'$  and  $q_1'$ , respectively, then  $s_1$  and  $s_2$  are considered to represent equivalent aspects. Otherwise, they are considered distinct from each other.
We take the following example:
Q: Is coffee good for your health?
S1: Coffee can make you slim down.
S2: Coffee can relieve headache.
S3: Coffee can help with weight loss.
Then we rewrite the answers contained in sentences as the following questions:
Q'1: Can coffee make you slim down?
Q'2: Can coffee relieve headache?
Q'3: Can coffee help you with weight loss?
We can tell that S1 and S3 are equivalent, as they both give coherent answers (yes) to each others' yes-no questions. We can also tell that S2 is distinct from S1 and S3, as it gives irrelevant answers to Q'1 and Q'3.
Task definition. A formal definition for answer consolidation is that given a question and a set of answer-mentioning sentences, answer consolidation aims at putting sentences into groups such that: (1) each sentence belongs to exactly one group, and (2) sentences from the same/different groups are equivalent/distinct. In this way, each sentence group corresponds to the same answer aspect(s). We show in §4.3 that although this definition may fail for a pair of sentences if they are partially relevant, it only occurs for 2.6% of sentences, which shows that our operational task definition works well for the majority of sentences in practice.
In this work, we treat answer consolidation as a stand-alone process applied after QA retrieval such that we only take the question and answer-mentioning sentences as input. In this way, the answer consolidation model is independent of the retriever and the reader architectures, and can flexibly adapt to different QA systems. We show in §6.3 that taking sentences instead of answer spans as input leads to better performance.
4 Question-Answer Consolidation Dataset (QUASI)
In this section, we describe the creation of our Question-Answer consolidation dataset (QUASI) including corpus collection (§4.1) and dataset annotation (§4.2). We then provide statistical and quantitative analysis of QUASI (§4.3).
4.1 Corpus
We created QUASI based on the Quora question pairs (QQP) corpus (Chen et al., 2018), which consists of  $364\mathrm{k}$  questions pairs, originally designed for predicting whether pairs of questions have the
${}^{2}$  A general process for changing a sentence to questions can be found at https://www.wikihow.com/ Change-a-Statement-to-Question.
<sup>3</sup>Answers of irrelevant are not considered coherent.
same meaning. We start with QQP since Quora questions have a high expected propensity of having multiple correct answers. In preprocessing, we removed questions containing spelling errors or non-English words using the Enchant library.4 and questions containing personal pronouns including {"I", "you", "we", "my", and "your"} as we found that such questions frequently ask about very specific personal experiences for which the answers may not necessarily contain any noteworthy claims, opinions, or facts in the answer.
Next, we retrieved sentences that were likely to contain multiple answers to the questions. Given a question, we retrieved relevant sentences from the web using a SoTA industry QA retriever, where each sentence was associated with a relevance score and a URL. To ensure the questions have multiple answers, we first removed sentences retrieved from quora.com and kept the questions if the relevance scores of the top three retrieved sentences were larger than specified low-confidence threshold. Finally, we kept the top 10 sentences for each remaining question and pass to crowd workers for sentence group annotation.
We used Amazon Mechanical Turk (AMT) to annotate QUASI. Each AMT HIT consisted of a question and 11 sentences (including the top-10 relevance scores and one additional attention-check sentence) where the crowd workers were required to: (1) identify sentences that actually contain answers to the question and (2) put answer-mentioning sentences into sentence groups with respect to their aspects. The workers were allowed to skip an answer-mentioning sentence if it was hard to put it into any groups (e.g., sentences containing more than one aspect).
The AMT interface is shown in Figure 1. Annotation was performed by dragging the sentences between blue boxes. The sentence groups could be added or removed using the two buttons. To submit the HIT, workers needed to put all sentences into boxes corresponding to either a specific sentence group, not an answer, or hard to put into groups.
Cost. Each HIT was assigned to three annotators with pay of \(0.50/\mathrm{HIT}\), leading to target an hourly pay rate of \)15. We randomly sampled 5k questions from §4.1 for HIT submission to AMT.
Figure 1: The interface used to collect the dataset. The second sentence is an attention-check sentence.
Quality Control. We used three strategies to ensure the annotation quality:
1. Workers selection. We only allowed crowd workers with acceptance rate  $\geq 98\%$  and had completed at least  $5\mathrm{k}$  hits to work on the task. We provided annotation guidelines and examples of 3 annotated hits to instruct the workers.
2. Qualification test. We manually annotated three hits as the qualification test. Workers were required to practice on these three hits and get the correct sentence groups on at least 2 hits to continue working on the task. We pay \(0.05 for each submitted hit in the qualification test. Although we had provided detailed instructions, only \(24\%\) of workers passed the qualification test.
3. Attention checker. For all hits, we added an attention-check sentence that was randomly sampled from other questions, so that it was unlikely to answer the question. As a part of the task, the worker needed to identify that this attention-check sentence did not involve an answer, otherwise she/he would be blocked from continuing to work on the task.
Label aggregation. To ensure data quality, we aggregated worker annotations. To derive the sentence set for answer consolidation, we begin by only considering sentences put into any sentence group(s) by all crowd workers as eligible, keeping 37,588 out of 50k sentences. Next, we derived the aggregated sentence groups from AMT annotations. As we are not aware of existing methods for this process, we proposed the following algorithm for constructing new sentence groups. First, we sort the sentences by their relevance scores and create a sentence group with the most relevant sentence
<sup>4</sup>https://abiword.github.io/enchant/
Type	Description	Example	Percentage
Formatting	Sentences only differ in letter case, punctuation, or short forms.	Q: How small can a black hole be?
S1: scientists think the smallest black hole are as small as just one atom.
S2: Scientists think the smallest black holes are as small as just one atom.	9%
Exact match	Sentences are different but answer spans are the same.	Q: What are some good stories of revenge in relationship?
S1: Shakespeare's “Hamlet” is one of the most famous plays of revenge.
S2: The play, Hamlet by William Shakespeare explores the concept of revenge.	47%
Lexical variation	Answer spans differ in articles, verb tenses, or have synonym substitutions.	Q: Is it hard to get a job as a fashion buyer?
S1: Fashion jobs in merchandising can be very challenging.
S2: Picking one out of many fashion jobs generally is an overwhelming challenge.	11%
Semantic variation	Answers are paraphrased, or identification requires commonsense reasoning.	Q: How does the respiratory system work?
S1: The respiratory system works by getting the good air in and the bad air out.
S2: The Respiratory System a simple system designed to get oxygen into the body, and to get rid of carbon dioxide and water.	30%
Ambiguous	We do not agree with the crowd workers' annotation.	Q: Is Zeus more powerful then Odin? (Not the same aspects.)
S1: Zeus is 10 ton more than Odin.
S2: In DC, Zeus is higher than Odin.	3%
Table 1: Types of equivalent sentences annotated by crowd workers. We randomly sampled 100 sentence pairs in the same group, manually annotated the answer span (underlined), and categorized them into different types.
being the only member. We then iterate over the remaining sentences with the following procedure. For a sentence  $s$ , there are three possible cases:
1. If there existed one group  $\mathcal{G}$  such that  $\forall s' \in \mathcal{G}, s$  and  $s'$  were put into the same group by all workers, and for all already added sentences  $s^* \notin \mathcal{G}, s$  and  $s^*$  were put into different groups by all workers, we added  $s$  to  $\mathcal{G}$ .
2. If for all already added  $s^*$ ,  $s$  and  $s^*$  were put into different groups by all workers, we created a new group with  $s$  being the only member.
3. Otherwise, we discarded sentence  $s$ , since there was disagreement on this sentence.
Finally, we keep each question for which the number of preserved sentences was larger than one. Our aggregation algorithm produced sentence groups on a subset of sentences, on which all workers agree on each pair of sentences about whether they belong to the same group or not. After this process, 4,699 out of 5,000 questions and 24,006 out of 37,588 sentences were kept.
4.3 Dataset Analysis
We provide statistical and qualitative analyses regarding QUASI in this section.
Annotation quality. We first analyze the quality of data annotation before label aggregation. In the first annotation task of identifying whether a sentence contains an answer, AMT workers achieved an inter-annotator Fleiss' kappa of 0.62, an average agreement rate of  $90.2\%$ , and a worker agreement with aggregate (WAWA) of  $82.5\%$  in  $F_{1}$ . WAWA is
used to compare the majority vote with all workers' annotations. In the second annotation task of sentence grouping, we first get the set of sentences that all workers put into some groups. We then calculate the workers' agreement on each pair of sentences regarding whether they belong to the same group or not. The inter-annotator Fleiss' kappa, average agreement rate, WAWA  $F_{1}$  are 0.46, 84.8%, and 75.9%, respectively. These results show that the overall annotation quality is usable, but with room for improvement. Accordingly, to further improve the data quality, we only keep group annotations on which all workers agree (as stated in §4.2).
Dataset statistics. Our final dataset consists of 4,699 questions, 24,006 sentences, and 19,676 groups. On average, there are 4.18 groups per question, and 1.22 sentences per group. Specifically,  $97.7\%$  of questions have multiple aspects (sentence groups), and  $45.4\%$  of questions have at least one pair of equivalent sentences. In terms of sentence groups,  $86.6\%$  of groups have only one sentence,  $8.8\%$  of groups have two sentences, and the remaining  $4.6\%$  of groups have 3 or more sentences. Overall, this analysis shows that our dataset contains both multi-aspect and redundant answers that align with the challenges of answer consolidation.
Types of equivalent sentences. To get a better understanding of the required knowledge to identify equivalent sentences, we randomly sampled 100 sentence pairs in the same group and manually labeled the pairs with the types shown in Table 1. We observed that if the machine reader has the correct answer spans,  $56\%$  (formatting and ex
act match) of the equivalent sentences could be directly identified by string comparison. Another  $11\%$  of the equivalent sentences only differed at the lexical level, which may be identified using lemmatization, removal of stop words, or a dictionary of synonym words.  $30\%$  of equivalent sentences are semantic variations such that identifying equivalence requires understanding of their meanings and potentially even commonsense reasoning. E.g., for the example given in Table 1, the answer consolidation model needs to understand that oxygen is good air and carbon dioxide corresponds to bad air. For the remaining  $3\%$  of pairs, we do not agree with the annotation. Either the sentences do not answer the question, or they do not contain the same aspect(s).
Limits of the task definition. During data annotation,  $2.6\%$  of answer-mentioning sentences are denoted as "hard to put into groups". After inspection, we find that these sentences contain more than one aspect of answers. For example, given the question What are the best places to visit and things to do in San Diego, CA?, one sentence may be The San Diego Zoo, Balboa Park, and SeaWorld are the top tourist attractions in San Diego., which contains 3 different answers. This sentence overlaps with multiple groups and thus cannot be placed in a single group. Given the low prevalence, we leave consideration of such sentences to future work.
5 Approach
In this section, we first tackle the classification setting of answer consolidation (§5.1). Given a question and answer-mentioning sentences, the task is to predict for a pair of sentences whether they are in the same group. We consider different types of models, including sentence embedding models, cross-encoders, and answer-aware cross-encoders.