SlowGuess/NAACL_2022_OCR huggingface dataset:
Then we consider the sentence grouping setting, presenting the method of transforming pairwise predictions to sentence groups (§5.2). For all methods, we use RoBERTa $_{\text{LARGE}}$  (Liu et al., 2019) as the encoder, noting that other pretrained language models (PLMs) can easily be incorporated as part of these methods.
5.1 Sentence Pair Classification
Sentence embedding models. Sentence embedding models (Reimers and Gurevych, 2019; Gao et al., 2021) produce for each sentence an embedding vector, with which we can use metrics such as cosine to calculate their similarity. Specifically,
given a question  $q$  and a sentence  $s$ , we first tokenize them to  $X_{q}$  and  $X_{s}$  using the RoBERTa tokenizer, and then concatenate them as inputs:
$$
<   s > X _ {q} X _ {s} <   / s >
$$
Following Gao et al. (2021), we take the  $
$  embedding in the last layer of PLM as the sentence embedding. Then for a pair of sentences, whether they are in the same group is decided by the cosine similarity of the sentence embedding. The similarity can be converted to binary predictions using the best threshold that is selected on the validation set.
The sentence embedding models can work in both zero-shot and supervised settings. In the zero-shot setting, we directly use the pretrained sentence embedding model to make predictions without fine-tuning. In the supervised setting, given a pair of sentence embedding  $h_1$ ,  $h_2$ , and label  $y \in \{0, 1\}$ , where 0 and 1 mean not in/in the same group respectively, we fine-tune the PLM based on the following regression objective of sentence-transformers (Reimers and Gurevych, 2019):
$$
\mathcal {L} _ {\mathrm {r e g}} = \left(\cos \left(h _ {1}, h _ {2}\right) - y\right) ^ {2}.
$$
Cross-encoders. Cross-encoders (Devlin et al., 2019; Liu et al., 2019) take a pair of sentences as the input and predict whether they are in the same group or not. Given a question  $q$  and two answer-mentioning sentences  $s_1$  and  $s_2$ , we first tokenize them as  $X_q$ ,  $X_{s_1}$ , and  $X_{s_2}$  using the RoBERTa tokenizer, and then take  $X_q$ ,  $X_{s_1}$  and  $X_q$ ,  $X_{s_2}$  as two segments of inputs, following the input formats of sentence pair classification tasks (Liu et al., 2019):
$$
<   \mathrm {s} > X _ {q} X _ {s _ {1}} <   / \mathrm {s} > <   / \mathrm {s} > X _ {q} X _ {s _ {2}} <   / \mathrm {s} >
$$
Prediction is independently performed on sentence pairs using a binary classifier on the first special (classification) token  $\langle s \rangle$  embedding in the last layer of the PLM. The cross-encoders work in both zero-shot and supervised settings. In the zero-shot setting, we fine-tune the model on the MNLI (Williams et al., 2018) dataset and take entailment as in the same group. In the supervised setting, given the sentence pair embedding (obtained from  $\langle s \rangle$ )  $h$  and the label  $y$ , we fine-tune
5We find that using 0 for not in the same group achieves better results than using -1.
the model using the binary cross-entropy loss:
$$
\begin{array}{l} p = \sigma (\boldsymbol {w} ^ {\intercal} \boldsymbol {h}), \\ \mathcal {L} _ {\mathrm {b c e}} = - \left(y \log p + (1 - y) \log (1 - p)\right), \\ \end{array}
$$
where  $p$  is the probability that the sentences are in the same group,  $\sigma$  is the sigmoid function,  $w$  is a parameter of the classifier. In inference, we convert  $p$  to binary predictions using the best threshold selected on the validation set. The cross-encoders require predicting on all sentence pairs and have higher computational costs than sentence embedding models. However, we observe in experiments that cross-encoders consistently outperform sentence embedding models in the supervised setting.
Answer-aware  $(\mathbf{A}^2)$  cross-encoders. §4.3 shows that  $56\%$  of equivalent sentences can be directly identified if the model knows the answer spans. Therefore, we provide the answer consolidation models with answers generated from the UnifiedQALarge model (Khashabi et al., 2020). As the UnifiedQA is trained on both extractive and abstractive datasets, the answers may not be text spans of sentences. Specifically, given a question  $q$ , two sentences  $s_1$  and  $s_2$ , and the generated answers  $a_1$  and  $a_2$ , we first tokenize them as  $X_p$ ,  $X_{s_1}$ ,  $X_{s_2}$ ,  $X_{a_1}$ , and  $X_{a_2}$ , respectively, then construct the input to cross-encoders as:
$$
<   \mathbb {S} > X _ {q} X _ {s _ {1}} X _ {a _ {1}} <   / \mathbb {S} > <   / \mathbb {S} > X _ {q} X _ {s _ {2}} X _ {a _ {2}} <   / \mathbb {S} >
$$
The training process and inference process are the same as the cross-encoders.
5.2 Sentence Grouping
In answer consolidation, our ultimate goal is to obtain the consolidated sentence groups. This is done in a two-step approach. The first step is to get the matrix of distances  $\mathcal{D}$  between pairs of sentences for a question. We perform this step using models trained in the classification setting. For sentence embeddings,  $\mathcal{D}$  is adopted as the pairwise cosine distance matrix. For cross-encoders, each entry of  $\mathcal{D}$  equals to 1 minus the predicted probability for a sentence pair. As  $\mathcal{D}$  derived in this way may not be always symmetric, we use  $\frac{1}{2} (\mathcal{D} + \mathcal{D}^{\top})$  as the distance matrix instead.
The next step is to transform the distance matrix into sentence groups. Here we apply agglomerate clustering (Han et al., 2011). It uses a bottom-up strategy, starting from letting each sentence form its own cluster, and then recursively merging the
clusters if their distance is smaller than a threshold. We use the average distance of sentence pairs as the inter-cluster distance measure and select the best threshold on the validation set. Agglomerate clustering stops when the distances between all clusters are larger than the threshold.
In this section, we present the experimental setup (§6.1), show the main results (§6.2), conduct an ablation study (§6.3), and provide error analysis (§6.4).
Dataset. We randomly split the 4,699 questions into an 80/10/10 split, which serves as the training, validation, and test set, respectively.
Evaluation metrics. We use different evaluation metrics in the two evaluation settings. For the classification setting, we first use the micro  $F_{1}$  measure. Considering that classes in the dataset are highly imbalanced (only  $11\%$  of sentence pairs are in the same group), we additionally use the Matthews correlation coefficient (MCC; Matthews 1975), which is considered a more class-balanced metric. For the sentence grouping setting, we use clustering metrics including adjusted rand index (ARI; Rand 1971) and adjusted mutual information (AMI; Nguyen et al. 2009). These two metrics take the predicted grouping and the groundtruth grouping, and measure the similarity between them. For all metrics, larger values indicate better performance, and a value of  $100\%$  indicates perfect classification/grouping.
Configuration. We implement the models using Huggingface's Transformers (Wolf et al., 2020). The models are optimized with Adam (Kingma and Ba, 2015) using a learning rate of  $1\mathrm{e} - 5$  with a linear decay to 0. We fine-tune all models for 10 epochs with a batch size of 32 questions (including all associated sentence pairs). The best model checkpoint and thresholds are selected based on the validation set. We report the average results on 5 runs of training using different random seeds.
Models. We use RoBERTa $_{\text{LARGE}}$  (Liu et al., 2019) as the encoder for all models. For sentence embedding models, we try RoBERTa fine-tuned on two intermediate tasks: 1) SRoBERTa (Reimers and Gurevych, 2019) is fine-tuned on natural language inference (NLI) datasets, achieving better
Model	Classification	Grouping
F1	MCC	ARI	AMI
Zero-shot Sentence Embedding
RoBERTa	22.2	9.2	55.8	33.3
SRoBERTa	41.9	35.1	57.8	37.7
SimCSE-RoBERTa	53.2	47.6	66.1	46.1
Zero-shot Cross-Encoders
RoBERTa-MNLI	52.4	47.0	69.0	48.5
A2RoBERTa-MNLI	40.4	34.6	60.9	40.0
Supervised Sentence Embedding
RoBERTa	73.5	70.3	79.2	58.5
SRoBERTa	80.7	78.8	85.3	64.3
SimCSE-RoBERTa	81.1	79.0	85.7	64.9
Supervised Cross-Encoders
RoBERTa	86.8	85.9	88.4	66.9
A2RoBERTa	88.2	86.8	89.6	68.2
RoBERTa-MNLI	88.7	87.4	89.6	68.2
A2RoBERTa-MNLI	89.0	87.8	90.4	68.9
results on semantic textual similarity (STS) tasks, and 2) SimCSE-RoBERTa (Gao et al., 2021) is fine-tuned in a self-supervised fashion, taking a sentence and predicting itself using a contrastive learning objective. For cross-encoders, in addition to directly running supervised fine-tuning on our data, we also try supplementary training on an intermediate labeled-data task (Phang et al., 2018), which fine-tunes cross-encoders on MNLI before supervised fine-tuning. Particularly in the latter setting, we observe it being necessary to re-initialize the classifier before supervised fine-tuning to obtain more promising performance.
The experimental results on both the pairwise classification and sentence grouping settings are reported in Table 2. We observe that in the zero-shot setting, intermediate-task training improves answer consolidation, while the performance remains far behind supervised models. In the supervised setting, cross-encoders consistently outperform the sentence embedding models. Overall, the answer-aware cross-encoder intermediately tuned on MNLI (A²RoBERTa-MNLI) achieves the best results on all metrics, showing that intermediate-task training on MNLI improves performance. Besides, we find that answer-aware cross-encoders outperforms regular cross-encoders, showing that answers generated by the machine reader provide additional information that helps consolidation.
Table 2: Main results on the test set of QUASI. The best results in the zero-shot and supervised settings are highlighted in **bold**.
Model	Classification	Grouping
F1	MCC	ARI	AMI
Supervised Sentence Embedding
SimCSE-RoBERTa (Q+A)	61.6	57.6	63.4	51.6
SimCSE-RoBERTa (S)	72.1	69.0	78.2	58.1
SimCSE-RoBERTa (S+A)	77.6	75.0	80.4	60.0
SimCSE-RoBERTa (Q+S)	81.1	79.0	85.2	64.9
SimCSE-RoBERTa (Q+S+A)	82.5	80.4	85.1	64.6
Supervised Cross-Encoders
RoBERTa-MNLI (Q+A)	66.7	62.9	75.1	53.4
RoBERTa-MNLI (S)	83.8	81.9	85.3	65.0
RoBERTa-MNLI (S+A)	85.7	84.1	87.5	66.7
RoBERTa-MNLI (Q+S)	88.7	87.4	89.6	68.2
RoBERTa-MNLI (Q+S+A)	89.0	87.8	90.4	68.9
Table 3: Results with different input formats on the test set. Q, S, A denotes question, sentences, and answers (generated by UnifiedQA), respectively. RoBERTa-MNLI  $(\mathrm{Q} + \mathrm{S} + \mathrm{A})$  is equivalent to  $\mathrm{A}^2$  RoBERTa-MNLI.
In this section, we study the model performance based on different input information given to supervised models. We denote the questions, sentences as Q, S, and answers generated by UnifiedQA as A. The results are shown in Table 3. Overall, models trained on all inputs (Q+S+A) achieve better results than those that have observed only a subset of the available inputs on most metrics. Removing the sentences leads to the largest drops in performance (e.g.,  $20.9\%$  in  $F_{1}$  for SimCSE-RoBERTa and  $22.3\%$  in  $F_{1}$  for RoBERTa-MNLI), which shows that sentences provide useful information for answer consolidation. Using sentences only leads to the second-largest drop in performance, showing that without grounding to questions and answers, consolidation is not simply addressed only with the sentences. Besides, removing questions also leads to more significant drops in performance than removing answers (e.g.,  $4.9\%$  in  $F_{1}$  for SimCSE-RoBERTa and  $2.3\%$  in  $F_{1}$  for RoBERTa-MNLI). This shows that it is necessary to understand the answer equivalence within the question context in order to consolidate answers.
To get a sense of what knowledge is needed to further improve model performance, we examined sentence pairs incorrectly classified as not in the same group by  $\mathrm{A}^2\mathrm{RoBERTa}$  in the validation and test sets, where 318 out of 2,614 pairs (12.2%) are wrongly classified. We randomly sample 50 such error cases and categorize them by the answer-equivalence type as defined in Table 1. Of the 50
Cause	Description	Example
Entailment (16.7%)	One answer is en-tailed by the other.	Q: What makes successful people different from average people?
S1: Wealthy people are not afraid of failure, unlike average people who often do not even try.
S2: The difference between average people and achieving people is their perception of and response to failure.
Entity/commonsense knowledge (23.8%)	The answers refer to the same entity or are equivalent by common sense.	Q: What are some best Hollywood romantic movies to watch?
S1: When the subject of romantic movies comes up, one of the first that comes to mind in any list of all-time greats is Casablanca.
S2: Humphrey Bogart and Ingrid Bergman's film about love and loss during WWII is basically required viewing for anyone who enjoys romantic movies.
Semantic equival-ence (57.1%)	The answers have the same semantic meaning but are expressed using differ-ent words.	Q: What is the equity risk premium?
S1: Equity Risk Premium is the difference between returns on equity/individual stock and the risk-free rate of return.
S2: Let us start with defining the equity risk premium: the Equity Risk Premium is the average extra return demanded by investors, on top of a risk free rate, as a compensation for investing in equity securities with average risk.
Spelling errors (2.4%)	There are spelling errors in the answers.	Q: Are all psychopaths narcissists?
S1: I came across that all psychopats are narcissists, but not all narcissists are psychopats.
S2: I have read it summed up this way: Not all narcissists are psychopaths, but all psychopaths are narcissists.
Table 4: Different causes of wrongly classified positive pairs.
pairs, 1 (2%) is from exact match, 8 (16%) are ambiguous, and the remaining 41 (82%) are from the semantic variation category, showing that it is the most challenging type to tackle.
We further study the specific causes of errors on the 42 unambiguous pairs. Examples of distinct error causes are described in Table 4 We find that  $16.7\%$  of the falsely classified sentence pairs contain one answer that entails the other instead of expressing the exact same answers, which should however be considered redundant answers by our definition.  $80.9\%$  of pairs are equivalent but require understanding the semantic meanings or entity-specific/common sense knowledge. The rest  $2.4\%$  contain spelling errors that negatively affect model inference.
In this paper, we formulate and propose the answer consolidation task that seeks to group answers into different aspects. This process can be used to construct a final set of answers that is both comprehensive and non-redundant. We contribute the Question-Answer consolidation dataset (QUASI) for this task and evaluate various models, including sentence embedding models, cross-encoders, and answer-aware cross-encoders. While the best-performing supervised models have achieved promising performance, without that abundant an
notation, unsupervised methods still remain far from perfect. This suggests room for further studies on more robust and generalizable solutions for answer consolidation that would largely benefit real-world open-domain QA systems.
We appreciate the reviewers for their insightful comments and suggestions. Wenxuan Zhou and Muhao Chen are supported by the National Science Foundation of United States Grant IIS 2105329.
Mustafa Abdool, Malay Haldar, Prashant Ramanathan, Tyler Sax, Lanbo Zhang, Aamir Manaswala, Lynn Yang, Bradley Turnbull, Qing Zhang, and Thomas Legrand. 2020. Managing diversity in airbnb search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2952-2960.
Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520-534, Online. Association for Computational Linguistics.
Petr Baudiš and Jan Šedivý. 2015. Modeling of the question answering task in the yodaqa system. In International Conference of the cross-language evaluation Forum for European languages, pages 222-228. Springer.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Nikita Bhutani, Yoshihiko Suhara, Wang-Chiew Tan, Alon Halevy, and H. V. Jagadish. 2019. Open information extraction from question-answer pairs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2294-2305, Minneapolis, Minnesota. Association for Computational Linguistics.
Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading comprehension dataset. ArXiv, abs/1611.09268.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. 2018. Quora question pairs. URL https://www.kaggle.com/c/quora-question-pairs.
Tanya Chowdhury and Tanmoy Chakraborty. 2019. Cqasumm: Building references for community question answering summarization corpora. In Proceedings of the ACM india joint international conference on data science and management of data, pages 18-26.
Charles LA Clarke, Maheedhar Kolla, Gordon V Cormack, Olga Vechtomova, Azin Ashkan, Stefan Böttcher, and Ian MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 659-666.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671-683, Online. Association for Computational Linguistics.
Alexander R Fabbri, Xiaojian Wu, Srini Iyer, and Mona Diab. 2021. Multi-perspective abstractive answer summarization. arXiv preprint arXiv:2104.08536.
Yixing Fan, Jiafeng Guo, Yanyan Lan, Jun Xu, Chengxiang Zhai, and Xueqi Cheng. 2018. Modeling diverse relevance patterns in ad-hoc retrieval. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 375-384.
Shogo Fujita, Tomohide Shibata, and Manabu Okumura. 2020. Diverse and non-redundant answer set extraction on community QA based on DPPs. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5309-5320, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jiawei Han, Jian Pei, and Micheline Kamber. 2011. Data mining: concepts and techniques. Elsevier.
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 37-46, Melbourne, Australia. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1896–1907, Online. Association for Computational Linguistics.
Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021. GooAQ: Open question answering with
diverse answer types. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 421-433, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Bernhard Kratzwald, Anna Eigenmann, and Stefan Feuerriegel. 2019. RankQA: Neural question answering with answer re-ranking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6076-6085, Florence, Italy. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474. Curran Associates, Inc.
Siyi Liu, Sihao Chen, Xander Uytendaele, and Dan Roth. 2021. MultiOpEd: A corpus of multiperspective news editorials. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4345-4361, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yuanjie Liu, Shasha Li, Yunbo Cao, Chin-Yew Lin, Dingyi Han, and Yong Yu. 2008. Understanding and summarizing answers in community-based question answering services. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 497-504, Manchester, UK. Coling 2008 Organizing Committee.
Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442-451.