SlowGuess/ACL_2020_OCR huggingface dataset:
Vikas Yadav, Steven Bethard, Mihai Surdeanu
University of Arizona, Tucson, AZ, USA
{vikasy, bethard, msurdeanu}@email.arizona.edu
Evidence retrieval is a critical stage of question answering (QA), necessary not only to improve performance, but also to explain the decisions of the corresponding QA method. We introduce a simple, fast, and unsupervised iterative evidence retrieval method, which relies on three ideas: (a) an unsupervised alignment approach to soft-align questions and answers with justification sentences using only GloVe embeddings, (b) an iterative process that reformulates queries focusing on terms that are not covered by existing justifications, which (c) a stopping criterion that terminates retrieval when the terms in the given question and candidate answers are covered by the retrieved justifications. Despite its simplicity, our approach outperforms all the previous methods (including supervised methods) on the evidence selection task on two datasets: MultiRC and QASC. When these evidence sentences are fed into a RoBERTa answer classification component, we achieve state-of-the-art QA performance on these two datasets.
Explainability in machine learning (ML) remains a critical unsolved challenge that slows the adoption of ML in real-world applications (Biran and Cotton, 2017; Gilpin et al., 2018; Alvarez-Melis and Jaakkola, 2017; Arras et al., 2017).
Question answering (QA) is one of the challenging natural language processing (NLP) tasks that benefits from explainability. In particular, multi-hop QA requires the aggregation of multiple evidence facts in order to answer complex natural language questions (Yang et al., 2018). Several multi-hop QA datasets have been proposed recently (Yang et al., 2018; Khashabi et al., 2018a; Welbl et al., 2018; Dua et al., 2019; Chen and Durrett, 2019; Khot et al., 2019a; Sun et al., 2019b; Jansen and Ustalov, 2019; Rajpurkar et al., 2018). While
several neural methods have achieved state-of-the-art results on these datasets (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019), we argue that many of these directions lack a human-understandable explanation of their inference process, which is necessary to transition these approaches into real-world applications. This is especially critical for multi-hop, multiple choice QA (MCQA) where: (a) the answer text may not come from an actual knowledge base passage, and (b) reasoning is required to link the candidate answers to the given question (Yadav et al., 2019b). Figure 1 shows one such multi-hop example from a MCQA dataset.
In this paper we introduce a simple  $a$  alignment-based iterative retriever (AIR), which retrieves high-quality evidence sentences from unstructured knowledge bases. We demonstrate that these evidence sentences are useful not only to explain the required reasoning steps that answer a question, but they also considerably improve the performance of the QA system itself.
Unlike several previous works that depend on supervised methods for the retrieval of justification sentences (deployed mostly in settings that rely on small sets of candidate texts, e.g., HotPotQA, MultiRC), AIR is completely unsupervised and scales easily from QA tasks that use small sets of candidate evidence texts to ones that rely on large knowledge bases (e.g., QASC (Khot et al., 2019a)). AIR retrieves justification sentences through a simple iterative process. In each iteration, AIR uses an alignment model to find justification sentences that are closest in embedding space to the current query (Kim et al., 2017; Yadav et al., 2018), which is initialized with the question and candidate answer text. After each iteration, AIR adjusts its query to focus on the missing information (Khot et al., 2019b) in the current set of justifications.
1https://github.com/vikas95/AIR-retriever
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4514-4525 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
AIR also conditionally expands the query using the justifications retrieved in the previous steps.
In particular, our key contributions are:
(1) We develop a simple, fast, and unsupervised iterative evidence retrieval method, which achieves state-of-the-art results on justification selection on two multi-hop QA datasets: MultiRC (Khashabi et al., 2018a) and QASC (Khot et al., 2019a). Notably, our simple unsupervised approach that relies solely on GloVe embeddings (Pennington et al., 2014) outperforms three transformer-based supervised state-of-the-art methods: BERT (Devlin et al., 2019), XLnet (Yang et al., 2019) and RoBERTa (Liu et al., 2019) on the justification selection task. Further, when the retrieved justifications are fed into a QA component based on RoBERTa (Liu et al., 2019), we obtain the best QA performance on the development sets of both MultiRC and QASC.
(2) AIR can be trivially extended to capture parallel evidence chains by running multiple instances of AIR in parallel starting from different initial evidence sentences. We show that aggregating multiple parallel evidences further improves the QA performance over the vanilla AIR by  $3.7\%$  EM0 on the MultiRC and  $5.2\%$  accuracy on QASC datasets (both absolute percentages on development sets). Thus, with 5 parallel evidences from AIR we obtain  $36.3\%$  EM0 on MultiRC and  $81.0\%$  accuracy on QASC hidden test sets (on their respective leaderboards). To our knowledge from published works, these results are the new state-of-the-art QA results on these two datasets. These scores are also accompanied by new state-of-the-art performance on evidence retrieval on both the datasets, which emphasizes the interpretability of AIR.
(3) We demonstrate that AIR's iterative process that focuses on missing information is more robust to semantic drift. We show that even the supervised RoBERTa-based retriever trained to retrieve evidences iteratively, suffers substantial drops in performance with retrieval from consecutive hops.
Our work falls under the revitalized direction that focuses on the interpretability of QA systems, where the machine's inference process is explained to the end user in natural language evidence text (Qi et al., 2019; Yang et al., 2018; Wang et al., 2019b; Yadav et al., 2019b; Bauer et al., 2018). Several
Question: Exposure to oxygen and water can cause iron to
(A) decrease strength (B) melt (C) uncontrollable burning (D) thermal expansion (E) turn orange on the surface (F) vibrate (G) extremes of temperature (H) levitate
Gold justification sentences:
1. when a metal rusts, that metal becomes orange on the surface
2. Iron rusts in the presence of oxygen and water.
Parallel evidence chain 1:
1. Dissolved oxygen in water usually causes the oxidation of iron.
2. When iron combines with oxygen it turns orange.
Parallel evidence chain 2:
1. By preventing the exposure of the metal surface to oxygen, oxidation is prevented.
2. When iron oxidizes, it rusts.
Figure 1: An example question that requires multi-hop reasoning, together with its gold justifications from the QASC dataset. The two parallel evidence chains retrieved by AIR (see section 3) provide imperfect but relevant explanations for the given question.
datasets in support of interpretable QA have been proposed recently. For example, datasets such as HotPotQA, MultiRC, QASC, Worldtree Corpus, etc., (Yang et al., 2018; Khashabi et al., 2018a; Khot et al., 2019a; Jansen and Ustalov, 2019) provide annotated evidence sentences enabling the automated evaluation of interpretability via evidence text selection.
QA approaches that focus on interpretability can be broadly classified into three main categories: supervised, which require annotated justifications at training time, latent, which extract justification sentences through latent variable methods driven by answer quality, and, lastly, unsupervised ones, which use unsupervised algorithms for evidence extraction.
In the first class of supervised approaches, a supervised classifier is normally trained to identify correct justification sentences driven by a query (Nie et al., 2019; Tu et al., 2019; Banerjee, 2019). Many systems tend to utilize a multi-task learning setting to learn both answer extraction and justification selection with the same network (Min et al., 2018; Gravina et al., 2018). Although these approaches have achieved impressive performance, they rely on annotated justification sentences, which may not be always available. Few approaches have used distant supervision methods (Lin et al., 2018; Wang et al., 2019b) to create noisy training data for evidence retrieval but these usually underperform due to noisy labels.
In the latent approaches for selecting justifica
2 In settings where external labeled resources are not used.
tions, reinforcement learning (Geva and Berant, 2018; Choi et al., 2017) and PageRank (Surdeanu et al., 2008) have been widely used to select justification sentences without explicit training data. While these directions do not require annotated justifications, they tend to need large amounts of question/correct answer pairs to facilitate the identification of latent justifications.
In unsupervised approaches, many QA systems have relied on structured knowledge base (KB) QA. For example, several previous works have used ConceptNet (Speer et al., 2017) to keep the QA process interpretable (Khashabi et al., 2018b; Sydorova et al., 2019). However, the construction of such structured knowledge bases is expensive, and may need frequent updates. Instead, in this work we focus on justification selection from textual (or unstructured) KBs, which are inexpensive to build and can be applied in several domains. In the same category of unsupervised approaches, conventional information retrieval (IR) methods such as BM25 (Chen et al., 2017) have also been widely used to retrieve independent individual sentences. As shown by (Khot et al., 2019a; Qi et al., 2019), and our table 2, these techniques do not work well for complex multi-hop questions, which require knowledge aggregation from multiple related justifications. Some unsupervised methods extract groups of justification sentences (Chen et al., 2019; Yadav et al., 2019b) but these methods are exponentially expensive in the retrieval step. Contrary to all of these, AIR proposes a simpler and more efficient method for chaining justification sentences.
Recently, many supervised iterative justification retrieval approaches for QA have been proposed (Qi et al., 2019; Feldman and El-Yaniv, 2019; Banerjee, 2019; Das et al., 2018). While these were shown to achieve good evidence selection performance for complex questions when compared to earlier approaches that relied on just the original query (Chen et al., 2017; Yang et al., 2018), they all require supervision.
As opposed to all these iterative-retrieval methods and previously discussed directions, our proposed approach AIR is completely unsupervised, i.e., it does not require annotated justifications. Further, unlike many of the supervised iterative approaches (Feldman and El-Yaniv, 2019; Sun et al., 2019a) that perform query reformulation in a continuous representation space, AIR employs a simpler and more interpretable query reformulation
strategy that relies on explicit terms from the query and the previously retrieved justification. Lastly, none of the previous iterative retrieval approaches address the problem of semantic drift, whereas AIR accounts for drift by controlling the query reformulation as explained in section 3.1.
As shown in fig. 2, the proposed QA approach consists of two components: (a) an unsupervised, iterative component that retrieves chains of justification sentences given a query; and (b) an answer classification component that classifies a candidate answer as correct or not, given the original question and the previously retrieved justifications. We detail these components in the next two sub-sections.
3.1 Iterative Justification Retrieval
AIR iteratively builds justification chains given a query. AIR starts by initializing the query with the concatenated question and candidate answer  $\text{text}^3$ . Then, AIR iteratively repeats the following two steps: (a) It retrieves the most salient justification sentence given the current query using an alignment-IR approach(Yadav et al., 2019a). The candidate justification sentences come from dataset-specific KBs. For example, in MultiRC, we use as candidates all the sentences from the paragraph associated with the given question. In QASC, which has a large  $\text{KB}^4$  of 17.4 million sentences), similar to Khot et al. (2019a) candidates are retrieved using the Heuristic+IR method which returns 80 candidate sentences for each candidate answer from the provided QASC KB. (b) it adjusts the query to focus on the missing information, i.e., the keywords that are not covered by the current evidence chain. AIR also dynamically adds new terms to the query from the previously retrieved justifications to nudge multi-hop retrieval. These two iterative steps repeat until a parameter-free termination condition is reached.
We first detail the important components of AIR.
Alignment: To compute the similarity score between a given query and a sentence from KB, AIR
<sup>3</sup>Note that this work can be trivially adapted to reading comprehension tasks. In such tasks (e.g., SQuAD (Rajpurkar et al., 2018)), the initial query would contain just the question text.
4In large KB-based QA, AIR first uses an off-the-shelf Lucene BM25(Robertson et al., 2009) to retrieve a pool of candidate justification sentences from which the evidence chains are constructed.
Figure 2: A walkthrough example showing the iterative retrieval of justification sentences by AIR on MultiRC. Each current query includes keywords from the original query (which consists of question + candidate answer) that are not covered by previously retrieved justifications (see  $2^{nd}$  hop). If the number of uncovered keywords is too small, the query is expanded with keywords from the most recent justification ( $3^{rd}$  hop). The retrieval process terminates when all query terms are covered by existing justifications.  $Q_{c}$  indicates the proportion of query terms covered in the justifications;  $Q_{r}$  indicates the query terms which are still not covered by the justifications. AIR can retrieve parallel justification chains by running the retrieval process in parallel, starting from different candidates for the first justification sentence in a chain.
uses a vanilla unsupervised alignment method of Yadav et al. (2019a) which uses only GloVe embeddings (Pennington et al., 2014). The alignment method computes the cosine similarity between the word embeddings of each token in the query and each token in the given KB sentence, resulting in a matrix of cosine similarity scores. For each query token, the algorithm selects the most similar token in the evidence text using max-pooling. At the end, the element-wise dot product between this max-pooled vector of cosine-similarity scores and the vector containing the IDF values of the query tokens is calculated to produce the overall alignment score  $s$  for the given query  $Q$  and the supporting paragraph  $P_{j}$ :
$$
s (Q, P _ {j}) = \sum_ {i = 1} ^ {| Q |} i d f (q _ {i}) \cdot a l i g n (q _ {i}, P _ {j}) \tag {1}
$$
$$
\operatorname {a l i g n} \left(q _ {i}, P _ {j}\right) = \max  _ {k = 1} ^ {| P _ {j} |} \cos \operatorname {S i m} \left(q _ {i}, p _ {k}\right) \tag {2}
$$
where  $q_{i}$  and  $p_k$  are the  $i^{th}$  and  $k^{th}$  terms of the query  $(Q)$  and evidence sentence  $(P_j)$  respectively.
Remainder terms  $(Q_r)$ : Query reformulation in AIR is driven by the remainder terms, which are the set of query terms not yet covered in the justification set of  $i$  sentences (retrieved from the first  $i$  iterations of the retrieval process):
$$
Q _ {r} (i) = t (Q) - \bigcup_ {s _ {k} \in S _ {i}} t \left(s _ {k}\right) \tag {3}
$$
where  $t(Q)$  represents the unique set of query terms,  $t(s_k)$  represents the unique terms of the  $k^{th}$
Alignment based on BERT embeddings marginally outperformed the one based on GloVe embeddings, but BERT embeddings were much more expensive to generate.
justification, and  $S_{i}$  represents the set of  $i$  justification sentences. Note that we use soft matching of alignment for the inclusion operation: we consider a query term to be included in the set of terms in the justifications if its cosine similarity with a justification term is larger than a similarity threshold  $M$  (we use  $M = 0.95$  for all our experiments - see section 5.2), thus ensuring that the two terms are similar in the embedding space.
Coverage  $(Q_{c})$ : measures the coverage of the query keywords by the retrieved chain of justifications  $S$ :
$$
Q _ {c} (i) = \frac {\left| \bigcup_ {s _ {k} \in S _ {i}} t (Q) \cap t \left(s _ {k}\right) \right|}{\left| t (Q) \right|} \tag {4}
$$
where  $|t(Q)|$  denotes the size of unique query terms.
The AIR retrieval process
Query reformulation: In each iteration  $j$ , AIR reformulates the query  $Q(j)$  to include only the terms not yet covered by the current justification chain,  $Q_{r}(j - 1)$ . See, for example, the second hop in fig. 2. To mitigate ambiguous queries, the query is expanded with the terms from all the previously retrieved justification sentences only if the number of uncovered terms is less than  $T$  (we used  $T = 2$  for MultiRC and  $T = 4$  for QASC (see section 5.2). See, for example, the third hop in fig. 2, in which the query is expanded with the terms of all the previously retrieved justification sentences. Formally:
$$
Q (j) = \left\{ \begin{array}{l l} Q _ {r} (j - 1), & \text {i f} | Q _ {r} (j - 1) | > T \\ Q _ {r} (j - 1) + (t (s _ {j - 1}) - t (Q)), & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
where  $j$  is the current iteration index.
Stopping criteria: AIR stops its iterative evidence retrieval process when either of the following conditions is true: (a) no new query terms are discovered in the last justification retrieved, i.e.,  $Q_{r}(i - 1) == Q_{r}(i)$ , or (b) all query terms are covered by justifications, i.e.,  $Q_{c} = 1$ .
3.2 Answer Classification
AIR's justification chains can be fed into any supervised answer classification method. For all experiments in this paper, we used RoBERTa (Liu et al., 2019), a state-of-the-art transformer-based method. In particular, for MultiRC, we concatenate the query (composed from question and candidate answer text) with the evidence text, with the
[SEP] token between the two texts. A sigmoid is used over the [CLS] representation to train a binary classification task (correct answer or not).
For QASC, we fine-tune RoBERTa as a multiple-choice QA (MCQA) (Wolf et al., 2019) classifier with 8 choices using a softmax layer (similar to (Khot et al., 2019a)) instead of the sigmoid. The input text consists of eight queries (from eight candidate answers) and their corresponding eight evidence texts. Unlike the case of MultiRC, it is possible to train a MCQA classifier for QASC because every question has only 1 correct answer. We had also tried the binary classification approach for QASC but it resulted in nearly  $5\%$  lower performance for majority of the experiments in table 2.
In QA tasks that rely on large KBs there may exist multiple chains of evidence that support a correct answer. This is particularly relevant in QASC, whose KB contains 17.2M facts. Figure 1 shows an example of this situation. To utilize this type of redundancy in answer classification, we extend AIR to extract parallel evidence chains. That is, to extract  $N$  parallel chains, we run AIR  $N$  times, ensuring that the first justification sentences in each chain are different (in practice, we start a new chain for each justification in the top  $N$  retrieved sentences in the first hop). After retrieving  $N$  parallel evidence chains, we take the union of all the individual justification sentences to create the supporting evidence text for that candidate answer.
We evaluated our approach on two datasets:
Multi-sentence reading comprehension (MultiRC), which is a reading comprehension dataset provided in the form of multiple-choice QA task (Khashabi et al., 2018a). Every question is based on a paragraph, which contains the gold justification sentences for each question. We use every sentence of the paragraph as candidate justifications for a given question. Here we use the original
<sup>6</sup>We used RoBERTa base with maximum sequence length of 512, batch size = 8, learning rate of 1e-5, and 5 number of epochs. RoBERTa-base always returned consistent performance on MultiRC experiments; many runs from RoBERTa-large failed to train (as explained by (Wolf et al., 2019)), and generated near random performance.
<sup>7</sup>We used similar hyperparameters as in the MultiRC experiments, but instead used RoBERTa-large, with maximum sequence length of 128.
The dataset creators make a similar observation (Khot et al., 2019a).
#	Computational steps	Supervised selection of justifications?	Method	F1m	F1a	EM0	Evidence selection
P	R	F1
DEVELOPMENT DATASET
Baselines
1	N	No	IR(paragraphs) (Khashabi et al., 2018a)	64.3	60.0	1.4	-
2	N	No	SurfaceLR (Khashabi et al., 2018a)	66.5	63.2	11.8	-
3	N	No	Entailment baseline (Trivedi et al., 2019)	51.3	50.4	-	-
Previous work
4	N	Yes	QA+NLI (Pujari and Goldwasser, 2019)	-	-	21.6	-
5	N	Yes	EERDPL + FT (Wang et al., 2019b)	70.5	67.8	13.3	-
6	N	Yes	Multee (GloVe) (Trivedi et al., 2019)	71.3	68.3	17.9	-
7	N	Yes	Multee (ELMo)* (Trivedi et al., 2019)	73.0	69.6	22.8	-
8	K × N	Yes	RS* (Sun et al., 2019c)	73.1	70.5	21.8	-	-	60.8
9	N	No	BERT + BM25 (Yadav et al., 2019b)	71.1	67.4	23.1	43.8	61.2	51.0
10	2N - N - 1	No	BERT + AutoROCC (Yadav et al., 2019b)	72.9	69.6	24.7	48.2	68.2	56.4
Alignment + RoBERTa(QA) baselines
11	-	No	Entire passage + RoBERTa	73.9	71.7	28.7	17.4	100.0	29.6
12	N	No	Alignment (k = 2 sentences) + RoBERTa	72.6	69.6	25.9	62.4	55.6	58.8
13	N	No	Alignment (k = 3 sentences) + RoBERTa	72.4	69.8	25.1	49.3	65.1	56.1
14	N	No	Alignment (k = 4 sentences) + RoBERTa	73.6	71.4	28.0	41.0	72.0	52.3
15	N	No	Alignment (k = 5 sentences) + RoBERTa	73.7	70.8	25.0	35.2	77.1	48.4
RoBERTa retriever + RoBERTa(QA) baselines
16	N	Yes	RoBERTa-retriever(All passages) + RoBERTa	70.5	68.0	24.9	63.4	61.1	62.3
17	N	Yes	RoBERTa-retriever(Fiction) + RoBERTa	72.8	70.4	24.7	47.8	73.9	58.1
18	N	Yes	RoBERTa-retriever(News) + RoBERTa	69.0	67.3	24.2	60.8	59.2	59.9
19	N	Yes	RoBERTa-retriever(Science-textbook) + RoBERTa	70.3	67.7	25.3	48.1	62.0	54.2
20	N	Yes	RoBERTa-retriever(Society_Law) + RoBERTa	72.8	70.3	25.3	50.4	68.5	58.0
21	K × N	Yes	RoBERTa-iterative-retriever + RoBERTa	70.1	67.6	24.0	67.1	58.4	62.5
RoBERTa + AIR (Parallel) Justifications
22	K × N	No	AIR (lexical) top chain + RoBERTa	71.0	68.2	22.9	58.2	49.5	53.5
23	K × N	No	AIR top chain + RoBERTa	74.7	72.3	29.3	66.2	63.1	64.2
24	2 × K × N	No	AIR Parallel evidence chains (p = 2) + RoBERTa	75.5	73.6	32.5	50.4	71.9	59.2
25	3 × K × N	No	AIR Parallel evidence chains (p = 3) + RoBERTa	75.8	73.7	30.6	40.8	76.7	53.3
26	4 × K × N	No	AIR Parallel evidence chains (p = 4) + RoBERTa	76.3	74.2	31.3	34.8	80.8	48.7
27	5 × K × N	No	AIR Parallel evidence chains (p = 5) + RoBERTa	77.2	75.1	33.0	28.6	84.1	44.9
Ceiling systems with gold justifications
29	-	Yes	EERgt + FT (Wang et al., 2019b)	72.3	70.1	19.2	-
30	-	Yes	RoBERTa + Gold knowledge	81.4	80	39	100.0	100.0	100.0
31	-	-	Human	86.4	83.8	56.6	-
TEST DATASET
32	N	No	SurfaceLR (Khashabi et al., 2018a)	66.9	63.5	12.8
33	N	Yes	Multee (ELMo)* (Trivedi et al., 2019)	73.8	70.4	24.5	-
34	2N - N - 1	No	BERT + AutoROCC (Yadav et al., 2019b)	73.8	70.6	26.1
35	5 × K × N	No	RoBERTa + AIR (Parallel evidence = 5)	79.0	76.4	36.3