The developed crowdsourcing annotation framework can be used for annotating other QA datasets with derivations. Through an experiment using two baseline models, we highlight several challenges of RC-QED. We will make the corpus of reasoning annotations and the baseline system publicly available at https://naoya-i.github.io/rc-qed/.

Data collection for RC-QED@!START@$^{\rm E}$@!END@ ::: Crowdsourcing interface	To acquire a large-scale corpus of NLDs, we use crowdsourcing (CS). Although CS is a powerful tool for large-scale dataset creation , , quality control for complex tasks is still challenging. We thus carefully design an incentive structure for crowdworkers, following Yang2018HotpotQA:Answering. Initially, we provide crowdworkers with an instruction with example annotations, where we emphasize that they judge the truth of statements solely based on given articles, not based on their own knowledge.

Experiments ::: Settings ::: Baseline	To check the integrity of the PRKGC model, we created a simple baseline model (shortest path model). It outputs a candidate entity with the shortest path length from a query entity on $G(S)$ as an answer. Similarly to the PRKGC model, it traverses the path to generate NLDs. It outputs Unanswerable if (i) a query entity is not reachable to any candidate entities on $G(S)$ or (ii) the shortest path length is more than 3.

We deployed the task on Amazon Mechanical Turk (AMT). To see how reasoning varies across workers, we hire 3 crowdworkers per one instance. We hire reliable crowdworkers with $\ge 5,000$ HITs experiences and an approval rate of $\ge $ 99.0%, and pay ¢20 as a reward per instance. Our data collection pipeline is expected to be applicable to other types of QAs other than entity-based multi-hop QA without any significant extensions, because the interface is not specifically designed for entity-centric reasoning.

Related work ::: RC datasets with explanations	There exists few RC datasets annotated with explanations (Table TABREF50). The most similar work to ours is Science QA dataset , , , which provides a small set of NLDs annotated for analysis purposes. By developing the scalable crowdsourcing framework, our work provides one order-of-magnitude larger NLDs which can be used as a benchmark more reliably. In addition, it provides the community with new types of challenges not included in HotpotQA.

Example annotations	Table TABREF53 shows examples of crowdsourced annotations.

Conclusions	Towards RC models that can perform correct reasoning, we have proposed RC-QED that requires a system to output its introspective explanations, as well as answers. Instantiating RC-QED with entity-based multi-hop QA (RC-QED$^{\rm E}$), we have created a large-scale corpus of NLDs. The developed crowdsourcing annotation framework can be used for annotating other QA datasets with derivations. Our experiments using two simple baseline models have demonstrated that RC-QED$^{\rm E}$ is a non-trivial task, and that it indeed provides a challenging task of extracting and synthesizing relevant facts from supporting documents. We will make the corpus of reasoning annotations and baseline systems publicly available at https://naoya-i.github.io/rc-qed/. One immediate future work is to expand the annotation to non-entity-based multi-hop QA datasets such as HotpotQA . For modeling, we plan to incorporate a generative mechanism based on recent advances in conditional language modeling.

Experiments ::: Settings ::: Dataset	We aggregated crowdsourced annotations obtained in Section SECREF3. As a preprocessing, we converted the NLD annotation to Unsure if the derivation contains the phrase needs to be mentioned. This is due to the fact that annotators misunderstand our instruction. When at least one crowdworker state that a statement is Unsure, then we set the answerability to Unanswerable and discard NLD annotations. Otherwise, we employ all NLD annotations from workers as multiple reference NLDs. The statistics is shown in Table TABREF36. Regarding $\mathcal {K}^+, \mathcal {K}^-$, we extracted 867,936 instances from the training set of WikiHop . We reserve 10% of these instances as a validation set to find the best model. For $\mathcal {D}$, we used Answerable questions in the training set. To create supervision of path (i.e. $\mathbf {p}_i$), we selected the path that is most similar to all NLD annotations in terms of ROUGE-L F1.

Data collection for RC-QED@!START@$^{\rm E}$@!END@ ::: Crowdsourcing interface ::: Judgement task (Figure @!START@UID13@!END@).	Given a statement and articles, workers are asked to judge whether the statement can be derived from the articles at three grades: True, Likely (i.e. Answerable), or Unsure (i.e. Unanswerable). If a worker selects Unsure, we ask workers to tell us why they are unsure from two choices (“Not stated in the article” or “Other”).

Data collection for RC-QED@!START@$^{\rm E}$@!END@ ::: Dataset	Our study uses WikiHop , as it is an entity-based multi-hop QA dataset and has been actively used. We randomly sampled 10,000 instances from 43,738 training instances and 2,000 instances from 5,129 validation instances (i.e. 36,000 annotation tasks were published on AMT). We manually converted structured WikiHop question-answer pairs (e.g. locatedIn(Macchu Picchu, Peru)) into natural language statements (Macchu Picchu is located in Peru) using a simple conversion dictionary. We use supporting documents provided by WikiHop. WikiHop collects supporting documents by finding Wikipedia articles that bridges a question entity $e_i$ and an answer entity $e_j$, where the link between articles is given by a hyperlink.

Experiments ::: Settings ::: Hyperparameters	We used 100-dimensional vectors for entities, relations, and textual relation representations. We initialize these representations with 100-dimensional Glove Embeddings  and fine-tuned them during training. We retain only top-100,000 frequent words as a model vocabulary. We used Bi-LSTM with 50 dimensional hidden state as a textual relation encoder, and an LSTM with 100-dimensional hidden state as an entity encoder. We used the Adam optimizer (default parameters)  with a batch size of 32. We set the answerability threshold $\epsilon _k = 0.5$.

Although we restrict the form of knowledge to entity relations, we use a natural language form to represent $r_i$ rather than a closed vocabulary (see Figure FIGREF1 for an example).

Baseline RC-QED@!START@$^{\rm E}$@!END@ model ::: Training	Let $\mathcal {K}^+$ be a set of question-answer pairs, where each instance consists of a triplet (a query entity $q_i$, a relation $r_i$, an answer entity $a_i$). Similarly, let $\mathcal {K}^-$ be a set of question-non-answer pairs. We minimize the following binary cross-entropy loss: From the NLD point of view, this is unsupervised training. The model is expected to learn the score function ${\rm sc(\cdot )}$ to give higher scores to paths (i.e. NLD steps) that are useful for discriminating correct answers from wrong answers by its own. Highly scored NLDs might be useful for answer classification, but these are not guaranteed to be interpretable to humans.

Data collection for RC-QED@!START@$^{\rm E}$@!END@ ::: Crowdsourcing interface ::: Derivation task (Figure @!START@UID14@!END@).	If a worker selects True or Likely in the judgement task, we first ask which sentences in the given articles are justification explanations for a given statement, similarly to HotpotQA . The “summary” text boxes (i.e. NLDs) are then initialized with these selected sentences. We give a ¢6 bonus to those workers who select True or Likely. To encourage an abstraction of selected sentences, we also introduce a gamification scheme to give a bonus to those who provide shorter NLDs. Specifically, we probabilistically give another ¢14 bonus to workers according to a score they gain. The score is always shown on top of the screen, and changes according to the length of NLDs they write in real time. To discourage noisy annotations, we also warn crowdworkers that their work would be rejected for noisy submissions. We periodically run simple filtering to exclude noisy crowdworkers (e.g. workers who give more than 50 submissions with the same answers).

This indicates that the challenge of RC-QED$^{\rm E}$ is in how to extract relevant information from supporting documents and synthesize these multiple facts to derive an answer. To obtain further insights, we manually analyzed generated NLDs. Table TABREF44 (a) illustrates a positive example, where the model identifies that altudoceras belongs to pseudogastrioceratinae, and that pseudogastrioceratinae is a subfamily of paragastrioceratidae. Some supporting sentences are already similar to human-generated NLDs, thus simply extracting textual relations works well for some problems. On the other hand, typical derivation error is from non-human readable textual relations. In (b), the model states that bumped has a relationship of “,” with hands up, which is originally extracted from one of supporting sentences It contains the UK Top 60 singles “Bumped”, “Hands Up (4 Lovers)” and .... This provides a useful clue for answer prediction, but is not suitable as a derivation. One may address this issue by incorporating, for example, a relation extractor or a paraphrasing mechanism using recent advances of conditional language models .

Baseline RC-QED@!START@$^{\rm E}$@!END@ model ::: Path ranking-based KGC (PRKGC)	Given a question $Q=(q, r)$ and a candidate entity $c_i$, we estimate the plausibility of $(q, r, c_i)$ as follows: where $\sigma $ is a sigmoid function, and $\mathbf {q, r, c_i}, \mathbf {\pi }(q, c_i)$ are vector representations of $q, r, c_i$ and a set $\pi (q, c_i)$ of shortest paths between $q$ and $c_i$ on $G(S)$. ${\rm MLP}(\cdot , \cdot )$ denotes a multi-layer perceptron. To encode entities into vectors $\mathbf {q, c_i}$, we use Long-Short Term Memory (LSTM) and take its last hidden state.

The attention values are calculated as follows: $\alpha _j = \exp ({\rm sc}(q, r, c_i, \pi _j)) / \sum _k \exp ({\rm sc}(q, r, c_i, \pi _k))$, where ${\rm sc}(q, r, c_i, \pi _j) = {\rm MLP}(\mathbf {q}, \mathbf {r}, \mathbf {c_i}, \mathbf {\pi _j})$. To obtain individual path representations $\mathbf {\pi _j}$, we follow toutanova-etal-2015-representing. We use a Bi-LSTM  with mean pooling over timestep in order to encourage similar paths to have similar path representations. For the testing phase, we choose a candidate entity $c_i$ with the maximum probability $P(r|q, c_i)$ as an answer entity, and choose a path $\pi _j$ with the maximum attention value $\alpha _j$ as NLDs. To generate NLDs, we simply traverse the path from $q$ to $c_i$ and subsequently concatenate all entities and textual relations as one string.

Yang2018HotpotQA:Answering propose HotpotQA, an “explainable” multi-hop Question Answering (QA) task that requires a system to identify a set of sentences containing supporting evidence for the given answer. We follow the footsteps of Yang2018HotpotQA:Answering and explore an explainable multi-hop QA task. In the community, two important types of explanations have been explored so far : (i) introspective explanation (how a decision is made), and (ii) justification explanation (collections of evidences to support the decision). In this sense, supporting facts in HotpotQA can be categorized as justification explanations. The advantage of using justification explanations as benchmark is that the task can be reduced to a standard classification task, which enables us to adopt standard evaluation metrics (e.g. a classification accuracy). However, this task setting does not evaluate a machine's ability to (i) extract relevant information from justification sentences and (ii) synthesize them to form coherent logical reasoning steps, which are equally important for NLU.

Baseline RC-QED@!START@$^{\rm E}$@!END@ model	To highlight the challenges and nature of RC-QED$^{\rm E}$, we create a simple, transparent, and interpretable baseline model. Recent studies on knowledge graph completion (KGC) explore compositional inferences to combat with the sparsity of knowledge bases , , . Given a query triplet $(h, r, t)$ (e.g. (Macchu Picchu, locatedIn, Peru)), a path ranking-based approach for KGC explicitly samples paths between $h$ and $t$ in a knowledge base (e.g. Macchu Picchu—locatedIn—Andes Mountain—countryOf—Peru), and construct a feature vector of these paths. This feature vector is then used to calculate the compatibility between the query triplet and the sampled paths. RC-QED$^{\rm E}$ can be naturally solved by path ranking-based KGC (PRKGC), where the query triplet and the sampled paths correspond to a question and derivation steps, respectively. PRKGC meets our purposes because of its glassboxness: we can trace the derivation steps of the model easily.

Data collection for RC-QED@!START@$^{\rm E}$@!END@ ::: Results ::: Agreement	For agreement on the number of NLDs, we obtained a Krippendorff's $\alpha $ of 0.223, indicating a fair agreement . Our manual inspection of the 10 worst disagreements revealed that majority (7/10) come from Unsure v.s. non-Unsure. It also revealed that crowdworkers who labeled non-Unsure are reliable—6 out 7 non-Unsure annotations can be judged as correct. This partially confirms the effectiveness of our incentive structure.

Crowdworkers found that 45.3% of 294 (out of 900) 3-step NLDs has missing steps to derive a statement. Let us consider this example: for annotated NLDs “[1] Kouvola is located in Helsinki. [2] Helsinki is in the region of Uusimaa. [3] Uusimaa borders the regions Southwest Finland, Kymenlaakso and some others.” and for the statement “Kouvola is located in Kymenlaakso”, one worker pointed out the missing step “Uusimaa is in Kymenlaakso.”. We speculate that greater steps of reasoning make it difficult for crowdworkers to check the correctness of derivations during the writing task.

Introduction	Reading comprehension (RC) has become a key benchmark for natural language understanding (NLU) systems and a large number of datasets are now available , , . However, these datasets suffer from annotation artifacts and other biases, which allow systems to “cheat”: Instead of learning to read texts, systems learn to exploit these biases and find answers via simple heuristics, such as looking for an entity with a matching semantic type , . To give another example, many RC datasets contain a large number of “easy” problems that can be solved by looking at the first few words of the question Sugawara2018. In order to provide a reliable measure of progress, an RC dataset thus needs to be robust to such simple heuristics. Towards this goal, two important directions have been investigated. One direction is to improve the dataset itself, for example, so that it requires an RC system to perform multi-hop inferences  or to generate answers . Another direction is to request a system to output additional information about answers.

Related work ::: Other NLP corpora annotated with explanations	There are existing NLP tasks that require models to output explanations (Table TABREF50). FEVER  requires a system to judge the “factness” of a claim as well as to identify justification sentences. As discussed earlier, we take a step further from justification explanations to provide new challenges for NLU. Several datasets are annotated with introspective explanations, ranging from textual entailments  to argumentative texts , , . All these datasets offer the classification task of single sentences or sentence pairs. The uniqueness of our dataset is that it measures a machine's ability to extract relevant information from a set of documents and to build coherent logical reasoning steps.

To address this issue, we propose RC-QED, an RC task that requires not only the answer to a question, but also an introspective explanation in the form of a natural language derivation (NLD). For example, given the question “Which record company released the song Barracuda?” and supporting documents shown in Figure FIGREF1, a system needs to give the answer “Portrait Records” and to provide the following NLD: 1.) Barracuda is on Little Queen, and 2.) Little Queen was released by Portrait Records. The main difference between our work and HotpotQA is that they identify a set of sentences $\lbrace s_2,s_4\rbrace $, while RC-QED requires a system to generate its derivations in a correct order. This generation task enables us to measure a machine's logical reasoning ability mentioned above. Due to its subjective nature of the natural language derivation task, we evaluate the correctness of derivations generated by a system with multiple reference answers. Our contributions can be summarized as follows: We create a large corpus consisting of 12,000 QA pairs and natural language derivations.

Related work ::: Analysis of RC models and datasets	There is a large body of work on analyzing the nature of RC datasets, motivated by the question to what degree RC models understand natural language , . Several studies suggest that current RC datasets have unintended bias, which enables RC systems to rely on a cheap heuristics to answer questions. For instance, Sugawara2018 show that some of these RC datasets contain a large number of “easy” questions that can be solved by a cheap heuristics (e.g. by looking at a first few tokens of questions). Responding to their findings, we take a step further and explore the new task of RC that requires RC systems to give introspective explanations as well as answers. In addition, recent studies show that current RC models and NLP models are vulnerable to adversarial examples , , . Explicit modeling of NLDs is expected to reguralize RC models, which could prevent RC models' strong dependence on unintended bias in training data (e.g. annotation artifact) , , , , as partially confirmed in Section SECREF46.

Task formulation: RC-QED ::: Input, output, and evaluation metrics	We formally define RC-QED as follows: Given: (i) a question $Q$, and (ii) a set $S$ of supporting documents relevant to $Q$; Find: (i) answerability $s \in \lbrace \textsf {Answerable},$ $\textsf {Unanswerable} \rbrace $, (ii) an answer $a$, and (iii) a sequence $R$ of derivation steps. We evaluate each prediction with the following evaluation metrics: Answerability: Correctness of model's decision on answerability (i.e. binary classification task) evaluated by Precision/Recall/F1. Answer precision: Correctness of predicted answers (for Answerable predictions only). We follow the standard practice of RC community for evaluation (e.g. an accuracy in the case of multiple choice QA). Derivation precision: Correctness of generated NLDs evaluated by ROUGE-L  (RG-L) and BLEU-4 (BL-4) . We follow the standard practice of evaluation for natural language generation . Derivation steps might be subjective, so we resort to multiple reference answers.

Task formulation: RC-QED ::: RC-QED@!START@$^{\rm E}$@!END@	This paper instantiates RC-QED by employing multiple choice, entity-based multi-hop QA  as a testbed (henceforth, RC-QED$^{\rm E}$). In entity-based multi-hop QA, machines need to combine relational facts between entities to derive an answer. For example, in Figure FIGREF1, understanding the facts about Barracuda, Little Queen, and Portrait Records stated in each article is required. This design choice restricts a problem domain, but it provides interesting challenges as discussed in Section SECREF46. In addition, such entity-based chaining is known to account for the majority of reasoning types required for multi-hop reasoning .

More formally, given (i) a question $Q=(r, q)$ represented by a binary relation $r$ and an entity $q$ (question entity), (ii) relevant articles $S$, and (iii) a set $C$ of candidate entities, systems are required to output (i) an answerability $s \in \lbrace \textsf {Answerable}, \textsf {Unanswerable} \rbrace $, (ii) an entity $e \in C$ (answer entity) that $(q, r, e)$ holds, and (iii) a sequence $R$ of derivation steps as to why $e$ is believed to be an answer. We define derivation steps as an $m$ chain of relational facts to derive an answer, i.e. $(q, r_1, e_1), (e_1, r_2, e_2), ..., (e_{m-1}, r_{m-1}, e_m),$ $(e_m, r_m, e_{m+1}))$.

Experiments ::: Results and discussion ::: QA performance.	To check the integrity of our baseline models, we compare our baseline models with existing neural models tailored for QA under the pure WikiHop setting (i.e. evaluation with only an accuracy of predicted answers). Note that these existing models do not output derivations. We thus cannot make a direct comparison, so it servers as a reference purpose. Because WikiHop has no answerability task, we enforced the PRKGC model to always output answers. As shown in Table TABREF45, the PRKGC models achieve a comparable performance to other sophisticated neural models.

Experiments ::: Results and discussion	As shown in Table TABREF37, the PRKGC models learned to reason over more than simple shortest paths. Yet, the PRKGC model do not give considerably good results, which indicates the non-triviality of RC-QED$^{\rm E}$. Although the PRKGC model do not receive supervision about human-generated NLDs, paths with the maximum score match human-generated NLDs to some extent. Supervising path attentions (the PRKGC+NS model) is indeed effective for improving the human interpretability of generated NLDs. It also improves the generalization ability of question answering. We speculate that $L_d$ functions as a regularizer, which helps models to learn reasoning that helpful beyond training data. This observation is consistent with previous work where an evidence selection task is learned jointly with a main task , , . As shown in Table TABREF43, as the required derivation step increases, the PRKGC+NS model suffers from predicting answer entities and generating correct NLDs.

Baseline RC-QED@!START@$^{\rm E}$@!END@ model ::: Training ::: Semi-supervising derivations	To address the above issue, we resort to gold-standard NLDs to guide the path scoring function ${\rm sc(\cdot )}$. Let $\mathcal {D}$ be question-answer pairs coupled with gold-standard NLDs, namely a binary vector $\mathbf {p}_i$, where the $j$-th value represents whether $j$-th path corresponds to a gold-standard NLD (1) or not (0). We apply the following cross-entropy loss to the path attention:

We output Unanswerable when (i) $\max _{c_i \in C} P(r|q, c_i) < \epsilon _k$ or (ii) $G(S)$ has no path between $q$ and all $c_i \in C$.

For example, in Figure FIGREF27, $q =$ Barracuda and $c_i =$ Portrait Records yield $\pi (q, c_i) = \lbrace $Barracuda—is the most popular in their album—Little Queen—was released in May 1977 on—Portrait Records, Barracuda—was released from American band Heart—is the second album released by:-1—Little Queen—was released in May 1977 on—Portrait Records$\rbrace $. To obtain path representations $\mathbf {\pi }(q, c_i)$, we attentively aggregate individual path representations: $\mathbf {\pi }(q, c_i) = \sum _j \alpha _j \mathbf {\pi _j}(q, c_i)$, where $\alpha _j$ is an attention for the $j$-th path.

Baseline RC-QED@!START@$^{\rm E}$@!END@ model ::: Knowledge graph construction	Given supporting documents $S$, we build a knowledge graph. We first apply a coreference resolver to $S$ and then create a directed graph $G(S)$. Therein, each node represents named entities (NEs) in $S$, and each edge represents textual relations between NEs extracted from $S$. Figure FIGREF27 illustrates an example of $G(S)$ constructed from supporting documents in Figure FIGREF1.

Data collection for RC-QED@!START@$^{\rm E}$@!END@ ::: Results	Table TABREF17 shows the statistics of responses and example annotations. Table TABREF17 also shows the abstractiveness of annotated NLDs ($a$), namely the number of tokens in an NLD divided by the number of tokens in its corresponding justification sentences. This indicates that annotated NLDs are indeed summarized. See Table TABREF53 in Appendix and Supplementary Material for more results.

Data collection for RC-QED@!START@$^{\rm E}$@!END@ ::: Results ::: Quality	To evaluate the quality of annotation results, we publish another CS task on AMT. We randomly sample 300 True and Likely responses in this evaluation. Given NLDs and a statement, 3 crowdworkers are asked if the NLDs can lead to the statement at four scale levels. If the answer is 4 or 3 (“yes” or “likely”), we additionally asked whether each derivation step can be derived from each supporting document; otherwise we asked them the reasons. For a fair evaluation, we encourage crowdworkers to annotate given NLDs with a lower score by stating that we give a bonus if they found a flaw of reasoning on the CS interface. The evaluation results shown in Table TABREF24 indicate that the annotated NLDs are of high quality (Reachability), and each NLD is properly derived from supporting documents (Derivability). On the other hand, we found the quality of 3-step NLDs is relatively lower than the others.

RC-QED: Evaluating Natural Language Derivations in Multi-Hop Reading Comprehension	Recent studies revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This allows systems to "cheat" by employing simple heuristics to answer questions, e.g. by relying on semantic type consistency. This means that current datasets are not well-suited to evaluate RC systems. To address this issue, we introduce RC-QED, a new RC task that requires giving not only the correct answer to a question, but also the reasoning employed for arriving at this answer. For this, we release a large benchmark dataset consisting of 12,000 answers and corresponding reasoning in form of natural language derivations. Experiments show that our benchmark is robust to simple heuristics and challenging for state-of-the-art neural path ranking approaches.

Question: Did they use any crowdsourcing platform?
Answer: Yes