Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

A multi-hop question answering (QA) dataset aims to test reasoning and inference skills by requiring a model to read multiple paragraphs to answer a given question.However, current datasets do not provide a complete explanation for the reasoning process from the question to the answer.Further, previous studies revealed that many examples in existing multi-hop datasets do not require multi-hop reasoning to answer a question.In this study, we present a new multihop QA dataset, called 2WikiMultiHopQA, which uses structured and unstructured data.In our dataset, we introduce the evidence information containing a reasoning path for multi-hop questions.The evidence information has two benefits: (i) providing a comprehensive explanation for predictions and (ii) evaluating the reasoning skills of a model.We carefully design a pipeline and a set of templates when generating a question-answer pair that guarantees the multi-hop steps and the quality of the questions.We also exploit the structured format in Wikidata and use logical rules to create questions that are natural but still require multi-hop reasoning.Through experiments, we demonstrate that our dataset is challenging for multi-hop models and it ensures that multi-hop reasoning is required.Machine reading comprehension (MRC) aims at teaching machines to read and understand given text.Many current models (Devlin et al., 2019;Liu et al., 2019;Yang et al., 2019) have defeated humans on the performance of SQuAD (Rajpurkar et al., 2016;Rajpurkar et al., 2018), as shown on its leaderboard 1 .However, such performances do not indicate that these models can completely understand the text.Specifically, using an adversarial method, Jia and Liang (2017) demonstrated that the current models do not precisely understand natural language.Moreover, Sugawara et al. ( 2018) demonstrated that many datasets contain a considerable number of easy instances that can be answered based on the first few words of the questions.Multi-hop MRC datasets require a model to read and perform multi-hop reasoning over multiple paragraphs to answer a question.Currently, there are four multi-hop datasets over textual data: ComplexWe-bQuestions (Talmor and Berant, 2018), QAngaroo (Welbl et al., 2018), HotpotQA (Yang et al., 2018), and R 4 C (Inoue et al., 2020).The first two datasets were created by incorporating the documents (from Web or Wikipedia) with a knowledge base (KB).Owing to their building procedures, these datasets have no information to explain the predicted answers.Meanwhile, the other two datasets were created mainly based on crowdsourcing.In HotpotQA, the authors introduced the sentence-level supporting facts (SFs) information that are used to explain the predicted answers.However, as discussed in Inoue et al. ( 2020), the task of classifying sentence-level SFs is a binary classification task that is incapable of evaluating the reasoning and inference skills of the model.Further, data analyses (Chen and Durrett, 2019;Min et al., 2019) revealed that many examples in HotpotQA do not require multi-hop reasoning to solve.Recently, to evaluate the internal reasoning of the reading comprehension system, Inoue et al.