Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering

1 INTRODUCTION . Open-domain Question Answering ( QA ) is the task of answering a question given a large collection of text documents ( e.g. , Wikipedia ) . Most state-of-the-art approaches for open-domain QA ( Chen et al. , 2017 ; Wang et al. , 2018a ; Lee et al. , 2018 ; Yang et al. , 2019 ) leverage non-parameterized models ( e.g. , TF-IDF or BM25 ) to retrieve a fixed set of documents , where an answer span is extracted by a neural reading comprehension model . Despite the success of these pipeline methods in singlehop QA , whose questions can be answered based on a single paragraph , they often fail to retrieve the required evidence for answering multi-hop questions , e.g. , the question in Figure 1 . Multi-hop QA ( Yang et al. , 2018 ) usually requires finding more than one evidence document , one of which often consists of little lexical overlap or semantic relationship to the original question . However , retrieving a fixed list of documents independently does not capture relationships between evidence documents through bridge entities that are required for multi-hop reasoning . Recent open-domain QA methods learn end-to-end models to jointly retrieve and read documents ( Seo et al. , 2019 ; Lee et al. , 2019 ) . These methods , however , face challenges for entity-centric questions since compressing the necessary information into an embedding space does not capture lexical information in entities . Cognitive Graph ( Ding et al. , 2019 ) incorporates entity links between documents for multi-hop QA to extend the list of retrieved documents . This method , however , compiles a fixed list of documents independently and expects the reader to find the reasoning paths . In this paper , we introduce a new recurrent graph-based retrieval method that learns to retrieve evidence documents as reasoning paths for answering complex questions . Our method sequentially retrieves each evidence document , given the history of previously retrieved documents to form several reasoning paths in a graph of entities . Our method then leverages an existing reading comprehension model to answer questions by ranking the retrieved reasoning paths . The strong interplay between the retriever model and reader model enables our entire method to answer complex questions by exploring more accurate reasoning paths compared to other methods . ∗Work partially done while the author was a research intern at Salesforce Research . 1Our code and data id available at https : //github.com/AkariAsai/learning_to_ retrieve_reasoning_paths . To be more specific , our method ( sketched in Figure 2 ) constructs the Wikipedia paragraph graph using Wikipedia hyperlinks and document structures to model the relationships between paragraphs . Our retriever trains a recurrent neural network to score reasoning paths in this graph by maximizing the likelihood of selecting a correct evidence paragraph at each step and fine-tuning paragraph BERT encodings . Our reader model is a multi-task learner to score each reasoning path according to its likelihood of containing and extracting the correct answer phrase . We leverage data augmentation and negative example mining for robust training of both models . Our experimental results show that our method achieves the state-of-the-art results on HotpotQA full wiki and HotpotQA distractor settings ( Yang et al. , 2018 ) , outperforming the previous stateof-the-art methods by more than 14 points absolute gain on the full wiki setting . We also evaluate our approach on SQuAD Open ( Chen et al. , 2017 ) and Natural Questions Open ( Lee et al. , 2019 ) without changing any architectural designs , achieving better or comparable to the state of the art , which suggests that our method is robust across different datasets . Additionally , our framework provides interpretable insights into the underlying entity relationships used for multi-hop reasoning . 2 RELATED WORK . Neural open-domain question answering Most current open-domain QA methods use a pipeline approach that includes a retriever and reader . Chen et al . ( 2017 ) incorporate a TF-IDF-based retriever with a state-of-the-art neural reading comprehension model . The subsequent work improves the heuristic retriever by re-ranking retrieved documents ( Wang et al. , 2018a ; b ; Lee et al. , 2018 ; Lin et al. , 2018 ) . The performance of these methods is still bounded by the performance of the initial retrieval process . In multi-hop QA , non-parameterized retrievers face the challenge of retrieving all the relevant documents , one or some of which are lexically distant from the question . Recently , Lee et al . ( 2019 ) and Seo et al . ( 2019 ) introduce fully trainable models that retrieve a few candidates directly from large-scale Wikipedia collections . All these methods find evidence documents independently without the knowledge of previously selected documents or relationships between documents . This would result in failing to conduct multi-hop retrieval . Retrievers guided by entity links Most relevant to our work are recent studies that attempt to use entity links for multi-hop open-domain QA . Cognitive Graph ( Ding et al. , 2019 ) retrieves evidence documents offline , and trains a reading comprehension model to jointly predict possible answer spans and next-hop spans to extend the reasoning chain . Instead , we train our retriever to find reasoning paths directly . Concurrent with our work , Entity-centric IR ( Godbole et al. , 2019 ) uses entity linking for multi-hop retrieval . Unlike our method , this method does not learn to retrieve reasoning paths sequentially , nor study the interplay between retriever and reader . Moreover , while the previous approaches require a system to encode all possible nodes , our beam search decoding process only encodes the nodes on the reasoning paths , which significantly reduces the computational costs . PullNet ( Sun et al. , 2019 ) learns to retrieve question-aware sub-graphs from text corpora and knowledge bases ( e.g. , Freebase ) , while we focus on open-domain QA solely based on text . Multi-step ( iterative ) retrievers Similar to our recurrent retriever , multi-step retrievers explore multiple evidence documents iteratively . Multi-step reasoner ( Das et al. , 2019 ) repeats the retrieval process for a fixed number of steps , interacting with a reading comprehension model by reformulating the query in a latent space to enhance retrieval performance . Feldman & El-Yaniv ( 2019 ) also propose a query reformulation mechanism with a focus on multi-hop open-domain QA . Most recently , Qi et al . ( 2019 ) introduce GoldEn Retriever , which reads and generates search queries for two steps to search documents for HotpotQA full wiki . These methods do not use the graph structure of the documents during the iterative retrieval process . In addition , all of these multi-step retrieval methods do not accommodate arbitrary steps of reasoning and the termination condition is hard-coded . In contrast , our method leverages the Wikipedia graph to retrieve documents that are lexically or semantically distant to questions , and is adaptive to any reasoning path lengths , which leads to significant improvement over the previous work in HotpotQA and SQuAD Open . 3 OPEN-DOMAIN QUESTION ANSWERING OVER WIKIPEDIA GRAPH . Overview This paper introduces a new graph-based recurrent retrieval method ( Section 3.1 ) that learns to find evidence documents as reasoning paths for answering complex questions . We then extend an existing reading comprehension model ( Section 3.2 ) to answer questions given a collection of reasoning paths . Our method uses a strong interplay between retrieving and reading steps such that the retrieval method learns to retrieve a set of reasoning paths to narrow down the search space for our reader model , for robust pipeline process . Figure 2 sketches the overview of our QA model . We use Wikipedia for open-domain QA , where each article is divided into paragraphs , resulting in millions of paragraphs in total . Each paragraph p is considered as our retrieval target . Given a question q , our framework aims at deriving its answer a by retrieving and reading reasoning paths , each of which is represented with a sequence of paragraphs : E = [ pi , . . . , pk ] . We formulate the task by decomposing the objective into the retriever objective Sretr ( q , E ) that selects reasoning paths E relevant to the question , and the reader objective Sread ( q , E , a ) that finds the answer a in E : argmax E , a S ( q , E , a ) s.t . S ( q , E , a ) = Sretr ( q , E ) + Sread ( q , E , a ) . ( 1 ) 3.1 LEARNING TO RETRIEVE REASONING PATHS . Our method learns to retrieve reasoning paths across a graph structure . Evidence paragraphs for a complex question do not necessarily have lexical overlaps with the question , but one of them is likely to be retrieved , and its entity mentions and the question often entail another paragraph ( e.g. , Figure 1 ) . To perform such multi-hop reasoning , we first construct a graph of paragraphs , covering all the Wikipedia paragraphs . Each node of the Wikipedia graph G represents a single paragraph pi . Constructing the Wikipedia graph Hyperlinks are commonly used to construct relationships between articles on the web , usually maintained by article writers , and are thus useful knowledge resources . Wikipedia consists of its internal hyperlinks to connect articles . We use the hyperlinks to construct the direct edges in G. We also consider symmetric within-document links , allowing a paragraph to hop to other paragraphs in the same article . The Wikipedia graph G is densely connected and covers a wide range of topics that provide useful evidence for open-domain questions . This graph is constructed offline and is reused throughout training and inference for any question . 3.1.1 THE GRAPH-BASED RECURRENT RETRIEVER . General formulation with a recurrent retriever We use a Recurrent Neural Network ( RNN ) to model the reasoning paths for the question q . At the t-th time step ( t ≥ 1 ) our model selects a paragraph pi among candidate paragraphs Ct given the current hidden state ht of the RNN . The initial hidden state h1 is independent of any questions or paragraphs , and based on a parameterized vector . We use BERT ’ s [ CLS ] token representation ( Devlin et al. , 2019 ) to independently encode each candidate paragraph pi along with q.2 We then compute the probability P ( pi|ht ) that pi is selected . The RNN selection procedure captures relationships between paragraphs in the reasoning path by conditioning on the selection history . The process is terminated when [ EOE ] , the end-ofevidence symbol , is selected , to allow it to capture reasoning paths with arbitrary length given each question . More specifically , the process of selecting pi at the t-th step is formulated as follows : wi = BERT [ CLS ] ( q , pi ) ∈ Rd , ( 2 ) P ( pi|ht ) = σ ( wi · ht + b ) , ( 3 ) ht+1 = RNN ( ht , wi ) ∈ Rd , ( 4 ) where b ∈ R1 is a bias term . Motivated by Salimans & Kingma ( 2016 ) , we normalize the RNN states to control the scale of logits in Equation ( 3 ) and allow the model to learn multiple reasoning paths . The details of Equation ( 4 ) are described in Appendix A.1 . The next candidate set Ct+1 is constructed to include paragraphs that are linked from the selected paragraph pi in the graph . To allow our model to flexibly retrieve multiple paragraphs within Ct , we also add K-best paragraphs other than pi ( from Ct ) to Ct+1 , based on the probabilities . We typically set K = 1 in this paper . Beam search for candidate paragraphs It is computationally expensive to compute Equation ( 2 ) over millions of the possible paragraphs . Moreover , a fully trainable retriever often performs poorly for entity-centric questions such as SQuAD , since it does not explicitly maintain lexical information ( Lee et al. , 2019 ) . To navigate our retriever in the large-scale graph effectively , we initialize candidate paragraphs with a TF-IDF-based retrieval and guide the search over the Wikipedia graph . In particular , the initial candidate set C1 includes F paragraphs with the highest TF-IDF scores with respect to the question . We expand Ct ( t ≥ 2 ) by appending the [ EOE ] symbol . We additionally use a beam search to explore paths in the directed graph . We define the score of a reasoning path E = [ pi , . . . , pk ] by multiplying the probabilities of selecting the paragraphs : P ( pi|h1 ) . . . P ( pk|h|E| ) . The beam search outputs the top B reasoning paths E = { E1 , . . . , EB } with the highest scores to pass to the reader model i.e. , S ( q , E , a ) = Sread ( q , E , a ) for E ∈ E. In terms of the computational cost , the number of the paragraphs processed by Equation ( 2 ) is bounded by O ( |C1| + B ∑ t≥2 |Ct| ) , where B is the beam size and |Ct| is the average size of Ct over the B hypothesises .

Summary: This paper proposes a method to find a sequence of reasoning paragraphs in Wikipedia to answer queries requiring multi-hop reasoning. They make the key observation that answering multi-hop queries might require retrieving evidence that have very less lexical overlap with the question. Given a query, the proposed method starts from a set of initial paragraphs retrieved by a tf-idf retriever and uses the outgoing Wikipedia anchor link to hop to the next evidence. They propose a simple recurrent neural network that takes in the current paragraph (and the hidden state) and decide which paragraph to hop to in the next step. Because of the available supervision for the paragraphs (in HotpotQA), they can train a supervised path selector. They also add a special EoE token that denotes the end of the reasoning path, thereby having the ability to produce reasoning paths of different lengths. After training the retriever a beam of reasoning paths is sent to the reader module. The reader module re-ranks the reasoning paths again and then use a standard BERTQA model and the top re-ranked chain of paragraphs to find the evidence.