Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering

1 INTRODUCTION . A requirement of scalable and practical question answering ( QA ) systems is the ability to reason over multiple documents and combine their information to answer questions . Although existing datasets enabled the development of effective end-to-end neural question answering systems , they tend to focus on reasoning over localized sections of a single document ( Hermann et al. , 2015 ; Rajpurkar et al. , 2016 ; 2018 ; Trischler et al. , 2017 ) . For example , Min et al . ( 2018 ) find that 90 % of the questions in the Stanford Question Answering Dataset are answerable given 1 sentence in a document . In this work , we instead focus on multi-evidence QA , in which answering the question requires aggregating evidence from multiple documents ( Welbl et al. , 2018 ; Joshi et al. , 2017 ) . Our multi-evidence QA model , the Coarse-grain Fine-grain Coattention Network ( CFC ) , selects among a set of candidate answers given a set of support documents and a query . The CFC is inspired by coarse-grain reasoning and fine-grain reasoning . In coarse-grain reasoning , the model builds a coarse summary of support documents conditioned on the query without knowing what candidates are available , then scores each candidate . In fine-grain reasoning , the model matches specific finegrain contexts in which the candidate is mentioned with the query in order to gauge the relevance of the candidate . These two strategies of reasoning are respectively modeled by the coarse-grain and fine-grain modules of the CFC . Each module employs a novel hierarchical attention — a hierarchy of coattention and self-attention — to combine information from the support documents conditioned on the query and candidates . Figure 1 illustrates the architecture of the CFC . The CFC achieves a new state-of-the-art result on the blind Qangaroo WikiHop test set of 70.6 % accuracy , beating previous best by 3 % accuracy despite not using pretrained contextual encoders . In addition , on the TriviaQA multi-paragraph question answering task ( Joshi et al. , 2017 ) , reranking Most of this work was done while Victor Zhong was at Salesforce Research . outputs from a traditional span extraction model ( Clark & Gardner , 2018 ) using the CFC improves exact match accuracy by 3.1 % and F1 by 3.0 % . Our analysis shows that components in the attention hierarchies of the coarse and fine-grain modules learn to focus on distinct parts of the input . This enables the CFC to more effectively represent a large collection of long documents . Finally , we outline common types of errors produced by CFC , caused by difficulty in aggregating large quantity of references , noise in distant supervision , and difficult relation types . 2 COARSE-GRAIN FINE-GRAIN COATTENTION NETWORK . The coarse-grain module and fine-grain module of the CFC correspond to coarse-grain reasoning and fine-grain reasoning strategies . The coarse-grain module summarizes support documents without knowing the candidates : it builds codependent representations of support documents and the query using coattention , then produces a coarse-grain summary using self-attention . In contrast , the fine-grain module retrieves specific contexts in which each candidate occurs : it identifies coreferent mentions of the candidate , then uses coattention to build codependent representations between these mentions and the query . While low-level encodings of the inputs are shared between modules , we show that this division of labour allows the attention hierarchies in each module to focus on different parts of the input . This enables the model to more effectively represent a large number of potentially long support documents . Suppose we are given a query , a set of Ns support documents , and a set of Nc candidates . Without loss of generality , let us consider the ith document and the jth candidate . Let Lq ∈ RTq×demb , Ls ∈ RTs×demb , and Lc ∈ RTc×demb respectively denote the word embeddings of the query , the ith support document , and the jth candidate answer . Here , Tq , Ts , and Tc are the number of words in the corresponding sequence . demb is the size of the word embedding . We begin by encoding each sequence using a bidirectional Gated Recurrent Units ( GRUs ) ( Cho et al. , 2014 ) . Eq = BiGRU ( tanh ( WqLq + bq ) ) ∈ RTq×dhid ( 1 ) Es = BiGRU ( Ls ) ∈ RTs×dhid ( 2 ) Ec = BiGRU ( Lc ) ∈ RTc×dhid ( 3 ) Here , Eq , Es , and Ec are the encodings of the query , support , and candidate . Wq and bq are parameters of a query projection layer . dhid is the size of the bidirectional GRU . 2.1 COARSE-GRAIN MODULE . The coarse-grain module of the CFC , shown in Figure 2 , builds codependent representations of support documents Es and the query Eq using coattention , and then summarizes the coattention context using self-attention to compare it to the candidate Ec . Coattention and similar techniques are crucial to single-document question answering models ( Xiong et al. , 2017 ; Wang & Jiang , 2017 ; Seo et al. , 2017 ) . We start by computing the affinity matrix between the document and the query as A = Es ( Eq ) ᵀ ∈ RTs×Tq ( 4 ) The support summary vectors and query summary vectors are defined as Ss = softmax ( A ) Eq ∈ RTs×dhid ( 5 ) Sq = softmax ( A ᵀ ) Es ∈ RTq×dhid ( 6 ) where softmax ( X ) normalizes X column-wise . We obtain the document context as Cs = BiGRU ( Sq softmax ( A ) ) ∈ RTs×dhid ( 7 ) The coattention context is then the feature-wise concatenation of the document context Cs and the document summary vector Ss . Us = [ Cs ; Ss ] ∈ RTs×2dhid ( 8 ) For ease of exposition , we abbreviate coattention , which takes as input a document encoding Es and a query encoding Eq and produces the coattention context Us , as Coattn ( Es , Eq ) → Us ( 9 ) Next , we summarize the coattention context — a codependent encoding of the supporting document and the query — using hierarchical self-attention . First , we use self-attention to create a fixedlength summary vector of the coattention context . We compute a score for each position of the coattention context using a two-layer multi-layer perceptron ( MLP ) . This score is normalized and used to compute a weighted sum over the coattention context . asi = tanh ( W2 tanh ( W1Usi + b1 ) + b2 ) ∈ R ( 10 ) âs = softmax ( as ) ( 11 ) Gs = Ts∑ i âsiUsi ∈ R2dhid ( 12 ) Here , asi and âsi are respectively the unnormalized and normalized score for the ith position of the coattention context . W2 , b2 , W1 , and b1 are parameters for the MLP scorer . Usi is the ith position of the coattention context . We abbreviate self-attention , which takes as input a sequence Us and produces the summary conditioned on the query Gs , as Selfattn ( Us ) → Gs ( 13 ) Recall that Gs provides the summary of the ith of Ns support documents . We apply another selfattention layer to compute a fixed-length summary vector of all support documents . This summary is then multiplied with the summary of the candidate answer to produce the coarse-grain score . Let G ∈ RNs×2dhid represent the sequence of summaries for all support documents . We have Gc = Selfattn ( Ec ) ∈ Rdhid ( 14 ) G′ = Selfattn ( G ) ∈ R2dhid ( 15 ) ycoarse = tanh ( WcoarseG ′ + bcoarse ) Gc ∈ R ( 16 ) where Ec and Gc are respectively the encoding and the self-attention summary of the candidate . G′ is the fixed-length summary vector of all support documents . Wcoarse and bcoarse are parameters of a projection layer that reduces the support documents summary from R2dhid to Rdhid . 2.2 CANDIDATE-DEPENDENT FINE-GRAIN MODULE . In contrast to the coarse-grain module , the fine-grain module , shown in Figure 3 , finds the specific context in which the candidate occurs in the supporting documents using coreference resolution 1 . Each mention is then summarized using a self-attention layer to form a mention representation . We then compute the coattention between the mention representations and the query . This coattention context , which is a codependent encoding of the mentions and the query , is again summarized via self-attention to produce a fine-grain summary to score the candidate . Let us assume that there are m mentions of the candidate in the ith support document . Let the kth mention corresponds to the istart to iend tokens in the support document . We represent this mention using self-attention over the span of the support document encoding that corresponds to the mention . 1We use simple lexical matching between candidates and support documents . Details are found in A.1 of the Appendix . Mk = Selfattn ( Es [ istart : iend ] ) ∈ Rdhid ( 17 ) Suppose that there are Nm mentions of the candidate in total . We extract each mention representation using self-attention to produce a sequence of mention representations M ∈ RNm×dhid . The coattention context and summary of these mentions M with respect to the query Eq are Um = Coattn ( M , Eq ) ∈ RNm×2dhid ( 18 ) Gm = Selfattn ( Um ) ∈ R2dhid ( 19 ) We use a linear layer to determine the fine-grain score of the candidate yfine = WfineGm + bfine ∈ R ( 20 ) 2.3 SCORE AGGREGATION . We take the sum of the coarse-grain score and the fine-grain score , y = ycoarse + yfine , as the score for the candidate . Recall that our earlier presentation is with respect to the jth out of Nc candidates . We combine each candidate score to form the final score vector Y ∈ RNc . The model is trained using cross-entropy loss . 3 EXPERIMENTS . We evaluate the CFC on two tasks to evaluate its effectiveness . The first task is multi-evidence question answering on the unmasked and masked version of the WikiHop dataset ( Welbl et al. , 2018 ) . The second task is the multi-paragraph extractive question answering task TriviaQA , which we frame as a span reranking task ( Joshi et al. , 2017 ) . On the former , the CFC achieves a new stateof-the-art result . On the latter , reranking the outputs of a span-extraction model ( Clark & Gardner , 2018 ) using the CFC results in significant performance improvement . 3.1 MULTI-EVIDENCE QUESTION ANSWERING ON WIKIHOP . Welbl et al . ( 2018 ) proposed the Qangaroo WikiHop task to facilitate the study of multi-evidence question answering . This dataset is constructed by linking entities in a document corpus ( Wikipedia ) with a knowledge base ( Wikidata ) . This produces a bipartite graph of documents and entities , an edge in which marks the occurrence of an entity in a document . A knowledge base fact triplet consequently corresponds to a path from the subject to the object in the resulting graph . The documents along this path compose the support documents for the fact triplet . The Qangaroo WikiHop task , shown in Figure 4 , is as follows : given a query , that is , the subject and relation of a fact triplet , a set of plausible candidate objects , and the corresponding support documents for the candidates , select the correct candidate as the answer . The unmasked version of WikiHop represents candidate answers with original text while the masked version replaces them with randomly sampled placeholders in order to remove correlation between frequent answers and support documents . Official blind , held-out test evaluation is performed using the unmasked version . We tokenize the data using Stanford CoreNLP ( Manning et al. , 2014 ) . We use fixed GloVe embeddings ( Pennington et al. , 2014 ) as well as character ngram embeddings ( Hashimoto et al. , 2017 ) . We split symbolic query relations into words . All models are trained using ADAM ( Kingma & Ba , 2015 ) . We list detailed experiment setup and hyperparemeters of the best-performing model in A.2 of the Appendix . We compare the performance of the CFC to other models on the WikiHop leaderboard in Table 1 . The CFC achieves state-of-the-art results on both the masked and unmasked versions of WikiHop . In particular , on the blind , held-out WikiHop test set , the CFC achieves a new best accuracy of 70.6 % . The previous state-of-the-art result by Cao et al . ( 2018 ) uses pretrained contextual encoders , which has led to consistent improvements across NLP tasks ( Peters et al. , 2018 ) . We outperform this result by 3 % despite not using pretrained contextual encoders 2 . In addition , we show that the division of labour between the coarse-grain module and the fine-grain module allows the attention hierarchies of each module to focus on different parts of the input . This enables the CFC to more effectively model the large collection of potentially long documents found in WikiHop .

Summary: This paper proposes a method for multi-hop QA based on two separate modules, which are called coarse-grained and fine-grained modules. The coarse-grained module reads all of the supporting documents for QA, whereas the fine-grained one reads the local context surrounding each candidate entity's mentions. Both modules are used to predict the score of a candidate entity being the answer, with the final result being the sum of the two scores.