Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention

1 INTRODUCTION . Transformers effectively model natural language in sequential form ( Vaswani et al. , 2017 ; Dai et al. , 2019 ; Devlin et al. , 2019 ; Yang et al. , 2019 ) . Nevertheless , in many NLP tasks , text does not simply appear as a linear sequence of tokens but rather carries meaningful structure in the form of paragraphs , headings , and hyperlinks . These structures can be represented abstractly as trees or graphs with nodes and edges ; and the tasks can be performed as joint reasoning on these more general structures as input . Multi-hop question answering ( Yang et al. , 2018 ) is one such task in which structure plays an important role , since the evidence required to formulate the answer is scattered across multiple documents , requiring systems to jointly reason across links between them . Recent approaches leverage pre-trained Transformers ( e.g. , BERT ) for multi-hop question answering ( QA ) by converting the structural reasoning task into sub-tasks that model flat sequences . For example , Min et al . ( 2019b ) decompose a multi-hop question into a series of single-hop questions ; Ding et al . ( 2019 ) conduct several steps of single-hop reading comprehension to simulate the multihop reasoning . The hope is that additional processing to fuse the outputs of the sub-models can recover all the necessary information from the original structure . While pre-trained Transformer language models have shown improvements on multi-hop QA , manipulating the inherent structure of the problem to fit the rigid requirements of out-of-the-box models can introduce problematic assumptions or information loss . This paper presents Transformer-XH ( meaning eXtra Hop ) , which upgrades Transformers with the ability to natively represent structured texts . Transformer-XH introduces extra hop attention in its layers that connects different text pieces following their inherent structure while also maintaining the powerful pre-trained Transformer abilities over each textual piece individually . Our extra hop attention enables 1 ) a more global representation of the evidence contributed by each piece of text as it relates to the other evidence , and 2 ) a more natural way to jointly reason over an evidence graph by propagating information along edges necessary to complete the task at hand . ∗Work done while interning at Microsoft . We apply Transformer-XH to two multi-evidence reasoning tasks : Hotpot QA , the multi-hop question answering task ( Yang et al. , 2018 ) , and FEVER , the fact verification benchmark whose claims often require multiple pieces of evidence to support ( Thorne et al. , 2018 ) . Rather than decomposing the task into a series of sub-tasks to fit the constraints of pre-trained Transformers , TransformerXH is a solution that fits the problem as it naturally occurs . It is a single model that represents and combines evidence from multiple documents to conduct the reasoning process . On HotpotQA ’ s FullWiki setting , which requires strong multi-hop reasoning capability ( Min et al. , 2019b ; Jiang & Bansal , 2019 ) , Transformer-XH outperforms CogQA ( Ding et al. , 2019 ) , the previous start-of-theart , by 12 points on answer F1 . On FEVER 1.0 shared task , Transformer-XH outperforms GEAR , the Graph Neural Network based approach significantly . On both applications , Transformer-XH beats the contemporary BERT based pipeline SR-MRS ( Nie et al. , 2019 ) , by 2-3 points . The results follow from our simple yet effective design , with one unified model operating over the inherent structure of the task , rather than melding the outputs from disparate sub-tasks adapted to the sequential constraints of pre-trained Transformers . Our ablation studies demonstrate TransformerXH ’ s efficacy on questions that are known to require multi-hop reasoning ( Min et al. , 2019b ) and on verifying multi-evidence claims ( Liu et al. , 2019b ) . Our analyses confirm that the source of Transformer-XH ’ s effectiveness success is due to the eXtra Hop attention ’ s ability to fuse and propagate information across multiple documents.1 2 MODEL . This section first discusses preliminaries on sequential Transformers , then we show how we incorporate eXtra hop attention to create Transformer-XH . 2.1 PRELIMINARIES . Transformers represent a sequence of input text tokens X = { x1 , ... , xi , ... , xn } as contextualized distributed representations H = { h1 , ... , hi , ... , hn } ( Vaswani et al. , 2017 ) . This process involves multiple stacked self-attention layers that converts the input X into { H0 , H1 , ... , H l , ... HL } , starting from H0 , the embeddings , to the final layer of depth L. The key idea of Transformer is its attention mechanism , which calculates the l-th layer output H l using the input H l−1 from the previous layer : H l = softmax ( Q ·KT√ dk ) · V T , ( 1 ) QT ; KT ; V T =W q ·H l−1 ; W k ·H l−1 ; W v ·H l−1 . ( 2 ) It includes three projections on the input H l−1 : Query ( Q ) , Key ( K ) , and Value ( V ) . Specifically , the slices of token hli in Eqn . ( 2 ) is : hli = ∑ j softmaxj ( qTi · kj√ dk ) · vj , ( 3 ) which first calculates its attention to all other tokens j in the sequence and then combines the token values vj into a new representation hli , using the normalized attention weights . Multiple attentions can be used in one Transformer layer and concatenated as multi-head attention ( Vaswani et al. , 2017 ) . The architecture is stacked to form rather deep networks , which leads to significant success of large pre-trained Transformer models ( Devlin et al. , 2019 ; Liu et al. , 2019a ) . A challenge of Transformer is that its attention is calculated over all token pairs ( Eqn . 3 ) , which is hard to scale to long text sequences . Transformer-XL ( eXtra Long ) addresses this challenge by breaking down longer texts , e.g. , a multi-paragraph document , into a sequence of text segments : { X1 , ... , Xτ , ... , Xζ } , and propagates the information between adjacent text segments using the following attention : H̃ l−1τ = [ Freeze ( H l−1 τ−1 ) ◦H l−1τ ] . ( 4 ) 1Code available at https : //aka.ms/transformer-xh . It concatenates ( ◦ ) the representation of the previous segment H l−1τ−1 to the current segment as segment level recurrences . The new representation H̃ l−1τ includes the information from previous segment and is integrated in the new attention mechanism : Q̃T ; K̃T ; Ṽ T =W q ·H l−1τ ; W k · H̃ l−1τ ; W v · H̃ l−1τ . ( 5 ) The attention over the previous segment allows Transformer-XL to effectively model long form text data recurrently as a sequence of text chunks ( Dai et al. , 2019 ) . Nevertheless , in many scenarios , the text segments are organized in nontrivial structures beyond a linear sequence . For example , documents are connected by hyperlinks in a graphical structure that does not readily simplify to form a linear sequence , prohibiting Transformer-XL ’ s recurrent approach . 2.2 TRANSFORMER-XH WITH EXTRA HOP ATTENTION . Transformer-XH models structured text sequence by linking them with eXtra Hop attention following their original structure . As illustrated in Figure 1a , to model three connected documents d2 → d1 → d3 , Transformer-XH uses eXtra Hop attention to propagate information along the graph edges , enabling information sharing between connected text sequence . Formally , the structured text data includes a set of nodes , X = { X1 , ... , Xτ , ... Xζ } , each corresponding to a text sequence , and an edge matrix E , which includes the connections ( e.g. , links ) between them . The goal is to learn representations H = { H̃1 , ... , H̃τ , ... H̃ζ } , that incorporate not only the local information in each sequence X , but also the global contexts on the entire structured text { X , E } . Transformer-XH achieves this by two attention mechanisms : in-sequence attention and eXtra Hop attention . The in-sequence attention is the same as vanilla Transformer : in layer l , token i gathers information from other tokens inside the same text piece τ : hlτ , i = ∑ j softmaxj ( qTτ , i · kτ , j√ dk ) · vτ , j . ( 6 ) The eXtra Hop attention uses the first token in each sequence – the added special token “ [ CLS ] ” – as an “ attention hub ” , which attends on all other connected nodes ’ hub token . In layer l , the τ -th text sequence attends over another text sequence η if there is an edge between them ( eτη = 1 ) : ĥlτ,0 = ∑ η ; eτη=1 softmaxη ( q̂Tτ,0 · k̂η,0√ dk ) · v̂η,0 . ( 7 ) Node τ calculates the attention weight on its neighbor η using hop query q̂τ,0 and key k̂η,0 . Then it uses the weights to combine its neighbors ’ value v̂η,0 and forms a globalized representation ĥlτ,0 . The two attention mechanism are combined to form the new representation of layer l : h̃lτ,0 = Linear ( [ h l τ,0 ◦ ĥlτ,0 ] ) , ( 8 ) h̃lτ , i = h l τ , i ; ∀i 6= 0 . ( 9 ) Note that the non-hub tokens ( i 6= 0 ) still have access to the hop attention in the previous layer through Eqn . ( 6 ) . One layer of eXtra Hop attention can be viewed as single-step of information propagation along edges E. For example , in Figure 1a , the document node d3 updates its representation by gathering information from its neighbor d1 using the hop attention d1 → d3 . When multiple TransformerXH layers are stacked , this information in d1 includes both d1 ’ s local contexts from its in-sequence attention , and cross-sequence information from the hop attention d2 → d1 of the l− 1 layer . Hence , an L-layer Transformer-XH can attend over information from up to L hops away . Together , three main properties equip Transformer-XH to effectively model raw structured text data : the propagation of information ( values ) along edges , the importance of that information ( hop attention weights ) , and the balance of in-sequence and cross-sequence information ( attention combination ) . The representations learned in H can innately express nuances in structured text that are required for complex reasoning tasks such as multi-hop QA and natural language inference . 3 APPLICATION TO MULTI-HOP QUESTION ANSWERING . This section describes how Transformer-XH applies to multi-hop QA . Given a question q , the task is to find an answer span a in a large open-domain document corpus , e.g . the first paragraph of all Wikipedia pages . By design , the questions are complex and often require information from multiple documents to answer . For example , in the case shown in Figure 1b , the correct answer “ Cambridge ” requires combining the information from both the Wikipedia pages “ Facebook ” and “ Harvard University ” . To apply Transformer-XH in the open domain multi-hop QA task , we first construct an evidence graph and then apply Transformer-XH on the graph to find the answer . Evidence Graph Construction . The first step is to find the relevant candidate documents D for the question q and connect them with edges E to form the graph G. Our set D consists of three sources . The first two sources are from canonical information retrieval and entity linking techniques : Dir : the top 100 documents retrieved by DrQA ’ s TF-IDF on the question ( Chen et al. , 2017 ) . Del : the Wikipedia documents associated with the entities that appear in the question , annotated by entity linking systems : TagMe ( Ferragina & Scaiella , 2010 ) and CMNS ( Hasibi et al. , 2017 ) . For better retrieval quality , we use a BERT ranker ( Nogueira & Cho , 2019 ) on the set Dir ∪ Del and keep the top two ranked ones in Dir and top one per question entity in Del . Then the third source Dexp includes all documents connected to or from any top ranked documents via Wikipedia hyperlinks ( e.g. , “ Facebook ” → “ Harvard University ” ) . The final graph comprises all documents from the three sources as nodes X . The edge matrix E is flexible . We experiment with various edge matrix settings , including directed edges along Wikipedia links , i.e . eij = 1 if there is a hyperlink from document i to j , bidirectional edges along Wiki links , and fully-connected graphs , which rely on Transformer-XH to learns the edge importance . Similar to previous work ( Ding et al. , 2019 ) , the textual representation for each node in the graph is the [ SEP ] -delimited concatenation of the question , anchor text ( the text in the hyperlink in parent nodes pointing to the child node ) , and the paragraph itself . More details on the evidence graph construction are in Appendix A.1 . Transformer-XH on Evidence Graph . Transformer-XH takes the input nodes X and edges E , and produces the global representation of all text sequences : HL = Transformer-XH ( X , E ) . ( 10 ) Then we add two task-specific layers upon the last layer ’ s representationHL : one auxiliary layer to predict the relevance score of the evidence node , and one layer to extract the answer span within it : p ( relevance|τ ) = softmax ( Linear ( h̃Lτ,0 ) ) ; ( 11 ) p ( start|τ , i ) , p ( end|τ , j ) = softmax ( Linear ( h̃Lτ , i ) ) , softmax ( Linear ( h̃Lτ , j ) ) . ( 12 ) The final model is trained end-to-end with cross-entropy loss for both tasks in a multi-task setting . During inference , we first select the document with the highest relevance score , and then the start and end positions of the answer within that document .

Summary: This paper introduces a way to train transformer models over document graphs, where each node is a document and edges connect related documents. It is inspired by the transformer-XL model, as well as Graph Neural networks. They apply this model to answer multi-hop questions on the HotPotQA dataset and outperform the previous SOTA. The model particularly improves performance on the bridge style questions of HotPotQA. They are able to do this in a single step, rather than a multi-stage process as done by previous approaches.