Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention
Authors: Chen Zhao
Chenyan Xiong
Corby Rosset
Xia Song
Paul Bennett
Saurabh Tiwary
Venue:  (2020)
Keywords: Transformer-XH
multi-hop QA
fact verification
extra hop attention
structured modeling

Abstract: Transformers have achieved new heights modeling natural language as a sequence of text tokens. However, in many real world scenarios, textual data inherently exhibits structures beyond a linear sequence such as trees and graphs; many tasks require reasoning with evidence scattered across multiple pieces of texts. This paper presents Transformer-XH, which uses eXtra Hop attention to enable intrinsic modeling of structured texts in a fully data-driven way. Its new attention mechanism naturally “hops” across the connected text sequences in addition to attending over tokens within each sequence. Thus, Transformer-XH better conducts joint multi-evidence reasoning by propagating information between documents and constructing global contextualized representations. On multi-hop question answering, Transformer-XH leads to a simpler multi-hop QA system which outperforms previous state-of-the-art on the HotpotQA FullWiki setting. On FEVER fact verification, applying Transformer-XH provides state-of-the-art accuracy and excels on claims whose verification requires multiple evidence.

Published as a conference paper at ICLR 2020
TRANSFORMER-XH: MULTI-EVIDENCE REASONING WITH EXTRA HOP ATTENTION
Chen Zhao" University of Maryland, College Park chenz edu
Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary Microsoft AI & Research cxiong, corosset, xiaso, pauben, satiwary @ microsoft.com
ABSTRACT
Transformers have achieved new heights modeling natural language as sequence of text tokens. However, many real world scenarios, textual data inherently ex- hibits structures beyond a linear sequence such as trees and graphs; many tasks require reasoning with evidence scattered across multiple pieces of texts. This pa- per presents Transformer-Xie XHH, which uses eXtra Hop attention to enable intrinsic modeling of structured texts in fully data-driven way. Its new attention mecha- nism naturally "hops" across the connected text sequences in addition to attend- better conducts
ing over tokens within each sequence. Thus, Transformer-X joint multi-evidence reasoning by propagating information between documents and constructing global contextualized representations. On multi-hop question answering, Transformer-XH leads simpler multi-hop QA system which out- performs previous state-of-the-ar on the HotpotQA FullWiki setting. On FEVER fact verification, applying Transformer-Xe provides state-of-the-an accuracy and excels on claims whose verification requires multiple evidence.
INTRODUCTION
Transformers effectively model natural language in sequential form (Vaswani et al. 2017 Dai et al., 2019 Devlin et al.) [019 Yang al. 2019). Nevertheless, in many NLP tasks, text does not simply appear as a linear sequence of tokens but rather carries meaningful structure in the form of para- graphs, headings, and hyperlinks. These structures be represented abstractly as trees graphs with nodes and edges; and the tasks can be performed as joint reasoning on these more general structures as input. Multi-hop question answering (Yang et al. 2018) is one such task in which structure plays an important role, since the evidence required to formulate the answer is scattered across multiple documents, requiring systems to jointly reason across links between them.
Recent approaches leverage pre-trained Transformers (e.g., BERT) for multi-hop question answer- ing (QA) by converting the structural reasoning task into sub-tasks that model flat sequences. For example, Min et al.) (2019b) decompose multi-hop question into series of single-hop questions; Ding et al. (2019) conduct several steps of single-hop reading comprehension to simulate the multi- hop reasoning. The hope is that additional processing to fuse the outputs of the sub-models can recover all the necessary information from the original structure. While pre-trained Transformer language models have shown improvements on multi-hop QA, manipulating the inherent structure of the problem to fit the rigid requirements of out-of-the-box models can introduce problematic assumptions information loss.
This paper presents Transformer-X XH (meaning eXtra Hop), which upgrades Transformers with the ability to natively represent structured texts. Transformer-XH introduces extra hop attention in its layers that connects different text pieces following their inherent structure while also maintaining the powerful pre-trained Transformer abilities over each textual piece individually. Our extra hop attention enables 1) a more global representation of the evidence contributed by each piece of text as it relates to the other evidence, and 2) a more natural way to jointly reason over an evidence graph by propagating information along edges necessary to complete the task at hand.
Work done while interning Microsoft.

---

Published as a conference paper at ICLR 2020
We apply Transformer-XH to two multi-evidence reasoning tasks: Hotpot QA, the multi-ho hopp ques- tion answering task (Yang et al. 2018), and FEVER, the fact verification benchmark whose claims often require multiple pieces of evidence to support (Thore al. 2018). Rather than decomposing the task into a series of sub-tasks to fit the constraints of pre-trained Transformers, Transformer- XH is solution that fits the problem as it naturally occurs. It is single model that represents and combines evidence from multiple documents to conduct the reasoning process. On HotporQA's FullWiki setting, which requires strong multi-hop reasoning capability (Min al. 2019b Jiang & Bansal 2019). Transformer- XH outperforms CogQA (Ding et al., 2019), the previous start-of-the- art, by points on answer FI. On FEVER 1.0 shared task, Transformer- XH outperforms GEAR, the Graph Neural Network based approach significantly. On both applications, Transformer-Xe beats the contemporary BERT based pipeline SR-MRS (Nie et al. 2019), by 2-3 points.
The results follow from our simple yet effective design, with one unified model operating over the inherent structure of the task, rather than melding the outputs from disparate sub-tasks adapted the sequential constraints of pre-trained Transformers. Our ablation studies demonstrate Transformer- XH's efficacy on questions that are known to require multi-hop reasoning (Min et al. 2019b) and on verifying multi-evident
claims (Liu et al.) 2019b). Our analyses confirm that the source of Transformer- XH's 's effectiveness success is due to the eXtra Hop attention 's ability to fuse and prop- agate information across multiple documents
2
MODEL
This section first discusses preliminaries sequential Transformers, then we show how we incor- porate eXtra hop attention to create Transformer-Xi
2.1 PRELIMINARIES
Transformers represent sequence of input text tokens X {21, Biy..., Fn contextualized (Vaswani et al. 2017). This process involves
distributed representations H = {/,...., ha....,hy,, multiple stacked self-attention layers that converts the input X into H H H. start-
ing from H", the embeddings, to the final layer of depth L.
The key idea of Transformer is its attention mechanism, which calculates the 1-th layer output HI using the input Hi-1 from the previous layer:
H' softmax Q KT ) VT de
(1)
QT; = W H² W H 11 1; V H²-1
(2)
It includes three projections on the input -² Query (Q), Key (K), and Value (V). Specifically, the slices of token hy in Eq).(2) is:
of kj ddz
A = Σ softmax,
Uiz
which first calculates its attention to all other tokens j in the sequence and then combines the token values By into a new representation 1/ using the normalized attention weights. Multiple attentions can be used in one Transformer layer and concatenated 2017). The architecture is stacked to form rather deep networks, which leads to significant success of large pre-trained Transformer models (Devlin al. 2019 Liu et al. 2019a).
multi-head attention (Vaswani et al.)
A challenge of Transformer is that its attention is calculated over all token pairs (Eqn. 3 which hard scale long text sequences. Transformer-XL (eXtra Long) addresses this challenge by breaking down longer texts, e.g., a multi-paragraph document, into a sequence of text segments: {X,,..., X,. .x, and propagates the information between adjacent text segments using the fol- lowing attention:
A [reeze(H](])) H
(4)
Coode available httttttt//aaaee ms//rranfformer-xtt
2

---

Published as a conference paper at ICLR 2020
Input Question:
Answer:
city was Facebook launched?
University private League research Massadi
Harvard
University
Node
WIKIEELI
Entity Linking
Transformer-0tt
d3
#
Facebook
Harvard Universiiyy
Retrieval
Facebook
siveratty a private boy League...
dg d2
Hanuans
Mark Zuckerberg
Social Media
d2 x8
xi
.ucckrbbrrg
Wilki Hyperlink
Social
attended Hansard...
Initial Documents
Connected Documents
d2
(a) Hop attention on the path d2 - d1 + dg.
(b) Transformer-Xee in Multi-hop QA
Figure 1: The eXtra Hop attention in Transformer-Xi XH (a) and its application to multi-hop QA (b).
concatenates (o) the representation of the previous segment H/-1 to the current segment as seg- ment level recurrences. The new representation f7;-1 includes the information from previous seg- ment and is integrated in the new attention mechanism:
QT kT, VT = R - W W A
(5)
The attention over the previous segment allows Transformer- XL to effectively model long form text data recurrently as sequence of text chunks (Dai et al.. 2019).
Nevertheless, in many scenarios, the text segments are organized in nontrivial structures beyond linear sequence. For example, documents are connected by hyperlinks in graphical structure that does not readily simplify to form linear sequence, prohibiting Transformer-XI. XL.'s recurrent approach.
2.2 TRANSFORMER-XIE WITH EXTRA HOP ATTENTION
Transformer-Xit models structured text sequence by linking them with eXtra Hop attention fol- lowing their original structure. As illustrated in Figure Ta to model three connected documents d2 + dj + d3. Transformer-XH uses eXtra Hop attention to propagate information along the graph edges, enabling information sharing between connected sequence.
Formally, the structured text data includes a set of nodes, X' sponding a text sequence, and an edge matrix E, which includes the connections (e.g.. links) between them. The goal learn representations H {H,, H..... ). that incorporate not only the local information in each sequence X, but also the global contexts on the entire structured text (X, E}.
..ch. each corre-
Transformer-Xe XH achieves this by two attention mechanisms: in-sequence attention and eXtra Hop attention. The in- in-sequence attention the same as vanilla Transformer: layer 1, token gathers information from other tokens inside the same text piece T:
T.. krd UrJ. Vdz
h Σ softmaxy(
(6)
The eXira Hop attention uses first token each sequence - the added special token "(CLS]" as "attention hub", which attends all other connected nodes' hub token. In layer I, the 7-th text sequence attends over another text sequence if there is an edge between them (eyy = 1):
Gf kn,0, ). 0n
h
Σ =1
softmax,
(7)
ddz
3

---

Published as a conference paper at ICLR 2020
Node 7 calculates the attention weight its neighbor 7, using hop query âr,0 and key knD Then it uses the weights to combine its neighbors' value 0y and forms a globalized representation N The two attention mechanism are combined to form the new representation of layer I:
ke,0 Linear([k o h=,
(8)
h 11',5 /
(9)
Note that the non-hub tokens (i # 0) still have access to the hop attention in the previous layer through Eqn. (6)
One layer of eXtra Hop attention can be viewed as single-step of information propagation along edges E. For example, in Figure Ta the document node d3 updates its representation by gathering information from its neighbor di using the hop attention di + da. When multiple Transformer- XH layers are stacked, this information in di includes both di 's local contexts from its in-sequence attention, and sequence information from the hop attention d2 - d1 of the / - layer. Hence, an L-layer Transformer-2 XH can attend over information from up to L hops away.
Together, three main properties equip Transformer-X the propagation of information (values) along edges, the importance of that information (hop at- tention weights), and the balance of in-sequence and cross-sequence information (attention combi- nation). The representations learned in H can innately express nuances in structured text that are required for complex reasoning tasks such as multi-hop QA and natural language inference.
effectively model raw structured text data:
APPLICATION TO MULTI-HOP QUESTION ANSWERING
This section describes how Transformer-Xir applies to multi-hop QA. Given question q, the task is to find an answer span in a large open-domain document corpus, e.g. the first paragraph of all Wikipedia pages. By design, the questions are complex and often require information from multiple documents to answer. For example, in the "Cambridge" requires combining the information from both the Wikipedia pages "Facebook" and "Harvard University". To apply Transformer-X in the open domain multi-hop QA task, first construct an evidence graph and then apply Transformer- XH on the graph to find the answer.
shown in Figure Ib] the correct answer
Evidence Graph Construction. The first step is find the relevant candidate documents D for the question and connect them with edges E to form the graph G. Our set D consists of three sources. The first two sources are from canonical information retrieval and entity linking techniques:
Dir: the top 100 documents retrieved by DrQA's TF-IDF on the question CChn et al. 2017).
Da: the Wikipedia documents associated with the entities that appear in the question, annotated by entity linking systems: TagMe (Ferragina & Scaiella 2010) and CMNS (Hasibi et al. (017).
For better retrieval quality, we use BERT ranker (Nogueira & Cho 2019) on the set D Da and keep the top two ranked ones in Dir and top one per question entity in Det- Then the third source D Deep includes all documents connected to or from any top ranked documents via Wikipedia hyperlinks (e.g., "Facebook" + "Harvard University").
The final graph comprises all documents from the three sources as nodes X. The edge matrix E is flexible. We experiment with various edge matrix settings, including directed edges along Wikipedia links, i.e. eij = if there is hyperlink from document to j. bidirectional edges along Wiki links, and fully-connected graphs, which rely on Transformer-HH to learns the edge importance.
Similar to previous work (Ding et al., 2019), the textual representation for each node in the graph is the [SEP]-delimited concatenation of the question, anchor text (the text in the hyperlink in parent nodes pointing to the child node), and the paragraph itself. More details on the evidence graph construction are in Appendix A.I
Transformer-Xe on Evidence Graph. Transformer-Xir takes the input nodes X' and edges E, and produces the global representation of all text sequences:
2 Transformer-XHH/ F.).
(10)
4

---

Published as a conference paper at ICLR 2020
Then add two task-specific layers upon the last layer's representation H²: one auxiliary layer to predict the relevance score of the evidence node, and one layer to extract the answer span within it:
p(relevance r) softmax (Linear(h+,,))
(11)
p(start|T, i). p(end|T..) softmax softmax(Linearr(25) (Li,)ar(h+,))), softmax(Linear( (h
(12)
The final model is trained end-to-end with cross-entropy loss for both tasks in multi-task setting. During inference, we first select the document with the highest relevance score, and then the start and end positions of the answer within that document.
APPLICATION TO FACT VERIFICATION
This section describes how Transformer-Xe XH applies to the fact verification task in FEVER Thorne et al. (2018). Given a claim and a trustworthy background corpus, i.e. Wikipedia, the task is to verify whether the evidence in the corpus SUPPORTS, REFUTES, there is NOT ENOUGH INFO to verify the claim. Similar to multi-hop QA, the first step is to construct an evidence graph using the text pieces in the background corpus and then Transformer-Xet can be easily applied conduct reasoning on these evidence pieces.
Evidence Graph Construction. Many previous FEVER systems first retrieve the evidence sen- tences for the claim and then reason verify it (Nic et al. 2019) Zhou et al. 2019). This first step is similar the retrieval stage in Hotpot QA. And the second step multi-evidence reasoning task, where Transformer-XH is applied.
We keep the evidence sentence retrieval step consistent with previous methods. The sentence re- trieval results of SR-MRS is not released at the time of our experiments, thus we instead use the BERT-based retrieval results from another contemporary work (Liu et al. 2019b).
We construct the evidence graph using the top five sentences from Liu et al. (2019b) the nodes X' and fully connected edges E. Following Liu al. (2019b), the representation of each node the concatenation of the claim, the Wikipedia title (entity name) of the document that includes the sentence, and the evidence sentence.
Transformer-XH on Evidence Graph. Transformer-XH takes the evidence graph {X, E} and learns to verify the claim to three categories: y E (SUPPORT, REFUSE, NOT ENOUGH EVI- DENCE). Similar to the application in Hotpot QA, it first produces the global representation of the graph:
H (aansfoomer-H(((,,,))
(13)
Then two task-specific layers are added upon the last layer. The first layer conducts the fact predic- tion per node using the "[CLS]" token:
p softmax(Linear
(14)
The second layer learns to measure the importance of each node in the graph:
p(s/T) softmax (Laiimaa((innna(((__)))))
(15)
The node level predictions and node importance are combined to the final prediction for the claim:
Plu E) p((TT) p(ylT).
(16)
Similar the Hotpot QA scenario, we use multi-task learning that combines the node prediction task and the claim verification task. The first task uses the evidence sentence label provided by FEVER and cross-entropy loss Eqn. [5 The second task uses the final verification label with cross-entropy loss on Eqn. 16
5 EXPERIMENTAL METHODOLOGIES
Our experiments are conducted on Hotpot QA. the multi-hop question answering benchmark Yang al. (2018), and FEVER, the fact verfication benchmark Thome et al. (2018).
5

---

Published as a conference paper at ICLR 2020
5.1 MULTI-HOP QUESTION ANSWERING ON HOTPOT QA
Dataset. HotpotQA includes 112k crowd-sourced questions designed to require multiple pieces of textual evidence, which the first paragraphs of Wikipedia pages. It has type of questions: bridge question require hopping via an outside entity, and comparison question compare a property of two entities. There are two settings in HotpotQA. The Distractor setting provides golden evidence paragraphs together with TF-IDF retrieved negatives. The FullWiki setting requires systems to retrieve evidence paragraphs from the full set of Wikipedia articles.
We focus on FullWiki setting since previous research found that the negative documents in Distractor may be too weak and mitigate the need for multi-hop reasoning (Min et al. 2019b). There are 90k Train, 7k Dev and 7k Test questions. The ground truth answer and supporting evidence sentences in Train and Dev sets are provided. Test labels are hidden; only one submission is allowed to the leaderboard per 30 days We evaluate our final model on Test and conduct ablations Dev.
Metrics. We use official evaluation metrics of HotpotQA: exact match (EM) and Fl on answer (Ans), supporting facts (Supp), and the combination (Joint). The supporting facts prediction is an auxiliary task that evaluates model's ability to find the evidence sentences. Joint EM is the product of the two EM result. Joint FI first multiplies the precision and recall from Ans and Supp, then combines the Joint precision and recall to F1.
Baseline. The main baselines include Cognitive QA (CogQA, Ding al. (2099) and Semantic Retrieval MRS (SR-MRS, Nie comprehension (MRC) models to find hop entities and candidate spans, and then uses a BERT based Graph Convolution Network to rank the candidate spans. SR-MRS is contemporary work and was the previous leaderboard rank one. It a BERT based pipeline and uses fine-tuned BERT models to first rank the documents (twice), then to rank sentences to find supporting facts, and finally conducts BERT MRC on the concatenated evidence sentences.
Nie et al. (019)). CogQA uses several fine-tuned BERT machine reading
We also re-implement CogQA and upgrade its IR with BERT IR model (BERT Dir U Det, same as Transformer-XH) for fair comparisons. We include other approaches on the FullWiki setting: Official Baseline (Yang et al. 20188) MUPPET (Feldman & El-Yaniv 2019). QFE (Nishida et al. 2019). and DecompRC (Min et al. 2019a),
Implementation Details. The in-sequence attention and other standard Transformer components in Transformer-HH are initialized by the pre-trained BERT base model (Devlin et al., 2019). The extra hop attention parameters are initialized randomly and trained from scratch. The final model uses three hop steps. For bridge questions, And for comparison questions, we build the fully-connected graph on the set Drr U D and train Transformer-X XH separately. We leave more implementation details in the Appendix.
build the evidence graph described in Section 3
5.2 FACT VERIFICATION ON FEVER
Dataset. The FEVER task provides claim sentence and requires the system to classify it into three categories: SUPPORTS, REFUTES, and NOT ENOUGH INFO, using the Wikipedia corpus as the evidence source. It provides 185,455 claims with manual labels and uses the Wikipedia dump in June 2017 which includes 5.4 million documents.
Metrics. There are two official evaluation metries in FEVER: Label Accuracy (LA), which eval- uates the classification accuracy of the verification labels, and FEVER Score, which evaluates both the correctness of the evidence sentences used in verification and the L.A. The latter close to Joint EM in Hotpot QA and is the main metric. We use the official evaluation scripts from FEVER task and we refer to Thorn et al. (2018) for more details of this task.
Experimental Setups. We follow the experiment settings used by previous research in FEVER 1.0 shared task, i.c. Nie et al. (2019), ZZou et al. (2019), and Liu et al. (2019b). Similar as Liu et al. (2019)) we also split the data into single and multi evidence categories and evaluate Transformer- XH on the two splits.
hhtps//hooppoogaaihhhiio

---

Published as a conference paper at ICLR 2020
Dev
Test
Ams
Supp
Join
Ans
Supp
Joint
FM
F1
HM
F1
RM
F1
EM
F1
EM
FI
FM
Fi
Official Baseline Yang al. 2018
23.9
32.9
5.1
40.9
47.2
40.8
24.0
32.9
3.9
37.7
1.9
16.2
DecompRC (Min 2019a)
43.3
30.0 40.7
QFE (Nishida ct al.
2019)
28.7 38.1
14.2
44.4
8.7
23.1
(Feldman (F & HT- Yaniv 2019
31.1 40.4
17.0
47.7
11.8 27.6
30.6 40.3
16.7
47.3
10.9
27.0
CogQA (Dng et al.
2019)
37.6 49.4
23.1
58.5
12.2
35.3
37.1 48.9
22.8
57.7
12.4
34.9
SR-MRS (Nic
2019)
46.5 58.8
39.9
71.5
26.6
49.2
45.3 57.3
38.7
70.8
25.1
47.6
CogQA (w. BERT IR) [Ours]
44.8 57.7
29.2
62.8
18.5 43.4
Transformer-Xi
54.0
66.2
41.7
72.1
27.7
52.9
51.6 64.1 40.9
71.4 26.1 51.3
Table 1: Results (%) on HotpotQA FullWiki Setting. Dev results of previous methods are reported in their papers. Test results are from the leaderboard. Contemporary method is marked by +
Question Type
Reasoning Type
Comparison (1487)
Bridge (5918)
Single-Ioo (3426)
Multi-Hop (3979)
EM
F1
EM
Fl
EM
Fl
EM
Fl
CogQA SR-MRS"
43.3
51.1
36.1
49.0
45.1
61.1
31.1
39.4
62.0
68.9
42.4
56.1
52.3
68.4
41.3
50.3
CogQA (w. BERT IR)
54.1
60.9
42.4
56.9
52.0
69.3
38.6
47.8
Transformer-X XH (w. BERT IR)
59.9
65.8
52.4
66.3
62.2
78.3
46.8
55.7
Transformer-XH SR-MRS)
64.3
70.7
47.9
62.3
58.1
74.3
45.3
55.2
Table 2: Dev Ans (%) different scenarios. Reasoning Types are estimated by Min et al.
(2019b)
via whether single-hop BERT has non-zero Ans F1. The numbers questions are shown in brackets.
Baselines. The baselines include GEAR (Zhou al.] 2019) and two contemporary work, SR- MRS (Nie et al. (219) and KGAT (Liu et al. 2019b). SR-MRS uses similar adaptations as Transformer-Xet XH from Hotpot QA to FEVER. GEAR is graph attention network based approach specially designed for fact verification. KGAT further improves GEAR's 's GAT by adding the kernel information, and is the previous STOA with BERT base. We also include the BERT Concat base- line Liu et al. (2019b) which concatenates the evidence sentences to a text sequence and applies BERT on
Implementation Details. We use the retrieval result from Liu et al. (2019b) and connect all sen- tences as a fully connected graph. We follow similar parameter settings as Hothot QA. We use pre-trained BERT base model to initialize the Transformer components. The extra hop attention parameters are initialized randomly and trained from scratch, and three hop steps are used. We train Transformer- -H for two epochs.
EVALUATION RESULTS
This section first presents the evaluation results HotpotQA and FEVER. Then it conducts ablation studies, analyses, and case studies on HotpotQA to understand the effectiveness of Transformer- XH.
6.1 OVERALL RESULT
HotpotQA FullWiki results are presented in Table [ Transformer-XH outperforms all previous methods by significant margins. Besides strong results, Transformer-XH's ability to natively rep- resent structured data leads to much simpler QA system. Previously. in order to utilize pre-trained BERT, Hotpot QA approaches adapted the multi-hop reasoning task to comprise multiple sub-tasks. For example, given the retrieved documents, CogQA (w. BERT IR) first leverages one BERT MRC model to find hop entities and then another BERT MRC to find candidate answer spans. After that, it ranks the candidate spans using BERT based GAT, which is the only structure modeling step. In comparison, Transformer-Xi XH is a unified model which directly represents structured texts and integrates BERT weights.

---

Published as a conference paper at ICLR 2020
Dev
Test
Single
Evidence
Multi Evidence
T.A
FEVER
1.A
FEVER
LA
FEVER
LA
FEVER
BERT Concat (Liu et al.
2019b
73.67
68.89
71.01
65.64
GEAR/GAT Zhou et
2019)
74.84
70.69
71.60
67.10
79.79
77.42
66.12
38.21
SR-MRS* INic 2019)
75.12
70.18
72.56
67.26
KGAT" (Liu et al. 20196)
78.02
75.88
72.81
69.40
80.33
78.07
65.92
39.23
Transforeer-HH
78.05
74.98
72.39
69.07
81.84
81.31
86.58
58.47
Table 3: FEVER Results. Contemporary work is marked by Single and Multi Evidence are results on Dev claims which one multiple sentences are labeled as evidence.
Table 2 further further inspects model performances on the Dev set by question types and reasoning types. Transformer-X XH significantly outperforms all baselines on bridge questions which require more multi-hop reasoning. And on the "multi-hop" questions, Transformer- has higher relative gains (39% over CogQA on EM) than the "single-hop" questions (27%), demonstrating its stronger multi- hop reasoning capability, We further study this in Section 6.3
To further investigate the reasoning ability of Transformer- we replace our retrieval pipeline with the top retrieved documents from the SR-MRS pipeline. More specifically, we use the top retrieved documents from SR-MRS to construct Transformer-2 XH'ss evidence graph while keeping all else constant. The resulting system, Transformer-XIi (w. SR-MRS), outperforms SR-MRS's multi-step BERT based reasoning on all metrics and question types. Transformer-Xit effectiveness robust with multiple IR systems.
FEVER fact verification results are shown in Table 3 Transformer-Xe outperforms SR-MRS by 4 FEVER score Dev and 1.8 Test. It performs on par with KGAT. More importantly, Transformer-XIt excels at verifying claims that require multiple pieces of evidence-ourperfornig the contemporary work KGAT by 20 FEVER scores on the multi-evidence claims, 49% relative improvement. Compared to KGAT, Transformer-Xi mainly loses on the "not enough evidence" category which is neither single nor multi evidence. This is artifact the FEVER task which our system is not specifically designed for.
This result also demonstrates Transformer-Xe XH's generality on tasks with multiple text inputs not in sequential formats. The only difference between Transformer-Xi when applied on multi-hop QA and FEVER is the last (linear) task specific layer; provides similar or better performances over contemporary approaches that were specifically designed for the fact verification task. Due to space constraints and the consistent effectiveness of Transformer-Xe on the two applications, the rest experiments mainly used HotpotQA to analyze the behavior of TrH.
6.2 ABLATION STUDIES
Model Variations. We show the results of different model variations on the top left of Table Single-Hop BERT uses BERT MRC model on each document individually, which significantly de- creases the accuracy, confirming the importance of multi-hop reasoning FullWiki setting (Min) et al. 2019a). GAT + BERT first uses Graph Attention Network (Veličković et al. 2018) on the evi- dence graph to predict the best node; then it uses BERT MRC on the best document. is 10% worse than Transformer-XX XH since the MRC model has no access to the information from other documents. No Node Prediction eliminates the node prediction task and only trains on span prediction task; the accuracy difference shows node prediction task helps the model training.
Graph Structures. We show Transformer-XHt performance with different graph structures on the bottom left of Table g Bidirectional Edges adds reverse edges along the hyperlinks; Fully Connected Graph connects all document pairs; Node Sequence randomly permutes the documents and connects them into a sequence to simulate the Transformer-Xi setting. Both Bidirectional Links and Fully Connected Graph have comparable performance with the original graph structure. Transformer- XH is able to learn meaningful connections using its hop attentions and is less depen- dent the pre-existing graph structural. The fully connected graph can be used if there is no strong edge patterns available in the task. However, the performance drops significantly on Node Sequence, showing that structured texts cannot be treated a linear sequence which cuts off many connections.
8

---

Published as a conference paper at ICLR 2020
Dev Ans
Dev Ans
Model Ablation
Hop Steps
Fi
EM
Single-Hop BERT MRC on Individual Documents
31.3
42.2
One Hop
50.3
64.6
GAT (Node Prediction) BERT (MRC Best Node)
48.9
61.9
Two Hops
51.6
66.4
No Node Prediction Multi-Task
43.2
55.3
Four Hops
51.4
66.1
Bidirectional Edges on Hyperlinks
50.6
65.0
Five Hops
50.6
64.7
Fully Connected Graph Node Sequence (Bidirectional Transformer-XL)
51.0
65.5
Six Hops
50.1
64.2
14.1
20.7
Transformer-Xe XII
52.4
66.3
Table 4: Ablation studies on the bridge questions on Dev answer accuracy (%), including model components (top left), graph structures (bottom left), and hop steps (right). Transformer-XHH's full model uses three hop step and unidirectional Wiki link graph.
Probability Density Function
Probability Density Funation 13
Density Function
All
All Ans Supp.>Ans
Ans
All
Supp->Ans
Supp->Ans
1 23 60
Probability
0
Scaled
First Hop Attentions.
(b) Second Hop Attentions.
(c) Third Hop Attentions.
Figure 2: Distributions of learned attention weights of three hops on three groups: From All (Node)
(to) All, All + (to) Ans (ground truth answer node), and Supp (nodes with the supporting facts)
(to) Ans. X-axes are attention values scaled by number of nodes.
Hop Steps. Recall that Transformer-X XH layer with extra hop attention corresponds to in- formation propagation (hop) step in the graph. Thus Transformer-XP with last K layers conducts K-step attention hops in the graph. We show results with different K the right side of Table 4 Transformer- XH reaches its peak performance with three hops (our full- full-model). This is expected as most Hotpot QA questions can be answered by two documents (Yang et al. 2018
6.3 HOP ATTENTION ANALYSIS
This experiment analyzes the hop attentions using full-model (three-hop) the fully connected graph to study their behavior without pre-defined structure. Figure 2 plots the distributions of the learned hop attentions on the Dev set. It shows strong shift away from the normal distribu- tion with more hops. Transformer-XHr learns to distinguish different nodes after multi-hop atten- tion: the attention score becomes a bimodal distribution after three hops, ignoring some non-useful nodes. Transformer-X XH also learns to focus on meaningful edges: the score is higher on the path Supp->Ans than Als-Ans. And the margin is larger as the hop step increases from one three.
6.4
CASE STUDY
Table 5 lists two examples from Transformer-HH and CogQA (w. BERT IR). The first case has clear evidence chain "201 ""inn"""" answer. However, the second case has too many distractors the first document. Without additional clues from document 2, it is likely that the single-hop hop entity extraction component in CogQA (w. BERT IR) misses the correct answer document in its candidate sets; and the later structural reasoning component
Entertainment"; both methods find the correct
not recover from this cascade error. In comparison, Transformer-Xe finds the correct answer by combining the evidence with the hop attentions between the two evidence pieces. We leave more positive and negative cases in Appendix A.5

---

Published as a conference paper at ICLR 2020
Q: 2014 S/S is the debut album of South Ko- rean boy group that was formed by who?
Q: Which man who presented 2022 FIFA
World Cup bid was born on October 22, 1930?
Document 1: 2014 S/S the debut album of
Document 1: 2022 FIFA World Cup bid was presented by Frank Lowy, Ben Buckley, Quentin Bryce and Elle Macpherson. Document Frank Lowy (born 22 October 1930), is an Australian-Israell businessman and Chairman of Westfield Corporation. Transformer-XH. Frank Lowy V CogQA (w. BERT IR): Quentin Bryce
South Korean group Winner.
Document 2: Winner is South Korean boy
group formed in 2013 by YG Entertainment.
Transtormer-XH: YG Entertainment CogQA (w. BERT IR): YG Entertainment v
Table 5: Examples of Transformer-HH and BERT pipeline results in Hotpot QA.
RELATED WORK
HotpotQA's FullWiki task is combination of open-domain QA (Chen et al. 2017) and multi-hop QA Yangg et al. 2018): the questions are designed to require multiple pieces of evidence and these evidence pieces are documents retrieved documents are inevitably noisy and include much stronger distractors than the TF-IDF retrieved documents in the Distractor setting (Min et al.
retrieve from Wikipedia. It is a challenging combination: The
2019a
Jiagg & Bansal 2019).
Various solutions have been proposed for Hotpot QA (Min et al.
2019b Feldman & Yaniv 2019
Nishida al., (2019). These solutions often use complicated pipelines to adapt the multi-hop task into a combination of single-hop tasks, in order to leverage the advantage of pre-trained models. For example, CogQA (Ding et al. 2019) uses two BERT based MRC model to find candidate spans and then another BERT initialized Graph Neural Network (GNN) to rank spans; SR-MRS (Nie et al.) 2019) uses three BERT based rankers to find supporting sentences, and then another BERT MRC model on the concatenated sentences to get the answer span. Transformer-XH is a simpler model that directly represents and reasons with multiple pieces of evidence using extra hop attentions.
Fact verification is natural language inference task while also requires retrieving "'open-domale and reasoning with multiple text pieces (" ("mulli-ovidence"" (Thorne et al. 2018 Nie et al. 2019 Liu cet al. 2019b). Many recent FEVER systems leverage Graph Neural Networks to combine informa- tion from multiple text nodes, while each node text is represented by BERT encodings (Zhou et al.) 2019 Liu et al., 2019b). Transformer-XEr is more unified solution that simply includes language modeling as part of its joint reasoning.
In addition Tr (Dai et al. 2019), other work is proposed improve the Transformer architecture on long text sequence. For example, T-DMCA (Liu et al. 2018) splits the sequence into blocks and then the attention merges different blocks. Sparse Transformer (Child al.) [019] intro- duces the sparse factorizations of the attention matrix. Transformer-Xit XX shares similar motivation and focuses on multiple pieces of text that are not sequential forms.
Transformer-X XH is also inspired by GNN Kipf & Welling 2017, Schlichtkrull et al. 2017; Veličković et al. 2018). which leverages neural networks model graph structured data for down- stream tasks (Sun et al. 20188] Zhao et al.] 2020). The key difference is that a "node" in Transformer- XH is text sequence, and modeling of the structure is conducted jointly with the representation of the text. Transformer-XI combines the Transformer's advantages in understanding text with the power that GNN has in modeling structure.
8 CONCLUSION
Transformer- XH and its eXtra Hop attention mechanism is a simple yet powerful adaptation of Transformer to learn better representations of structured text data as it naturally occurs. It innately integrates with pre-trained language models to allow for complex reasoning across multiple tex- tual evidence pieces. When applied to HotporQA, Transformer-XH significantly shrinks the typical multi-hop QA pipeline, eliminating many cascading errors that arise from the linear sequence in- put constraints of pre- traitrdined Transformers. The same simplicity also applies to FEVER, with one Transformer XH all we needed to obtain a much stronger answer accuracy. With its simplicity and efficacy, we envision Transformer-Xi will benefit many applications in the near future.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to Answer Open- Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics, pp. 1870-1879, 2017.
Rewon Child, Scott Gray, Alec Radford, and Nya Sutskever. Generating Long Sequences with Sparse Transformers. arXiv preprint arXiv: 1904. 10509, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer- XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-299829 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4171-4186, 2019.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive Graph for Multi-Hop Reading Comprehension at Scale. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2694-2703, 2019.
Yair Feldman and Ran El- Yaniv. Multi-Hop Paragraph Retrieval for Open-Domain Question An- swering. In Proceedings of the S7th Annual Meeting of the Association for Computational Lin- guistics, pp. 2296-23090 2019.
Paolo Ferragina and Ugo Scaiella. Tagme: On-the-fly Annotation of Short Text Fragments (by Wikipedia Entities). In Proceedings of the 19th ACM international conference on Information and knowledge management, pp. 1625-1628, 2010.
Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsberg. Entity Linking in Queries: Efficiency vs.Effecctivness. In European Conference on Information Retrieval, pp- 40-53, 2017.
Yichen Jiang and Mohit Bansal. Avoiding Reasoning Shorteuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2726-2736, 2019.
Thomas N Kipf and Max Welling. Semi-supervised Classification with Graph Convolutional Net- works. In International Conference on Learning Representations, 2017.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating Wikipedia by Summarizing Long Sequences. In International Conference on Learning Representations, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv preprint arXiv: /907. / 1692, 2019a.
Zhenghao Liu, Chenyan Xiong, and Maosong Sun. Kernel Graph Attention Network for Fact Veri- fication. arXiv preprint arXiv:1910.0977 2019b.
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettle- moyer. Compositional Questions Do Not Necessitate Multi-hop Reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computationall Linguistics, pp. 4249-4257, 2019a.
Sewon Min, Victor Zhong, Luke Zettlemoyer and Hannaneh Hajishirzi. Multi-hop Reading Com- prehension through Question Decomposition and Rescoring. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6097-6109, 2019b.
Yixin Nie, Songhe Wang, and Mohit Bansal. Revealing the Importance of Semantic Retrieval for Machine Reading at Scale. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019.
11

---

Published as a conference paper at ICLR 2020
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito. Hisako Asano, and Junji Tomita. Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pp. 2335-2345, 2019.
Rodrigo Nogueira and Kyunghyun Cho.
Passage Re-ranking with BERT. arXiv preprint
WwXiv:1901.0405.. 2019.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling Relational Data with Graph Convolutional Networks, arXiv preprint 1rXiv:1703.06100 2017.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 4231-4242, 2018.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodooulppoulss, and Arpit Mittal. The Fact Extraction and VERification (FEVER) Shared Task. In Proceedings of the First Work- shop on Fact Extruction and VERification (FEVER), pp. 1-9, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In Advances in neural infor- mation processing systems, pp. 5998-6008 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph Attention Networks. In International Conference on Learning Representations, 2018.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J Smola, and Zheng Zhang. Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Ques- tion Answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 2369-269922 2380, 2018
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems, pp. 5754-5764, 2019.
Chen Zhao, Chenyan Xiong, Xin Qian, and Jordan Boyd-Graber Complex factoid question answer- ing with a free-text knowledge graph. The Web Conference, 2020.
Jie Zhou, Xu Han, Cheng Yang. Zhiyuan Liu, Lifeng Wang, Changeheng Li, and Maosong Sun. GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 892-901, 2019.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
The appendix includes details of the evidence graph construction for Hotpot QA, ablation studies in the BERT IR component, more details and results on Hotpot QA.
A.1 HOTPOTQA EVIDENCE GRAPH CONSTRUCTION DETAILS
The evidence graph construction includes two stages. The first stage is BERT IR, which extract related documents directly from question. The second stage expands the related documents along Wikipedia links. The first applied on all questions while the second is only required by Bridge questions.
The first stage uses two methods to find documents. The first method uses DrQA's retrieval sys- tem (Chen et al. 2017), which is unsupervised TF-IDF. We keep the top 100 DrQA retrieved ones Dir for each question. The second method uses TagMe (Ferragina & Scaiella 2010) and CMNS (Hasibi al.] (017), commonly used entity linkers, annotate questions. We keep the TagMe output entity and three highest scored entities per surface form (a phrase in the question linked with entities) from CMNS, and use its corresponding Wikipedia document as Dei-
We use BERT ranker (Nogueira & Cho 2019) to re-rank the initial set D,, U Det. The input to the BERT is the concatenation of question and first paragraph of document:
[CLS] Question [SEP] First Paragraph of Document.
Then linear layer is added on the last layer's [CLS] representation to score the relevance of the document. We use BERT base and fine-tune it using the relevance label (from supporting facts) with cross-entropy loss. The top two highest scored documents from Dår and the top document per entity position (surface form) in Del are kept as the first stage BERT IR documents.
The second stage expands the first stage BERT IR documents by Wikipedia hyperlinks to obtain Dexp- document is included if it linked to or links to document in the first stage. We use the same BERT ranker rank Dexp and keep the top 15 documents in Deap-
The final evidence graph nodes per question includes the top two highest ranked documents Dirs top one per entity name in Det, and top 15 from the expanded documents Deap-
The comparison questions only require information from two question entities; thus when building the evidence graph do not expand them (i.e. there Deap).
The retrieval pipeline trieval system used in SR-MRS (Nie al.) 2019). When using SR-MRS retrieved documents for documents, we use top 10 documents on bridge questions and top two documents on comparison questions.
multi-stage retrieval enhanced with entity linking. close to the re-
In the next section, we show that Transformer- XH is robust to different number of documents kept in the evidence graph and performs similarly using the documents retrieved from SR-MRS.
A.2 ABLATION STUDIES ON DOCUMENT RETRIEVAL
This experiment studies the effectiveness and influence of different retrieval settings. We use differ- ent numbers of top K ranked documents from the BERT ranker, run Transformer-XH the corre- sponding evidence graph, and evaluate its performance on Bridge questions in the Dev set. We also evaluate the Supporting facts Recall and Answer Recall. Supp Recall evaluates whether the doeu- ment with the supporting fact is included in the first stage retrieved documents. Ans Recall evaluates whether their exists a document in the evidence graph that includes the ground truth answer. The results in Table T
Our BERT IR system performs better than CogQA's TF-IDF and on par with SR-MRS, as expected. The latter uses similar retrieval pipeline with our BERT IR system; Transformer-XH is robust on different retrieval settings and keeps its effectiveness when applied on Top 10 documents from SR-MRS (including both stages).
13

---

Published as a conference paper at ICLR 2020
Method
Supp Recall
Ans Recall
Dev Ans EM
Dev Ans F1
Top 10 THIDF (CogQA)
70.8
Top 2 w. BERT IR Q Entites
72.3
88.1
48.7
62.6
Top 5 w. BERT IR Entites
76.5
89.6
47.1
60.8
Top 10 w. BERT IR + Q Entites
78.9
91.1
46.6
60.3
SR-MRS Top 10 (All Together)
n.a.
86.1
47.9
62.3
Table 6: Ablation study the retrieval systems. Top-10 10 TFIDF is the used by CogQA (Ding
et al. 2019). BERT IR is the retrieval system used by CogQA (w. BERT IR) and Transformer- XH. Top refers to using the 2/5/10 highest ranked documents from the BERT ranker in the first stage. SR-MRS Top 10 uses the 10 retrieved documents per question provided by Nie et al. (2019). All
retrieval methods include entities linked the question and are expanded along Wiki links, except when evaluating the 1st stage Supp Recall.
A.3 OTHER HOTPOT QA COMPONENTS
This section describes the other components for HotpotQA dataset. The whole QA system starts with question type classification. We train transformer-XI separately on each type of questions over their evidence graph. Besides answer prediction, we also adopt BERT based model for predicting supporting sentences.
A.3.1 QUESTION CLASSIFICATION
The first component of our system is to classify the question to bridge and comparison types. We adopt BERT classification fine-tuning setting on HotpotQA questions using the question type labels provided in HotpotQA. The classifier achieves 99.1% accuracy on the dev set. We use the classifier to split the questions into Comparison and Bridge.
A.3.2 SUPPORTING FACTS CLASSIFICATION
The supporting facts prediction task is extract all sentences that help get the answer. For bridge question, these sentences usually cover different pieces of questions. And for comparison questions, supporting facts are the properties of two question entities. We design one model architecture for this task, but we train two models each type reflect the inherent difference.
We use BERT as our base model and on top of BERT, we conduct multi-task learning scheme. The first task is document relevance prediction, similar Transformer-XH, [CLS] token of BERT to predict the relevance score. The other task is sentence binary classification, concatenate the first and last token representation of each sentence in the document through linear layer, the binary output decides whether this sentence is supporting sentence.
add linear layer on the
Bridge question supporting facts prediction For bridge questions, we predict supporting facts after answer prediction from Transformer to resume the inference chain. We start by predicting supporting facts in the answer document. The other document chosen from the parents of the answer document in the evidence graph. F] Compare with the contemporary Nie et al. (2019), which does not limit the search space along the inference chain (i.e., the answer document may not be relevant the other supporting page), our method more naturally fits the task purpose.
Comparison question supporting facts prediction For comparison questions, after extracting the first step documents D, we simply run this supporting facts prediction model select the top-2 documents, and predict the corresponding supporting facts.
A.3.3 TRAINING DETAILS
We use DGL Wang et al. 2019) for implementing Transformer-XHt and CogQA (w. BERT IR) with batch size (i.e., one graph for each batch), and keep the other parameters same as default BERT setting. We train Transforme- XH separately on two different types of questions, following previous
'If the answer document does have parent node, we choose from all documents
14

---

Published as a conference paper at ICLR 2020
id
Example
Explanation In which year the King who made 1925 Birthday Honours born?
I(+)
With necessary evidence available, Transformer-Xe conducts multi-hop reasoning, and extracts the correct span.
P: 1865 V
Document I:
The 1925 Birthday Honours were ap- pointments by King George V to various orders and honours.
Document 2:
George V (3 June 1865 20 January 1936) King of the United Kingdom.
2(-)
Q: Where was the world cup hosted that Algeria quali- fied for the first time into the round of 16?
Transformer-XIH does not predict the correct answer, since document I does not link to any other docu- ments.Thus, the information does not propagate to the correct answer docu- ment 2014 FIFA CUP.
A: Brazil P: Spain
Document 1 (Algeria the FIFA World Cup): 2014, Algeria qualifted for the first time into the round of 16.
Document 2 (2014 FIFA CUP): It took place in Brazil from 12 June 13 July 2014, after the country was awarded the hosting rights in 2007.
3(-)
Q: What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell? Chief of Protocol
Transformer-XH predicts the correct answer document Shirley Temple. However it could not distinguish from the wrong answer ambassador which she was named but not held that position.
ambassador
Document 1: Kiss and Tell is a 1945 American comedy film starring then 17-year-old old Shirley Temple as Corliss Archer.
adult, Shirley Temple was named United States ambassador to Ghana and to Czechosla- vakia, and also served as Chief of Protocol of the United States.
Document
Table 7: Additional examples for model prediction on HotpotQA dataset, the first example is the correct prediction (+), the other two examples are the wrong predictions (-).
research (Ding al. 2019). We train Transformer- XH and the GNN of CogQA (w. BERT IR) for epochs. All other BERT based models use the default BERT parameters and train the model for epoch.
A.4 IMPLEMENTATION DETAILS OF COGQA (W. BERT IR)
This section discusses our implementation of CogQA (W. BERT IR). We start with the same docu- ments from BERT IR, the one used by Transformer-Xee and then implement the following steps:
A.4.1 HOP ENTITY EXTRACTION
For each document from the previous step, we run BERT MRC model and limit the span candidates as hyperlinked entities for hop entity extraction (e.g., Figure 1. "Harvard University" a hop en- tity). Following Ding et al. (2019), we predict the top three entities that above the relative threshold that is the start span probability of [CLS] position.
A.4.2 ANSWER SPAN EXTRACTION
Doin For each document (add the hop entity document), following Ding et al. (2019). we run BERT MRC model extract spans (e.g., "Combridge" in Figure 1.). We predict the top span that above the threshold that is the start span probability of [CLS] position.
We train both hop entity extraction and span extraction tasks with same BERT model but different prediction layers. For each training example, we extract the link between two given supporting pages. The page includes the link (e.g., "Harvard University" in Figure 1.) is the supporting page
15

---

Published as a conference paper at ICLR 2020
for hop entity extraction, while the other page is the answer page (e.g., "Combridge" in Figure 1.) for answer span extraction.
A.4.3 GAT MODELING
All the entities and answer spans form the final graph. The nodes are the entities and spans, and edges are the connections from the entities to the extracted hop entities or spans.
We use BERT for each node representation with question, anchor sentences and context, follow- ing Ding et al. (201)). We run GAT (Veličković et al. 2018) on top of BERT to predict the correct answer span node.
A.4.4 COMPARISON QUESTIONS
After predicting supporting facts, we concatenate the sentences and follow Min et (2019b) to run a BERT MRC model to predict either span or yes/no as the answer.
A.5
ADDITIONAL, CASE STUDY
We provide addition case studies in Table The first case can be directly predicted through the clear evidence chain "the 1925 Birthday "onours"=""Gerr"" ""-,"1855". In the second case, the first document ("Algeria at the FIFA World Cup") has no link to any other documents, therefore the model can not access the correct answer. The third case is more reading comprehension oriented, where the model can not distinguish the correct and wrong spans inside one sentence.
16


Reviews:
[{"review_id": "HkeM5zCEtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the Transformer XH model architecture, a transformer architecture that scales better to long sequences / multi-paragraph text compared to the standard transformer architecture. As I understand, Transformer XH is different from the standard transformer architecture in two main ways: 1) adding attention across paragraphs (or sub-sequences) via the CLS token and 2) defining the structure of that attention based on the entities in the paragraphs (or sub-sequences). The paper tackles an important problem (learning from long sequences) and achieves good empirical results on HotpotQA.\n\nModification #1 has already been explored by previous works which should be discussed in the paper. \"Generating Wikipedia by Summarizing Long Sequences\" by Liu, Saleh, et al. propose the T-DMCA model, with a similar motivation: enabling the transformer to work on/scale to longer sequences. T-DMCA and Transformer XH have some difference (T-DMCA seems to have more capacity while Transformer XH is simpler); I think it is necessary to compare against Transformer XH against T-DMCA on HotpotQA to know whether a new architecture is really necessary for HotpotQA. \"Generating Long Sequences with Sparse Transformers\" from Child et al. also proposes another general transformer architecture that can handle long sequences, and it would be ideal to compare against this architecture as well. Sparse Transformers reduce the time complexity of attention to reduce O(n√ n), which seems similar to the reduction that Transformer XH gets.\n\nFor modification #2 (defining the attention structure beforehand using e.g. entity linking), it does not seem too difficult to learn the attention structure directly instead, as confirmed by the ablation in Table 3 which uses a fully connected graph structure. A model that learned the attention pattern or used a fully connected graph would be more general (but more similar to T-DMCA and sparse transformers).\n\nThe empirical results are good. It's nice that a simple/straightforward architecture like Transformer XH works quite well (compared to some previous approaches which were not as elegant). However, I do not feel that prior work has explored the best transformer architectures for HotpotQA (such as Sparse Transformers or T-DMCA), and since this work is specifically proposing a new transformer architecture, I think it is important to compare directly against other transformer architectures. Other (previously) SOTA models like SR-MRS are quite simple (conceptually), so it's likely that such models will also be outperformed by transformer architectures that are better adapted to long sequences. In general, I think that the most relevant baseline already present is SR-MRS rather than CogQA; the fact that such a simple approach like SR-MRS works well is an important fact about HotpotQA to take into account (even if SR-MRS is concurrent). In other words, SR-MRS shows that previous models like CogQA are likely weaker baselines.\n\nBecause of the missing baselines and limited novelty compared to prior work, I overall lean towards rejecting the paper (despite the good empirical results).\n\nI do have a few specific questions for the authors:\n- Would you mind providing further details about the following sentence? \"For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set Dir ∪ Del and keep top K ranked ones.\"\n- Is only answer-level supervision used? Or is supporting-fact level supervision used to train any of the rankers or pipeline?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlHcZMptB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces a way to train transformer models over document graphs, where each node is a document and edges connect related documents. It is inspired by the transformer-XL model, as well as Graph Neural networks. They apply this model to answer multi-hop questions on the HotPotQA dataset and outperform the previous SOTA. The model particularly improves performance on the bridge style questions of HotPotQA. They are able to do this in a single step, rather than a multi-stage process as done by previous approaches.\n\nStrengths: The model is described in a very detailed manner with contrasts drawn to previous models, which provides excellent motivation for the decisions taken by the authors. I enjoyed reading section 2 as it very succinctly describes previous approaches and introduces transformer-XH. The paper has very detailed and insightful ablation studies, including hop steps and hop attentions, and other graph structures.\n\nWeaknesses:\n\nIt took me a lot of effort to figure out that the transformer-XH is only applied to the bridge questions, and part of the overall gains are due to a better retrieval pipeline on the comparison questions. Please explicitly make this clear. \n\nAlso, there seems to be a lot of gain even on single-hop questions, and its not clear if overall performance improvement can be attributed to modeling the graph structure, as opposed to other confounding factors. Can the authors elaborate a bit more on why this might improve performance on single hop questions?\n\nVery good evaluation on HotPotQA, but would be even stronger if this were applied to at least one other task/dataset.\n\nQuestions: \n\n1. Any intuition as to why the EM performance improvement on single-hop questions is the about the same as the performance improvement on the multi-hop questions (~5%)? \n\n2. In Example 2 in Table 4, it is not clear from the text as to why the BERT pipeline fails to get the correct result. If I understand correctly, both models use the same document graph construction method? Is this not the case? i.e. does the pipeline model have access to the exact same documents that form the Transformer-XH's document graph? Could you explain this cascade error here a bit more?\n\n3. I assume that we can use directed as well as undirected edges in the document graph? Would be good to clarify this. \n\n4. In equations 11 and 12, are you missing normalization operators to specify a distribution, perhaps a softmax?\n\n5. \"Our pipeline\" in Table 1 is confusing. Would be nice to mention in the caption that it is a baseline constructed using BERT on your retrieval method etc.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1ghg2ACYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is proposing an extension of the Transformer matching and diffusion mechanism to multi-document settings.\nTo do so, the authors introduce a special representation for gathering document level information which is then used for propagation among documents latent representations.\nThe extension seems quite simple and natural.\nThe method is evaluated on multi-hop machine reading over the hotpotqa dataset in the Fullwiki settings.\nHowever, it could have made sense to evaluate the method in the distractor settings too.\nIn this context, the evidence graph where the model is trained is built using the canonical retrieval technique.\nThen, the method is using a pre-trained NER model to extract entities on the question and the candidate documents on Wikipedia for matching.\nFinally, a BERT ranker model is used to re-rank the retrieved candidate documents.\nThe proposed method seems to heavily dependant on this hand-crafted extraction process.\nUnfortunately, one concern is that the reasoning model, while been quite original, is not tested in large scale retrieval cases to assess its robustness.\nIndeed, the number of retrieved documents to create the evidence graph seems to not have been mentioned.\nThe method improves the current state of the art.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work examines a problem that is of considerable interest to the community and does a good job of presenting the work. The AC recommends acceptance.
Decision: Accept (Poster)