Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper is proposing an extension of the Transformer matching and diffusion mechanism to multi-document settings.
To do so, the authors introduce a special representation for gathering document level information which is then used for propagation among documents latent representations.
The extension seems quite simple and natural.
The method is evaluated on multi-hop machine reading over the hotpotqa dataset in the Fullwiki settings.
However, it could have made sense to evaluate the method in the distractor settings too.
In this context, the evidence graph where the model is trained is built using the canonical retrieval technique.
Then, the method is using a pre-trained NER model to extract entities on the question and the candidate documents on Wikipedia for matching.
Finally, a BERT ranker model is used to re-rank the retrieved candidate documents.
The proposed method seems to heavily dependant on this hand-crafted extraction process.
Unfortunately, one concern is that the reasoning model, while been quite original, is not tested in large scale retrieval cases to assess its robustness.
Indeed, the number of retrieved documents to create the evidence graph seems to not have been mentioned.
The method improves the current state of the art.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Summary: This paper introduces a way to train transformer models over document graphs, where each node is a document and edges connect related documents. It is inspired by the transformer-XL model, as well as Graph Neural networks. They apply this model to answer multi-hop questions on the HotPotQA dataset and outperform the previous SOTA. The model particularly improves performance on the bridge style questions of HotPotQA. They are able to do this in a single step, rather than a multi-stage process as done by previous approaches.

Strengths: The model is described in a very detailed manner with contrasts drawn to previous models, which provides excellent motivation for the decisions taken by the authors. I enjoyed reading section 2 as it very succinctly describes previous approaches and introduces transformer-XH. The paper has very detailed and insightful ablation studies, including hop steps and hop attentions, and other graph structures.

Weaknesses:

It took me a lot of effort to figure out that the transformer-XH is only applied to the bridge questions, and part of the overall gains are due to a better retrieval pipeline on the comparison questions. Please explicitly make this clear. 

Also, there seems to be a lot of gain even on single-hop questions, and its not clear if overall performance improvement can be attributed to modeling the graph structure, as opposed to other confounding factors. Can the authors elaborate a bit more on why this might improve performance on single hop questions?

Very good evaluation on HotPotQA, but would be even stronger if this were applied to at least one other task/dataset.

Questions: 

1. Any intuition as to why the EM performance improvement on single-hop questions is the about the same as the performance improvement on the multi-hop questions (~5%)? 

2. In Example 2 in Table 4, it is not clear from the text as to why the BERT pipeline fails to get the correct result. If I understand correctly, both models use the same document graph construction method? Is this not the case? i.e. does the pipeline model have access to the exact same documents that form the Transformer-XH's document graph? Could you explain this cascade error here a bit more?

3. I assume that we can use directed as well as undirected edges in the document graph? Would be good to clarify this. 

4. In equations 11 and 12, are you missing normalization operators to specify a distribution, perhaps a softmax?

5. "Our pipeline" in Table 1 is confusing. Would be nice to mention in the caption that it is a baseline constructed using BERT on your retrieval method etc.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper introduces the Transformer XH model architecture, a transformer architecture that scales better to long sequences / multi-paragraph text compared to the standard transformer architecture. As I understand, Transformer XH is different from the standard transformer architecture in two main ways: 1) adding attention across paragraphs (or sub-sequences) via the CLS token and 2) defining the structure of that attention based on the entities in the paragraphs (or sub-sequences). The paper tackles an important problem (learning from long sequences) and achieves good empirical results on HotpotQA.

Modification #1 has already been explored by previous works which should be discussed in the paper. "Generating Wikipedia by Summarizing Long Sequences" by Liu, Saleh, et al. propose the T-DMCA model, with a similar motivation: enabling the transformer to work on/scale to longer sequences. T-DMCA and Transformer XH have some difference (T-DMCA seems to have more capacity while Transformer XH is simpler); I think it is necessary to compare against Transformer XH against T-DMCA on HotpotQA to know whether a new architecture is really necessary for HotpotQA. "Generating Long Sequences with Sparse Transformers" from Child et al. also proposes another general transformer architecture that can handle long sequences, and it would be ideal to compare against this architecture as well. Sparse Transformers reduce the time complexity of attention to reduce O(n√ n), which seems similar to the reduction that Transformer XH gets.

For modification #2 (defining the attention structure beforehand using e.g. entity linking), it does not seem too difficult to learn the attention structure directly instead, as confirmed by the ablation in Table 3 which uses a fully connected graph structure. A model that learned the attention pattern or used a fully connected graph would be more general (but more similar to T-DMCA and sparse transformers).

The empirical results are good. It's nice that a simple/straightforward architecture like Transformer XH works quite well (compared to some previous approaches which were not as elegant). However, I do not feel that prior work has explored the best transformer architectures for HotpotQA (such as Sparse Transformers or T-DMCA), and since this work is specifically proposing a new transformer architecture, I think it is important to compare directly against other transformer architectures. Other (previously) SOTA models like SR-MRS are quite simple (conceptually), so it's likely that such models will also be outperformed by transformer architectures that are better adapted to long sequences. In general, I think that the most relevant baseline already present is SR-MRS rather than CogQA; the fact that such a simple approach like SR-MRS works well is an important fact about HotpotQA to take into account (even if SR-MRS is concurrent). In other words, SR-MRS shows that previous models like CogQA are likely weaker baselines.

Because of the missing baselines and limited novelty compared to prior work, I overall lean towards rejecting the paper (despite the good empirical results).

I do have a few specific questions for the authors:
- Would you mind providing further details about the following sentence? "For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set Dir ∪ Del and keep top K ranked ones."
- Is only answer-level supervision used? Or is supporting-fact level supervision used to train any of the rankers or pipeline?
review_title: Official Blind Review #3

Meta-review: This work examines a problem that is of considerable interest to the community and does a good job of presenting the work. The AC recommends acceptance.
Decision: Accept (Poster)