Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper is proposing a multi-hop machine reading method tested on hotpotqa in the Full Wikipedia setting and squad-open datasets.
For hotpotqa, It could also have been interesting to evaluate the method of the distractor ones.
First, the proposed method constructs a graph over the Wikipedia pages represented by their respective summary paragraphs.
In this representation, the hyperlinks among pages represent the edges.
Then, the authors trained a normalized RNN model to retrieve the candidate reasoning paths from the question.
The model is bootstrap using TF-IDF page retrieval techniques.
Then, a Beam-search decoding strategy is used to retrieve "reasoning path" which is then pass through a BertQA model using a simple question-reasoning-path concatenation technique.
One originality of the method is the negative sampling strategy that includes negative TF-IDF retrieval as starting points to robustify the sequential extraction process.
The detailed experiments and ablation tests give to illustrate the experimental relevance of the proposed method.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary
========
This paper introduces a graph-based recurrent retrieval model for retrieving evidence documents in a multi-hop reasoning question answering task. The main idea is that (1) the graph formed by Wikipedia links between passages can be used as constraint for constructing reasoning chains, and (2) the joint encoding of the question and current passage can be used to retrieve a subsequent passage in the reasoning chain. The paper describes a model for implementing the above retrieval system, and how they jointly train with a reading comprehension model. They demonstrate the effectiveness of the system on HotPotQA, showing improvements over previously published models, and SQuaD-Open, showing competitive results.

Overall Comments
===============
The paper is an interesting, but incremental, improvement to the area of question answering. Overall, there are two main concerns about this work. First, while the results are somewhat strong, the ideas presented are small variations on existing systems. For example, Godbole et al 2019 and Ding et al. 2019 both explore using graphical structural to constraint iterative, multi-hop, retrieval. Also, Feldman et al 2019, describe an encoder based approach to encode question and paragraph context for iterative retrieval. Asides from smaller modeling differences (choice of RNN, training regime, BERT reader, etc.) to account for the difference in results, the main difference seems to be the joint training of the retrieval system with the reader. Secondly, the paper lacks clarity on some formal definitions and definition of the graph, making it hard to understand the content precisely.

Detailed Comments
================
Below are some detailed comments about specific parts of the paper, in order of importance:

1. One important limitation of this technique is the reliance on a linked documents for constructing the retrieval system. It is not clear from the paper how much of the results are obtained from constraining the set of retrieved passages (after the initial retrieval) to Wikipedia links. And whether, for example, substituting Wikipedia links with links derived from an off-the-shelf entity linking system would suffice.

2. Given that the retrieval model is restricted to link structure in Wikipedia that induces the proposed retrieval graph, I assume that there are “reasoning paths” that do not exist in the graph, given Wikipedia’s policy of avoiding adding redundant links within a Wikipedia page. It would have been informative to conduct an “Oracle” experiment: that is, given the initial  set of retrieved nodes and the graph structure, are there *any* paths that provide the correct answer and reasoning chain? That is to say, what is the upper-bound performance on the proposed system given the currently induced Wikipedia graph?

3. In Section 3, and even later on in the paper, it was not clear what “E” denotes. It never seems to be defined, and is used interchangeably with “graph node”, “wikipedia page”, “wikipedia paragraph” and “reasoning path”. Are these the same thing? It would be much clearer to define what E means, and perhaps separate the different concepts (node, passage, reasoning path) properly.

4. In Section 3, it seems that ‘q’ is not defined. Is it the question?

5. In Section 3.1, it is not clear what the graph actually contains. Does it contain all the paragraphs from Wikipedia? Just the paragraphs with links? The first paragraph of every Wikipedia page? What granularity of the wikipedia page becomes an individual node in the graph?

6. In Section 3.1.1., the representation of the starting retrieval (i.e., time-step = 0), h_0, is not defined. Later in the section, the paper mentions the use of TF-IDF for the initial set of nodes, instead of the learned retrieval model. This seems a bit unusual design decision without further explanation. Particularly when taking the results in Table 4, showing TF-IDF based retrieval performs worse that the learned retrieval system from the proposed model.

7. In Section 3, C_{t} (the candidate set of paragraphs) is not defined. This is an important set to define. Is it the set of paragraphs derived from Wikipedia links, starting from the current node?

8. In Section 3.1.2, “Loss function”, the term g_{r} is not defined.

9. In Section 4.4 “Analysis on reasoning path length”, it would have been useful to see the performance of the model with different path lengths. This analysis is somewhat common on multi-hop reasoning tasks, and should be included.

10. Typo in Section 4.4: “..., and out model is likely too terminate …”  should be “ likely to terminate “

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes a method to find a sequence of reasoning paragraphs in Wikipedia to answer queries requiring multi-hop reasoning. They make the key observation that answering multi-hop queries might require retrieving evidence that have very less lexical overlap with the question. Given a query, the proposed method starts from a set of initial paragraphs retrieved by a tf-idf retriever and uses the outgoing Wikipedia anchor link to hop to the next evidence. They propose a simple recurrent neural network that takes in the current paragraph (and the hidden state) and decide which paragraph to hop to in the next step. Because of the available supervision for the paragraphs (in HotpotQA), they can train a supervised path selector. They also add a special EoE token that denotes the end of the reasoning path, thereby having the ability to produce reasoning paths of different lengths. After training the retriever a beam of reasoning paths is sent to the reader module. The reader module re-ranks the reasoning paths again and then use a standard BERTQA model and the top re-ranked chain of paragraphs to find the evidence.

Overall, the paper presents a well-designed system for handling multi-hop queries and the explicit recurrent state is a nice contribution and addition to the IR model proposed in Godbole et al., 2019. The paper is clearly written for the most part.

===Update (11/12/2019)===
The authors have addressed all my comments and have improved the results since. I am recommending acceptance. Nice work.

Strengths:
— The proposed method has demonstrated strong results on 2 datasets in challenging open-domain settings. The ablation results are helpful.
— The paper is clearly written and was straightforward to follow

Weaknesses:

1.  The paper mentions that it studies the interplay between the retriever and reader. It is unclear how it is doing so, since the retriever and the reader are not explicitly interacting with each other. Cant the retriever and the reader be trained separately? 
2.  It is unclear / not motivated, why there is an extra step of re-ranking required in the reading stage? In other words, what kinds of extra inductive bias is this additional step of re-ranking providing since the same kind of supervision was used while training the retriever model. I do note that the ablation study is helpful and it is clear that it is effective, but it would be nice to see a discussion regarding why this second step of re-ranking helps.
3. Since the reader model (BERT reader) takes the top scoring chain of paragraphs concatenated together, that would imply that it is currently limited by the number of positional embeddings in the BERT model (512 tokens). I think this limitation should be explicitly mentioned and possible remedies discussed.
4. The current approach is heavily dependent on Wikipedia graph and will not work if the hyperlink graph is not provided. It would have been nice to have an entity linker component that could also create the graph structure. I believe concurrent work such as Godbole et al., 2019 has addressed this and the paper should mention this while contrasting with their work. 
5. From figure 2, I got an impression that since the reader scored the span in "Top 2 reasoning path” higher, that was selected. But after section 3.2, I was left confused because it looks like the reader model consumes the top scoring chain after the second stage of re-ranking. This is not clear from the figure and should be fixed.
6. Discussion on scalability: Although the retriever is clearly very effective for such questions, the running time would be prohibitive (for open domain QA) as at test time, query dependent context representations is constructed for each of the paragraph in the reasoning chain. I would like to see a discussion / some running time comparison where query independent paragraph representations are constructed and the network just encodes the query independently at test time.

Minor: Typo liked -> linked (Sec 4.3, line 5)
review_title: Official Blind Review #3

Meta-review: The paper proposed a multi-hop machine reading method for hotpotqa and squad-open datasets. The reviewers agreed that it is very interesting to learn to retrieve, and the paper presents an interesting solution. Some additional experiments as suggested by the reviewers will help improve the paper further. 
Decision: Accept (Poster)