Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper makes contributions to passage retrieval in the open-domain, multi-hop setting.  It builds on recent work which applied dense passage retrieval iteratively to work for multi-hop tasks (such as MDR and IRRR).  It proposes three modifications to existing work: condensed retrieval, which summarizes retrieved passages at each hop to reduce the search space; focused late interaction, which is a variation on Colbert to consider only top-k scores for each query embedding; and latent hop ordering, which is a learned strategy to order the passages which improves over existing heuristics.  Each of these modifications are relatively minor, but together lead to very strong results on the often used HotpotQA and the newly released HoVer benchmarks.
main_review: The paper builds on recent advances in multi-hop retrieval using dense representations.  The contributions are not ground-breaking, but clear.  The ideas are well executed, with strong results and clearly presented.  Overall this is a solid candidate for publication at NeurIPS.

Originality:  The contributions consist of 3 improvements to multi-hop dense retrieval.  2 of these (focused late interaction and latent hop ordering) are very minor and can be considered incremental.  The remaining one, condensed retrieval, is a neat and novel idea which incorporates summarization into multi-hop retrieval.  All three contributions are clear, well executed and clearly demonstrated to improve performance.  The end result is a much better, 'polished' system for multi-hop dense retrieval.

Significance:  Unfortunately 'multi-hop reasoning' (despite the exciting name) suffers from a lack of natural benchmarks and real-world applications, which somewhat limits the significance of advances in this area.  Nevertheless, this is not a weakness of the paper.  The results are strong, and the methods are clear and simple enough for others to adopt.  

Quality:  The experiments are conducted against very strong baselines, and the improvements are convincing.  There are enough ablations to demonstrate the contribution of each component.  

Clarity:  The presentation is clear and generally easy to follow.  The authors make it very clear what the current SotA is and exactly what their contributions are.

Questions/feedback:
- What is the criteria for claiming "saturation" for HotpotQA?  Given no human upper-bound, it would be helpful to make this section more formal.  While many are eager to 'retire' HotpotQA, I don't think simply attaining 90+ accuracy is sufficient to make the claim.  One suggestion could be to look at the remaining examples, and show they are somehow irrelevant (e.g. bad annotation, ambiguous question etc.)
- What does top-20 passage recall mean exactly when condensing passages?
- Some of the important implementation / training details can be moved to the main text from the appendix.  Currently the manuscript is pretty light in details (if one doesn't read the appendix).
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed Baleen a system that improves the accuracy of multi-hop retrieval while learning robustly from weak training signals in the many-hop setting. Baleen generally contains three modules -- Focused Late Interaction, Latent Hop Ordering, and Condenser. Focused Late Interaction is an improved version of ColBERT, in which they only consider $\text{top-}\hat{K}$ partial scores instead of all scores. Latent Hop Ordering proposed a way to supervise the retriever using weak supervision per-hop. The condenser summaries the sentence-level information retrieved per-hop so it could be better used for the next hop. Experimental results on two multi-hop-based datasets, namely HotpotQA and HoVer show that the proposed framework is quite effective for retrieving multi-hop evidence. 
main_review: Strength:
1. The authors implemented a multi-hop retrieval system that achieved very strong performance on two multi-hop-based datasets.
2. The idea behind the proposed modules is simple yet effective.
3. Reasonable ablations showing the effectiveness of each module.

Weaknesses:
Although the system proposed in this paper is quite strong, there are not many insights I can draw from the paper as many of the proposed architecture changes are heuristic-based, e.g., taking $\text{top-}\hat{K}$ partial scores is straightforward but how do we choose $\hat{K}$? The idea of using a condenser is also quite intuitive, but how do we choose the negatives and how many negatives are needed? As we can see in the appendix, there are so many hyperparameters in the appendix and I am not sure to what extent those hyperparameters would affect the performance. 

Question for the authors:
1. Compared to some of the previous literature, e.g., IRRR, DPR, the proposed framework seems much more computationally expensive; do you have an idea of how slow the proposed method is compared to those models for both training and inference?

limitations_and_societal_impact: The authors have sufficiently addressed the limitations of their work and I don't think there is a potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a new multi-hop reasoning system called Baleen which conducts iterative retrieval and condensing. Baleen repeats the iterative process for pre-defined time steps T (e.g., T=2 for HotpotQA and T=4 for HoVer)  and at every time step, it takes the query from the previous time step and retriever top K passages, and create a new query for the current t step by extracting and condensing the facts (sentences) from the retrieved passages. When they finish retrieval (i.e., t=T), they feed the final query to the reader model to get the final predictions. 
They conducted experiments on HotpotQA and HoVer. The experimental results on HotpotQA show its competitive passage retrieval performance and they show state-of-the-art results on HoVer, significantly outperforming the baselines. 

main_review: **Originality.** The multi-hop retrieval and reasoning has been widely studied and considered a challenging task. Although I think the proposed approach itself is solid and technically sound, more discussions on the prior multi-hop retrieval work would be helpful. Several prior work augments search queries for multiple steps with retrieved segments from the context e.g., IRRR (Qi et al., 2020) or GOLDEN (Qi et al., 2021). Another direction is to reformulate the queries in *a latent space* (Multi-step retriever-reasoner by Das et al., 2019 or MUPPET by Feldman and El-Taniv, 2019), and they should be mentioned in the paper as well. 

**Quality.** Overall, I think the proposed approach is technically sound and well-motivated. The experimental results show its effectiveness on two multi-hop reasoning datasets. Yet, I have two concerns and questions. First, Baleen assumes that the latent ordering hop setting, where we have gold paragraph annotations but lack the annotation on its order. I think this setting relies on the two target datasets, HotpotQA and HoVer; many QA datasets come without gold paragraph annotations (e.g., TriviaQA), and relying on the existence of the gold paragraph annotations may limit the framework’s applicability to a wider set of datasets. Do you think your approach can work if we do not have gold passage annotations (e.g., heuristically decide gold passages using entity linking or something)? The second concern is on evaluation and comparison with state-of-the-art multi-hop retrievers. Although I understand the main focus of the paper isn’t on improving reader components, I would like to see at least the official supporting fact retrieval performance on HotpotQA, not only unofficial paragraph retrieval scores. The HoVer baseline (TF-IDF + BERT) is a bit weak. Is it possible to test some more recent and strong baselines like SemanticRetrievalMRS (Nie et al., 2019, https://github.com/easonnie/semanticRetrievalMRS) on HoVer? I think they study on both HotpotQA and FEVER so the models and codes should be applied to HoVer as well. 

**Clarity.** 
I think the paper is well written and easy to read. 

**Significance.** 
This paper introduces a new approach for multi-hop retrieval and is probably useful for future work in this domain. Although it advances state-of-the-art results on two datasets, I wish the evaluation results with official evaluation metrics on HotpotQA were presented. The ablation results might also provide useful insights. Heuristically constructing reasoning orders or chains have been widely used, but they show that it doesn’t perform well on complex questions.

limitations_and_societal_impact: I appreciate that the authors provide detailed discussions on the limitations and social impacts and I generally agree with the points discussed there and I enjoy reading them. Regarding the hallucinations by giant generative models, some recent retriever-generator frameworks such as Fusion-in-Decoder (Izacard and Grave, 2020) provide attention scores which help us to understand which passages are looked at by the model during answer generation, so I’m not sure if it’s often more ungrounded compared to retriever-extractor frameworks.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: **Summary**: This paper presents a new retrieval system for multi-hop QA, i.e., answering the questions requires multiple relevant documents. On top of existing dense retrieval techniques, this paper proposes three independent techniques to tackle multi-hop retrieval problems, including a topk multi-vector scoring function, condensed fact selection for query reformulation and a latent method to recover the proper retrieval order during training. This system achieves significant improvements on two recent multi-hop datasets.

**Contribution**:
1. A strong multi-hop retrieval system that outperforms existing systems by a large margin.
2. This is the first work showing that dense retrieval can be generalized to more than 2 hops. 
3. Thorough experiment results and ablations. 
main_review: Originality: This is a new system with a combinations of some existing techniques or their modifications. For example, the latent hopping order is already explored by IRRR (Qi et al.) However, I think the overall system is valuable given the nice work of combining the best practice and getting significant improvements. The related work is adequately discussed.

Quality: The experiments are solid and the proposed methods are reasonable. The authors are also very clear about the limitations of their work. For instance, the proposed system requires sentence-level evidence annotations.

Clarity: Overall, the paper is well-written and easy to follow. For further improvements, the authors could provide more details about the implementations. For example, how many sentence-level facts are appended to the current query and other hyperparameters.

Significance: The results of this paper are important and valuable. It has pushed the retrieval performance of HotpotQA. Any end-task models should be able to reuse these results. 


limitations_and_societal_impact: I don't see any negative impact of this work.
review_title: null

Meta-review: The paper propose a multi-hop reasoning system called Baleen based on the idea of iterative retrieval. It includes three main ideas: 1) condensed retrieval which summarizes the documents at each hop; focused late interaction which ranks the top-k scores and only includes those for later computation; latent hop ordering which learns to order the passages. The method is evaluated on two datasets, HotpotQA and  HoVer. Both experiments demonstrate the proposed method achieves significant better performance than baselines. 

All reviewers agree on the quality of the work.  The performance is solid. The paper is clear and easy to follow. The authors may want to address the questions raised by reviewers. In addition, the reference part is rather nonstandard. Please cite the correct source, i.e. the official publication should be cited if it is published, instead of the arxiv version. 
Decision: Accept (Spotlight)