Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
Authors: Omar Khattab
Christopher Potts
Matei Zaharia
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: neural retrieval
multi-hop question answering
claim verification
reasoning
ColBERT

Abstract: Multi-hop reasoning (i.e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge. To retrieve evidence passages, multi-hop models must contend with a fast-growing search space across the hops, represent complex queries that combine multiple information needs, and resolve ambiguity about the best order in which to hop between training passages. We tackle these problems via Baleen, a system that improves the accuracy of multi-hop retrieval while learning robustly from weak training signals in the many-hop setting. To tame the search space, we propose condensed retrieval, a pipeline that summarizes the retrieved passages after each hop into a single compact context. To model complex queries, we introduce a focused late interaction retriever that allows different parts of the same query representation to match disparate relevant passages. Lastly, to infer the hopping dependencies among unordered training passages, we devise latent hop ordering, a weak-supervision strategy in which the trained retriever itself selects the sequence of hops. We evaluate Baleen on retrieval for two-hop question answering and many-hop claim verification, establishing state-of-the-art performance.

Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval
Omar Khattab
Christopher Potts Stanford University cgpotts@stanfore edu
Matei Zaharia Stanford University matei@cs. stanford, edu
Stanford University okhattabbstanfors
Abstract
Multi-hop reasoning (i.e., reasoning across two or more documents) is key ingredient for NL.P models that leverage large corpora to exhibit broad knowledge. To retrieve evidence passages, multi-hop models must contend with fast-growing search space across the hops, represent complex queries that combine multiple information needs, and resolve ambiguity about the best order in which hop between training passages. We tackle these problems via Baleen, a system that improves the accuracy of multi-hop retrieval while learning robustly from weak training signals the many-hop setting. To tame the search space, we propose condensed retrieval, a pipeline that summarizes the retrieved passages after each hop into a single compact context. To model complex queries, we introduce a focused late interaction retriever that allows different parts of the same query representation to match disparate relevant passages. Lastly, to infer the hopping dependencies among unordered training passages, we devise latent hop ordering, a weak-supervision strategy in which the trained retriever itself selects the sequence of hops. We evaluate Baleen on retrieval for two-hop question answering and many-hop claim verification, establishing state-of- tie-hhr-rt art performance.
Introduction
In open-domain reasoning, model is tasked with retrieving evidence from a large corpus to answer questions, verify claims, or otherwise exhibit broad knowledge. In practice, such open-domain tasks often further require multi-hop reasoning, where the evidence must be extracted from two or more documents. To do this effectively, model must learn use partial evidence it retrieves to bridge its way to additional documents leading to an answer. In this vein, HotPoiQA [31] contains complex questions answerable by retrieving two passages from Wikipedia, while HoVer [3] contains claims that can only be verified (or disproven) by combining facts from up to four Wikipedia passages.
Table iillustrates this using the claim "The MVP of [a] game Red Flaherty umpired was elected to the Baseball Hall of Fame" " from the HoVer validation set. To verify this claim, HoVer Ver model must identify facts spread across three Wikipedia pages: Red Flaherty umpired in the World Series in 1955, 1958, 1965, and 1970. The MVP of the 1965 World Series was Sandy Koufax. Koufax was later elected to the Baseball Hall of Fame.
This three-hop claim illustrates three major challenges in multi-hop retrieval. First, multi-hop queries encompass multiple information needs; the claim above referenced facts from three disparate passages. Second, retrieval errors in each hop propagate directly retrieves information about the Baseball Hall of Fame, confuses Red Flaherty with, say, Robert Flaherty, or singles out the MVP of, say, the 1958 World Series, which Flaherty also umpired. Third, due to the dependency between hops, retrievers must learn an effective sequence of hops, where previously-retrieved clues lead to other relevant passages. These inter-passage dependencies
subsequent hops. This can happen if the model
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: An example (mulli-hop) claim Qo to verify and an illustration of condensed retrieval queries Qi after each hop 2 Page titles are in bold. We shorten the Wikipedia sentences for presentation.
Qu
The MVP of [a] game Red Flaherty umpired was elected the Baseball Hall of Fame.
Q1
The MVP of [a] game Red Flaherty umpired was elected to the Baseball Hall of Fame. Red Flaherty:
He umpired in World Series 1955, 1958, 1965, and 1970.
Q2 The MVP of [a] game Red Flaherty umpired was elected to the Bascball Hall of Fame. Red Flaherty; He umpired in World Series 1955, 1958, 1965, and 1970. 1965 World Series: It is remembered for MVP Sandy Koufax.
Q3 The MVP of [a] game Red Flaherty umpired was elected to the Baseball Hall of Fame. Red Flaherty: He umpired in World Series 1955, 1958, 1965, and 1970. 1965 World Series: It is remembered for MVP Sandy Koufax. Sandy Koufax: He was elected to the Baseball Hall of Fame.
can be non-obvious for many-hop problems with three more passages, and are often left unlabeled, as it can be expensive to annotate one (or every) sequence in which facts could be retrieved.
These challenges call for highly expressive query representations, robustness to retrieval errors, and scalability to many hops over massive document collections. Existing systems fall short on one or more of these criteria. For instance, many state-of-the-art systems rely bag-of-words (3] or single-vector dot-pre dot-product [29] retrievers, whose capacity to model an open-domain question is limited [17, let alone complex multi-hop queries. Furthermore, existing systems embed trade-offs when it comes to "hopping": they employ brittle greedy search, which limits recall per hop; or they employ beam search over an exponential space, which reduces scalability to many hops; or they assume explicit links that connect every passage with related entities, which ties them to link-structured corpora. Lastly, to order the hops, many systems use fragile supervision heuristics (c.g.. finding passages whose page titles appear in other passages) tailored for particular datasets like HotPotQA.
We tackle these problems with Baleen a scalable multi-hop reasoning system that improves accuracy and robustness. We introduce a condensed retrieval architecture, where the retrieved facts from each hop are summarized into short context that becomes part of the query for subsequent hops, if any (Table T.. Unlike beam search, condensed retrieval allows effective scaling to many hops, and we find that it complements greedy search (i.e., taking the best passage per hop) in improving recall considerably. We then tackle the complexity of queries by proposing a focused late interaction passage retriever (FLIPR), robust learned search model that allow different parts of the same query representation to match disparate relevant passages. FLIPP inherits the scalability of the vanilla late interaction paradigm of ColBERT [16] but uniquely allows the same query to exhibit tailored matching patterns against each target passage. Lastly, we devise latent hop ordering, a weak-supervision strategy that uses the retriever itself select effective hop paths.
We first test Baleen on the two-hop HotPotQA benchmark, finding evidence of saturation in retrieval: we achieve 96.3% answer recall in the t0p-20 retrieved passages, up from 89.4% for existing work. We then test Baleen's 's ability to scale accurately to more hops, reporting our main results using the recent many-hop Ho Ver task. We build strong many-hop baseline model that combines recent results from the open-domain question answering 17) and multi-hop 23 literatures. This baseline combines CoIBERT retrieval and standard greedy search with ELECTRA (5) re-ranker, and generalizes typical text-based heuristics to order the hops for training. After verifying its strong results on HotPotQA, we show that outperforms the official TF-IDF + BERT baseline of Ver by over 30 points in retrieval accuracy. Against this strong baseline itself, Baleen improves passage retrieval accuracy by another 17 points, raising it to over 90% at A 100 passages. Baleen also improves the evidence extraction F1 score dramatically. outperforming even the oracle retrieval + BERT results by Jiang et al. [I3] Our ablations ($5.4) show that Baleen's 's FLIPR retriever, condenser architecture, and latent supervision are essential to its strong performance.
hattps://github httpr///a/ao.tttaaoooooooottta// futuredata/Ballen
In marine biology, "baleen" refers to the filter-feeding system that baleen whales use to capture small organisms, filtering out seawater. So too does system seek to capture relevant facts from a of documents.

---

2 Background and related work
Open-domain reasoning There is significant recent interest in NLP models that can solve tasks by retrieving evidence from large corpus. The most popular such task is arguably open-domain question answering (OpenQA; Chen et al. (2), which extends the well-studied machine reading comprehension (MRC) problem over supplied question question-passage pairs to answering questions given only a large text corpus like Wikipedia. Other open- open-domain tasks span claim verification (e.g., FEVER; Thome et al. (24)), question generation (e.g., with Jeopardy as in Lewis et al. [9), and open dialogue (e.g.. Wizard of Wikipedia; Dinan et al. [8)) among others. Many of these datasets are compiled in the models for these tasks, the most relevant to our work are OpenQA models that include learned retrieval components. These include ORQA 18), REALM [10,, and DPR (15). Lewis et al. [19] introduced RAG, which by virtue of seq2seq architecture can tackle OpenQA as well as other open-domain problems, including claim verification and question generation.
KILT benchmark [21] for knowledge-intenate
tasks. Among
Multi-hop open-domain reasoning Most of the open-domain tasks from can finding one relevant passage in the corpus, often by design. In contrast, number of recent works explore multi-hop reasoning over multiple passages. These include QAngaroo (27] and 2WikiMulti- HopQA [1I) among others. While thus "multi-hop", these tasks supply the relevant passages for each example (possibly with small set of "distractor" candidates), and thus do not require retrieval from large corpus. To knowledge, HotPotQA was the first large-scale open-domain multi-hop task, particularly in its retrieval-oriented "fullwiki" setting. HotPotQA catalyzed much follow-up research. However, it is limited only two-hop questions, many of which may not require strong multi-hop reasoning capabilities (see Chen and Durrett 3 Wang et al. (26)). softening the retrieval challenges discussed in ş Very recently, Jiang et al. (13) introduced the HoVer many-hop verifica- tion dataset, which contains two-, three-, and four-hop examples. Ho contains just over 18,000 training examples, about 5x smaller than HotPotQA, adding to the challenge posed by Ho
can be solved by
Multi-hop open-domain models et al. [I] Zhao et al. (32)) assume explicit links that connect every passage with related entities. We argue that this risks tying systems to link-structured knowledge bases (like Wikipedia) or producing brittle architectures tailored for datasets constructed by following hyperlinks (like HotPorQA). Recently, Xiong et al. (29] and Qi et al. (23 introduce MDR and IRRR, ofate-of-the-art systems that assume no explicit link structure. Instead. they use an iterative retrieval paradigm paradigm-akin to that introduced by Das al. 66) and Feldman and Yaniv g that retrieves passages relevant to the question, reads these passages, and then formulates new query for another hop if necessary. We adopt this iterative formulation and tackle three major challenges for multi-hop retrieval.
To conduct the "hops", many prior multi-hop systems (e.g., Asai
ColBERT: late interaction paradigm Most learned-retrieval systems for OpenQA ($2) encode every query and passage into a single dense vector. Khattab and Zaharia [16] argue that such single- vector representations are not sufficiently expressive for retrieval many scenarios and introduce late interaction, a paradigm that represents every query and every document at finer granularity: it uses a vector for each constituent token. Within this paradigm, they propose ColBERT, a ofute-o--the-rr retriever wherein BERT encoder embeds the query into a matrix of N vectors (given N tokens) and encodes every passage in the corpus as matrix of M vectors (for M tokens per passage). The passage representations are query-independent and thus are computed offline and indexed for fast retrieval.
During retrieval with query g. ColBERT assigns a score to a passage by finding the maximum- similarity (MaxSim) score between each vector in q's representation and all the vectors of d and then summing these MaxSim scores. This MaxSim-decomppsed interaction enables scaling to massive collections with millions of passages, as the token-level passage vectors can all be indexed for fast nearest-neighbor search. Khattab and Zaharia (16) find this fine- grained matching to outperform single-vector representations, a finding extended to open- domann question answering by Khattab et al. [7]. In this work, we propose FLIPR ($3.1). a retriever with an improved late interaction mechanism for complex multi-hop queries. Whereas existing retrievers, whether traditional (e.g., TF-IDF) or neural [8] 15 16), seek passages that match all of the query, multi-hop queries can be long and noisy and need to match disparate contexts. FLIPR handles this explicitly with,fousse late interaction.
3

---

Input
Query 2-1
FLIPR Retriever
Two-Stage Condenser
t=1
Condensed Facts
Baleen T-Hop Retriever
Updated Query, Qt
Top-K Passages
FLIPR Index
Done?
No, t<T
Yes, t=
Task-Specific Reader ~> Prediction
Figure 1: The architecture of Baleen with iterative retrieval and condensing. The model begins with iteration t = 1, provided initial user input Qu. For every f, FLIPR takes in Qt-1 and retrieves the top- K passages, the condenser summarizes the relevant facts, and an updated query Qt produced. When = T, Qr is fed to the reader, which outputs final prediction.
Relevance-guided supervision Khattab et al. 17 propose an iterative strategy for weakly- supervising retrievers called relevance-guided supervision (RGS). RGS assumes that no labeled passages are supplied for training the retriever. Instead, every training question is associated with short answer string whose presence in a passage taken as weak signal for relevance. RGS starts with off- the-shelf retriever and uses to collect the top-k passages for each training question, dividing these passages into positive and negative examples based on inclusion of the short answer string. These examples used to train stronger retriever, which is then used to repeat this process one or two times. We draw inspiration from RGS when designing latent hop ordering: unlike RGS, do have gold-labeled passages for training but crucially have multiple hops, whose order is not given. Lastly, our supervision for the first hop shares inspiration with GoldEn [22]. GoldEn uses of-dords model and thus effectively term overlap to identify the "more easily retrieved" passage among just two for the two-hop task HotPotQA.
Baleen
We now introduce Baleen, which uses an iterative retrieval paradigm to find relevant facts in T 2 1 successive hops. On HotPotQA ($4) and HoVer ($5), we use Baleen with T and 4, respectively. As illustrated in Table the input Baleen is textual query Qo like question answer or a claim to verify. The goal of hop is to take in query Q.-1 and output an updated query Qt containing the initial input query and pertinent facts extracted from the & hops.
The Baleen architecture is depicted in Figure 1 In every hop, FLIPR (3.I) uses Qt-1 to retrieve K passages from the full corpus. These passages are fed into two-stage condenser ($3.3) which reads these passages and extracts the most relevant facts, which we model as individual sentences. The facts are collected into single sequence and added to Qe for the subsequent hop, if any. Once all hops are complete, Balcen's 's ssek-sccc reader processes Qr, which now contains the query and all condensed facts. solve the downstream task. If desired, the top-k passages from each hop can also be collected and fed to a downstream model after retrieval is complete. Baleen's retriever, condenser, and reader implemented Transformer encoders (25) and trained individually. We use BERT-base 77] for FLIPR and, like Qi et al. (23 and Xiong et al. (29). use ELECTRA--arge 5 for the other components. For supervision, we introduce the latent hop ordering scheme ($3.2).
3.1 FLIPR: focused late interaction
The FLIPR encoders and interaction mechanism are shown Figure Our query encoder reads Q and outputs vector representation of every token in the input. Each query embedding interacts with all passage embeddings via a MaxSim operator, permitting us to inherit the efficiency and scalability of ColBERT [16]. While vanilla ColBERT sums all the MaxSim outputs indiscriminately. FLIPR considers only the strongest-matching query embeddings for evaluating each passage: it sums only the - partial scores from N scores, with Ñ < N.
We refer to this top-k filter a "focused" interaction. Intuitively, it allows the same query to match multiple relevant passages that are contextually unrelated, by aligning-during during training and inference-a different subset of the query embeddings with different relevant passages. These

---

d
d
dM
Top-A Maxdims
Maxdim
MaxSim
Mixdim
Maxsim
FLIPR
Query Encoder
Passage Encoder
Query Takens
Facts Tokens
Passage Tokens
dyt
(a)
(b)
The proposed FLIPR retriever. Sub-figure (a) depicts encoders, which represent every query and every passage as set embeddings. Given query and document d, sub-figure (b) illustrates our focused late-interaction mechanism. It finds the maximum-similae query embedding qi and then focuses on the top-k matches only. This analogously applied to the fact embeddings (in orange), and both scores finally added.
Figure 2:
score of each
important subsets of the query are not always clear in advance. For instance, Q1 in Table TabbleDshee should find the page about the 1965 World Series. This requires ignoring many distractions (e.g.. World Series 1970 and Bascball Hall of Fame) that seem important in isolation while focusing on other phrases, namely prominent "MVP" and the World Series 1965, distinction evident only after finding the document and attempting to match against the query.
FLIPR applies this mechanism over the representation of the query and the facts separately, keeping the top-N and top-i. from N and I. embeddings. respectively. More formally, the FLIPR score Sq.d of passage given query q the summation Sq.d SQ q.d + Sq.d over the scores corresponding the original query and to the facts. Define M, max"" Eg L'j that is, the maximum similarity of the tth query embedding against the passage embeddings. The partial score Se, is computed as
N Σ top :
Sqd
where top,y is an operator that keeps the largest Ñ values in a set. We define S similarly, using the fact matches M." instead of M.O.
We confirm the gains FLIPR against the state-of-the-art ColBERT retriever $5.4 By decom- posing retrieval into token-level maximum similarities and pre-computing document representations, FLIPR maintains the scalability of the late interaction paradigm of ColBERT, reducing the candidate generation stage of retrieval into highly-efficient approximate k-nearest neighbor searches. See Appendix B for the implementation details of FLIPR.
3.2 Supervision: latent hop ordering
For every training example, datasets supply unordered gold passages. However, dependencies often exist in retrieval as Table [ [lllusstaate:: a system needs to retrieve and read the page of Red Flaherty to deduce that it needs information about 1965 World Series. Such dependencies can be complex for 3- and 4-hop examples, as in Ver, especially as multiple passages may be appropriate to retrieve together in the same hop.
We propose generic mechanism for identifying the best gold passage(s) to learn retrieve next. The key insight is that among the gold passages, weakly-supervised retriever with strong inductive biases would reliably "prefer" those passages it can more naturally retrieve, given the hops so far. Thus, given Q.-1, we train the retriever with every remaining gold passage and, among those, label as positives (for Q.-1) only those ranked highly by the model. We subsequently use these weakly-labeled positives to train another retriever for all the hops.
Latent hop ordering is summarized in Algorithm D which assumes we have already trained single- hop (i.e., first-hop) retriever Rg in the manner of relevance- guided supervision (see $ To start, we use R₁ retrieve the top-k (e.g.. lo 1000) passages for each training question (Line 2). We then divide these passages into positives Pı and negatives N1 (Line 3). Positives are the highly-ranked gold passages (i.c., the intersection of gold passages and the top-k passages with, e.g., 10) and
5

---

Algorithm 1: Latent Hop Ordering, a simplified procedure
Input: Training queries & unordered gold passages; Corpus of all passages; A single-hop retriever, Rı
Output: A multi-hop retriever,
Qu the original training queries;
2 Ranking; + R1 netrieve Qu);
Ni t ExtractPositivess Rankinng );
Q1 < @e.exaadd OracleFucts( (i)));
for hop 4- to T do
Train a new retriever R. using queries Qt-1 and negatives Nt-1. For positives, use gold passages not part of any P. for s < t.
Ranking, + R 1e
Ne Extract ExtreciPosiivess(annnnn, Q+ Qt-LLeppan(Onnceeaatssst
end
10 For every hop t E [2]. designate Qt as queries, Pt as positives, and Pt as negatives.
11 Train a final retriever R using this combined training data for all hops.
12 return R
negatives are the non-gold passages. Subsequently, expand first-hop queries with the oracle facts from P1 to obtain the queries for the second hop (Line 4).
We train second-hop retriever R₂ using the aforementioned queries and negatives (Line 6). As we do not have second-hop positives, as a form weak supervision, we train the retriever with all gold passages (per query) besides those already in Pt. Once trained, we use R2 to discover the second-hop positives Pg. to collect negatives N2. and to expand the queries (Lines and 8). We repeat this procedure for the third hop onward. Once this iterative procedure is complete, positives (and negatives) corresponding to every retrieval hop for each query. We combine these sets to train a single multi-hop retriever (Lines 10 and 1 11), which takes query Qr-1 and Icarns to discriminate the positives in P. from the negatives in N₂ for every t E [T]. In $5.4, we indeed find that this generic latent procedure outperforms typical hand-crafted heuristic that relies on trying to match titles against the text.
have bootstrapped
3.3 Condenser: per-hop fact extraction
After each hop, the condenser proceeds two stages. the first stage, it reads Q.-1 and each of the top-K passages retrieved by FLIPR. The input passages are divided into their constituent sentences and every sentence is prepended with special token. The output embedding corresponding to these sentence markers are scored via linear layer, assigning score to each sentence in every retrieved passage. We train the first-stage condenser with a cross-entropy loss over the sentences of two passages, positive and negative.
Across the K passages, the top-k facts are identified and concatenated into the second-stage condenser model. As above, we include special tokens between the sentences, and the second-stage -tagee condenser assigns score every fact while attending jointly over all of the top-k facts, allowing a direct comparison within the encoder. Every fact whose score is positive is selected proceed, and the other facts are discarded. We train the second-hop condenser over a set of 7-9 facts, some positive and others (sampled) negative, using linear combination of cross-entropy loss for each positive fact (against all negatives) and binary cross-cntrope loss for each individual fact.
In practice, this condensed architecture offers novel tradcoff's against re-ranking pipeline. On one hand, condensed retrieval scales better to more hops, as it represents the facts from K long passages using only few sentences. This may provide stronger interpretability, as the inputs to the retriever (e.g., the example Table I become more focused. On the other hand, re-ranking requires only passage-level labels, which can be cheaper to collect, and provides the retriever with broader context, which can help resolve ambiguous references. Empirically, we find the two approaches to be complementary. In $5.4. we show that condensing is competitive with re-ranking (despite much shorter context; Appendix CC and that single retriever that combines both pipelines together substantially outperforms a standard rerank-only pipeline.

---

Table 2: Retrieval results HotPotQA--ullwii (dev) showing saturation in finding the gold pair of passages and passages with the answer string. Baleen and its ablation baseline ColBERT-Hop exceed 93% P-R@ @ 20 and 94-96% Ans-Rod @ 20, largely reducing the problem to reading the retrieved passages. We obtain P-EM for MDR from Xiong et al. 29 and for IRRR [23] from the authors.
Model
Retriever
Architect.
Supervis.
P-EM
P-R@20
Ans-R@20
MDR
DPR
Beam
Heuristic Heuristic
81.2
82.9
89.4
IRRR
ELECTRA/MM2
Rerank
84.1
ColBERT-Hop (ablation)
ColBERT
Rerank
Hcuristic
85.2
90.3
94.7
Baleen
FLIPR
Condense
Latent
86.7
93.3
96.3
3.4 Reader:
task-specific processing
After all hops conducted, reader is used to extract answers verify the claim. We use standard Transformer encoder, in particular ELECTRA-large [5], which we feed the final output Qe of our multi-hop retrieval and condensing pipeline. We train for question answering similar to Khattab et al. 17 or claim verification similar to Jiang et al. [3..
4 Diagnosing retrieval saturation on HotPotQA
Before beginning our primary evaluation on the many-hop HoVer benchmark, we first investigate the capacity of HotPotQA in reflecting the challenges multi-hop retrieval ($ We use HotPotQA's "fullwiki" setting, where the input to the model question whose answer requires retrieving two passages from Wikipedia. Similar to related work (29), we report passage-leve exact-match (EM) and Recall@ (R@k) k) on the development set. These are the percentage of questions for which the model correctly identifies the two gold passages and for which the model's 's top-k retrieved passages contain both gold passages, respectively P whose short answer string is found in the top-k passages (excluding yes/no questions). Here, the top-20 passages from Baleen are simply the union of the top- 10 passages retrieved during the first hop and the top-10 10 passages from the second hop, without duplicates.
We also report Answer-Recall @ k, the percentage of questions
We compare against the state-of-the-ard models MDR and IRRR. We additionally compare with ColBERT-Hopp a strong baseline that ablates Baleen's 's retriever, condenser, and supervision: ColBERT- Hop uses the state-of-the-ar neural retriever CoIBERT (see $2). employs typical re-ranker architec- ture, and uses heuristic supervision. The results are shown in Table Baleen demonstrates strong results, finding both gold passages in over 93% of all questions and finding passages that contain the short answer string for over 96% of all questions in just the top k=20 passages. We argue that this effectively reduces the reen-retrieval task to a narrow distractor- setting one, in which reader models supplied with only few passages (ones that (nearly) guaranteed contain the answer) and are then expected to extract the answer from just those passages.
In other words, such high recall largely shifts the competition in downstream quality to the reader, since almost always has access to the answer in the top-k set and is thus responsible for the majority of downstream failures. While reading the retrieved passages may still pose additional, important challenges beyond retrieval that remain open, should be just one of the many challenging aspects of multi-hop reasoning at scale. This motivates our focus on the many-hop Ho Ver task.
Main evaluation on Hover
We now conduct our primary evaluation, applying Baleen on the many-hop claim verification dataset HoVer, following the evaluation protocol of Jiang al. [13. First, study the retrieval performance of Baleen in $5.2 Subsequently ($5.3), we investigate its sentence extraction and claim verification performance, which constitutes test of overall end-to-end system effectiveness. We compare
For MDR, we use the authors released retrieval to compute P-R220. The authors originally report 80.2, but do not penalize the MDR retriever for finding both gold passages split across different pairs.
7

---

Table 3: Our main passage retrieval results HoVer. To compare with Jiang al. 13 the Retrieval 100 results are for supported claims. The other results are for all claims. "Marked result are from Jiang et al. The table reports fine- ene-grained dev-set scores, and we submitted to the organizers and obtained 64.6 Psg-EM and 88.9 Psg-F1 for Baleen on the held-out test set.
Retrieval 100
Passage EM
Passage F1
Model Hops
3
All
2
4
TF-IDF + BERT*
44.6
80.0
39.2
15.6
12.5
34.0
5.8
1.0
60.2
69.9
58.2
53.4
ColBERT Hop
74.8
95.8
77.9
47.6
Baleen
92.2
97.7
93.1
85.1
63.6
75.8
62.5
52.6
89.2
90.2
89.9
86.8
Oracle BERT*
34.0
50.9
28.1
26.2
80.6
81.7
79.1
82.2
Human
77.0
85.0
82,4
65.8
93.5
92.5
95.3
91.4
four-hop Baleen system against the official baseline and the strong ColBERT-Hop baseline from $4 See Appendix [B for implementation details and hyperparameters.
5.1 Task description
The input the model is claim, which is often one or few sentences long. The model is to attempt to verify this claim, outputting "Supported" if the corpus contains enough evidence to verify the claim, or otherwise "Unsupported" if the corpus contradicts- or otherwise fails to support-thhs claim. Alongside this binary label, the model must extract facts (i.e., sentences) that justify its prediction. In particular, HoVer contains two-, three-, and four-hop claims that require facts from 2-4 different Wikipedia pages, and the model must find and extract one more supporting facts from each of these pages for every claim. Models are not given the number of hops per claim.
5.2 Evaluating retrieval effectiveness
Table the retrieval and passage extraction quality of the systems. We report three metrics, each sub-divided by the number of hops of each claim. The first is Retrieval@ 100, which is the percentage of claims for which the system retrieves all of the relevant passages within the top-100 results. To compare with Jiang et al. [13, we report the Retrieval @ 100 results only for supported claims. All other metrics are reported for all claims. The second metric is Passage EM, which the percentage of claims for which the system can provide the exact set of relevant passages. The third metric Passage FI, which uses the standard Fl measure offer a relaxed version of EM.
At the top of the table, the TF-IDF bascline retrieves top-100 results in one round of retrieval. At the bottom of the table, include the results reported by Jiang et al. 133 for their BERT ranking model when supplied with oracle retrieval, and also their reported human performance. We show Baleen's performance after four hops, where Baleen's 's FLIPR retrieves 25 passages in each hop. Note that we exclude passages retrieved in hop / from the retrieval in further hops l' > L. As the table shows, Baleen achieves strong results, outperforming the official baseline model by 47.6 points in Retrieval @ 1 100, 51.1 points in Passage EM, and 29.0 points in Passage F1. In fact, Baleen's performance also consistently exceeds "Oracle BERT" passage extraction.
5.3 Evaluating end-to-end effectiveness
Next, we evaluate the performance of fact/sentence extraction and thereby the end-to-end claim verification of Baleen. The results are Table 4 where Sentence EM and Sentence Fl are defined similar to the corresponding metrics in Table but focus on sentence-level extraction. Moreover, Verification Accuracy the percentage of claims that are labelled correctly as "Supported" or "Unsupported". In Table 4 we that Baleen's 's sentence-level results mirror its document-level results, with large performance gains across the board against the baseline model. Balcen also outperforms the oracle-retrieval BERT model of Jiang et al. [3], emphasizing the strong retrieval
At the time of writing, MDR's s open-source implementation only supports two-hop retrieval and the IRRR implementation is not publicly available. Moreover, both Xiong et al. (29) and Qi et al. (23) explore training only with swo gold passages, though IRRR includes evaluation yet- unreleased 3-hop test set.

---

Table 4: Our main sentence extraction results Ho "Marked results from Jiang et al. [13 The table reports fine- tineggrined results on the dev set and, where available, includes "dev/test" scores separated by slash for the held-out test score. In test evaluation the Ho leaderboard, Baleen increases the overall "HoVer Score" metric from 15.3 for the baseline to 57.5.
Sentence
EM
Sentence
Verification
Model # of Hops
All
2
3
4
All
2
4
Accuracy
TF-IDF + BERT*
4.8/4.5
13.6
1.9
0.2
50.6/49.5
57.2
49.8
45.0
73.7
Baleen 1-hop
40.9
15.4
4.3
72.3
77.5
72.4
Baleen 2-hop
37.0
46.9
35.7
28.4
80.8
81.2
81.8
78.7
Baleen 3-hop
38.9
47.1
37.0
33.2
81.4
81.2
82.3
80.0
Baleen 4-hop
39.2/39.8
47.3
37.7
33.3
81.5/80.4
81.2
82.5
80.0
84.5/84.9
Oracle BERT"
19.9
25.0
18.4
17.1
71.9
68.3
71.5
76.4
81.2
Human
56.0
75.0
73.5
42.1
88.7
86.5
93.1
87.3
88.0
Table 5: Retrieval @ 100 comparison between Baleen and five ablation settings.
Model
All
2
3
4
Architecture Ablations
BalcenconDENSe (main architecture)
92.2
97.7
93.1
85.1
BalсeDиERANK
91.2
98.7
93.0
80.0
BalсeneY
94.5
99.2
94.4
90.0
Retrieval Ablations (over arch. A)
D
w/o FLIPR (uses ColBERT retrieval modeling)
87.4
97.3
89.5
73.4
w/o I.HO (uses a recursive httc/ppssage-oveett heuristic for ordering)
84.8
97.1
88.3
65.6
Full Model Ablation (over arch.
ColBERT-Hoo (wlo FLIPR, LHO))
74.8
95.8
77.9
47.6
and condensing performance Baleen. the leaderboard test-set evaluation (scores expected to be
posted soon), Baleen increases the overall "HoVer Score" metric from 15.3 for the baseline to 57.5.
5.4 Ablation studies
Lastly, Table reportt various settings and ablations of Baleen, corresponding to its contributions. Here, models A and F are simply the four-hop Baleen and ColBERT-Hop the previous experiments, respectively. Models B and C test the effect of different architectural decisions on Baleen. In particular, model replaces our condensed retrieval architecture with simpler re-ranking architecture (Appendix {B.2). which reflects common design choice for multi-hop systems. In this case, FLIPR given the top-ranked passage as context each hop, helping expose the value of extracting facts in condensed retrieval, which exhibits much shorter contexts as we show in Appendix $C Second, model trains a FLIPR retriever that uses both a condenser pipeline and a reranking pipeline independently. and then retains the overall top- 100 unique passages retrieved (Appendix $B.2). The results suggest that condensing offers complimentary gains, and the hybrid model attains the highest scores.
Moreover, models D and E test modifications to the retriever, while using the condenser of model A. Specifically, model D replaces our FLIPR retriever with simpler ColBERT retriever, while retaining all other components of our architecture. This allows us to contrast our proposed focused late interaction with the "vanilla" late interaction ColBERT. Further, model E replaces our weak- supervision strategy for inferring effective hop orders with a hand-crafted rule (Appendix (B.3) that deliberately exploits construction bias HotPotQA and Ho Ver, namely that passage titles can offer strong signal for hop ordering.

---

Discussion
Research limitations This work investigates multi-hop reasoning at a large scale by focusing on the challenges presented by retrieval in this setting. While reading the retrieved (or supplied) passages to solve a downstream task is another important aspect, one that has received more attention in the literature, we do not attempt to advance the state of the art approaches for multi-hop reader models. The majority of multi-hop retrieval work to date has focused on two-hop retrieval. We leverage the recently-released Ho dataset scale our investigation four retrieval hops, and Baleen allows for an arbitrary number of hops in principle, but more evaluation is needed before can claim generality to arbitrary number of hops. We propose condenser architecture that summarizes contexts extractively. This architectural decision allows us to scale easily to many hops as it condenses the hop information into a short context, but doing so relies on the availability on sentence-level training information (as in HotPotQA and Ho and extracting sentences might in principle lose important context. We expect future work to be able to infer the sentence-level labels in latent manner and we are excited about work like decontextuallizttonn 4 that tries to retain important context for standalone sentences. Lastly, considering that learning neural retrievers for large-scale many-hop tasks is recently- emerging topic, we had to design our own strong baseline to strengthen our Ho Ver evaluation, after confirming that our baseline was very strong on HotPotQA.
Environmental and societal impact By turning to retrieval from text corpora (see 2 as a mecha- nism to find and use knowledge, we seek to build scalable and efficient models that can reason and solve downstream tasks with concrete, checkable provenance from text sources and without growing the models' number of trainable parameters dramatically. We use Wikipedia, which has favorable licensing (generally under CC BY-SA 3.0), and publicly-released datasets Ho and HotPotQA (CC BYSSA 4.0 licenses). HotPorQA and Ho are crowd-sourced, and we expect models trained on them reflect the biases of the underlying data. Morcover, automated systems for answering questions or verifying claims can have misleading outputs or may even be abused. That said, we believe that focusing on retrieval and extractive models as a way to help users find information can offer net gains to society, especially in contrast with large generative models that may hallucinate or make statements ungrounded in a human-written corpus.
7 Conclusion
In this paper, we propose Baleen, a system for multi-hop reasoning that tackles the complexity of multi-hop queries with the focused late interaction mechanism for retrieval and mitigates the exponential search space problem by employing an aggressive condensed retrieval pipeline in every hop, which consolidates the pertinent information retrieved throughout the hops into relatively short context while preserving high recall. Moreover, Baleen deals with the difficulty of ordering passages for many-hop supervision via latent hop ordering. By employing a strong retriever, incorporating a condenser, and avoiding brittle heuristics, Baleen can robustly learn from limited training signal. We evaluated Baleen's retrieval on HotPoiQA and the recent many-hop Ho claim verification dataset and found that it greatly outperforms the baseline models.
Acknowledgments and Disclosure of Funding
We would like to thank Ashwin Paranjape, Megha Srivastava, and Ethan A. Chi for valuable discus- sions and feedback. This research was supported in part by affiliate members and other supporters of the Stanford DAWN project-Ant Financial, Facebook, Google, and VMware- a Stanford HAI Cloud Credit grant from AWS, and the NSF under CAREER grant CNS-165155 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
well as Cisco, SAP,
References
[1] Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learning to retrieve reasoning paths over wikipedia graph for question answering. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia,
10

---

April 26-30, 2020. OpenReview.net, 2020. URL http:://geenrrrro...ttttttoo SJgVHKrYDH
[2] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computatioonll Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada, URL https
2017. Association for Computational Linguistics. doi: /acclattoology .wwpggg...rg/PP wwtg///-///77
[3] Jifan Chen and Greg Durrett. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1026-40332 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 108653//11/11-1110 URL
[4] Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461, 2021.
[5] Kevin Clark, Minh- Thang Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://opeereviier wwt/teruuiiidrreeeeeeeee XMM1BBBB
[6] Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. Multi-step retriever-reader interaction for scalable open-domain question answering. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httLtttss///poereviiie http://wwwwwweettteeeeeeee wwt//eteturnneee d HkfFPSh00KK
[7] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41 Minneapolis, Minnesota. 2019. Association for Computational Linguistics. doi: 10.18653/ 1/1655//11/11112223 URL httpp::/aaclaathologgg...o wrt/N19-1/229
[8] Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Con- ference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL http:ttts///pporrriiai wttp://wwww..eeettere..eeeeeeee.
[9] Yair Feldman and Ran El-Yaniv. Multi-hop paragraph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2296 22099, Florence, Italy, 2019. Association for Computational Linguisties. doi: 10.18653/v1/P19- 1222.
1222
[10] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval-augmmented language model pre-training. arXiv preprint arXiv:2002. 08909, 2020.
htttps://a/x org/ abs///b///// 10000.089000
[11] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10.186534 v1/2020.coling-maii maiee-aainnn.... URL
org/ 2020.020..0 col coling main. 580
[12) Gautier Izacard, Fabio Petroni, I Lucas Hosseini, Nicola De Cao, Sehastian Riedel, and Edouard Grave. memory efficient baseline for open domain question answering. arXiv preprint S.kv:200/2/1/556 2020. URL httttssss///xiii http://rxxi...or/a//////a/// 2012. 15156
[13] Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoVer: A dataset for many-hop fact extraction and claim verification. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3441-3460, Online, 2020.
11

---

Association for Computational Linguistics. doi: 10. 10653//111/2200 10.1865////2000.mmmnnngs. www.iingseeenl...ai.. emnlp.309. URL httpp://acllarthologyy anddg//.0o/..2....nnnng.... wtw.o/oooiiniinggeme......... 309
[14] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus, IEEE Trunsuctions on Big Data, 2019.
[15] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, 2020. Association for Computational Linguistics. doi: 10. URL https: littps://aclantee httpps//acclaathoolog// org 202000000.... momlp-main
550
[16]
Efficient and effective passage search via contex- tualized late interaction over BERT. In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Vir- tual Event, China, July 25-30, 2020, pages 39-48. ACM, 2020. doi: 10.1145/3377271.34000 URL https://doi. org/ 10. 1145/3397271.... 3401075
Omar Khattab and Matei Zaharia. ColBERT:
[17] Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guides supervision for openqa with ColBERT. Transactions of the Association for Computational Linguistics, 9:929-944, 2021.
[18] Kenton Lee, Ming-War Wei Chang, and Kristina Toutanova. Latent retrieval for weakly su- pervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6099 Florence, Italy, 2019, Association for Computational Linguistics. doi: 10.18663/11/11-16122 URL https
7 //aclanthology.. hwwphhoooogyooggg///.. www.:gg...g....1110 Pttg/P19-1612
[19] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmente- generation for knowledge-intensie tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- -ien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12. 2020, virtual, 2020. URL https: p/pooeedi/// neurips cc cc/papep//ppppppppppcccccccce. 22200/222hhhha/ Att.eeet---bttrt..ttaa
NLP
[20] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A numan-generater MAchine reading COmprehension dataset. arXiv preprint arXiv: /6/ 1. 100268, 2016. URL http::ttt//////t/// www.eriivv..rgr./ooo org/ 1611. abs/1611.09268 09268
[21] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online, 2021. Association for Computational Linguistics. doi: 10. 10.118533vv120222 maacc-mainnm2m URL https://aclanthonn
org/ 2021
maain.200
[22] Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. Answering complex open-domain questions through iterative query generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Hong
Joint Conference on Natural Language Processing GEMNLP-IJCNLE pages Kong, China, 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19-1122 URI. U htap::::///claahhloggoooooo org/
[23] Peng Qi, Haejun Lee, Oghenetegiri Sido, Christopher D Manning, et al. Retrieve, rerank, read, then iterate: Answering open-domain questions of arbitrary complexity from text. arXiv preprint 20X::22010.2252,, 2020. URL htthhttp::. arxiv. org/ abs///bs/000.... 12527
12

---

[24] James Thome, Andreas Vlachos, Christos Christodoulopoulls. and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human I. Language Technologies, Volume / (Long Pupers), pages 809-819, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 1118653/11/N1811000 URL https / /occlanthooogy.org/gleooo
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL httpp::// Abstract.
neur www.eipp...pppppeeeeeeee pctpaappr//1///aaaal
[26] Haoyu Wang, Mo Yu, Xiaoxiao Guo, Rajarshi Das, Wenhan Xiong, and Tian Gao. Do multi-hop readers dream of reasoning chains? Proceedings of the 2nd Workshop on Machine Reading for Question Answering. pages 91-97, Hong Kong, China, 2019. Association for Computational Linguistics, doi: 10. 10.11663////DD1955000 1/D19-5813. URL URL htttt::::aaaanttaaataa
org 5813
[27] Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302, 2018. doi: 1..166/taclaa_0000 URL pestps://aclantain aclanthology
org 2rg/Q18-100 1021
(28) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, 2020. Association for Computational Linguistics. doi: 10018663//170220.emmm-ddddee https://aclanthooogg
URL
/020. mmllp-dmmos_. 6
[29] Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, et al. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint arXiv:2009. 12756 2020.
URL (tttps:::: :/ap:::///rxxiii /arxiv...vvv org/ abs absp///........... 2009 12756
[30] Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. Efficient passage retrieval with hash- ing for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). pages 979-986, Online, 2021. Association for Computational Linguistics. doi: 10. 10.1865//1120221.aaa accl-short. 123. URL hnttps//acclatthll.
org/oog///// attp:/222..cllshheeo acl-short....... 123
[31] Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Mamning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2369-2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 116553vv//DLL 1259. URL / http///ccaattooooy..oooo www//og----... 1259
[32] Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul N. Bennett, and Saurabh Tiwary. Transformer-xh: Multi-evidence reasoning with extra hop attention. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. htRpss/tss:// httpp:://oppprrreeeeeee//////aap
13


Reviews:
[{"review_id": "kSkm3TxUS_m", "reviewer": "Reviewer_EV88", "summary": "This paper proposed Baleen a system that improves the accuracy of multi-hop retrieval while learning robustly from weak training signals in the many-hop setting. Baleen generally contains three modules -- Focused Late Interaction, Latent Hop Ordering, and Condenser. Focused Late Interaction is an improved version of ColBERT, in which they only consider $\\text{top-}\\hat{K}$ partial scores instead of all scores. Latent Hop Ordering proposed a way to supervise the retriever using weak supervision per-hop. The condenser summaries the sentence-level information retrieved per-hop so it could be better used for the next hop. Experimental results on two multi-hop-based datasets, namely HotpotQA and HoVer show that the proposed framework is quite effective for retrieving multi-hop evidence. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The authors implemented a multi-hop retrieval system that achieved very strong performance on two multi-hop-based datasets.\n2. The idea behind the proposed modules is simple yet effective.\n3. Reasonable ablations showing the effectiveness of each module.\n\nWeaknesses:\nAlthough the system proposed in this paper is quite strong, there are not many insights I can draw from the paper as many of the proposed architecture changes are heuristic-based, e.g., taking $\\text{top-}\\hat{K}$ partial scores is straightforward but how do we choose $\\hat{K}$? The idea of using a condenser is also quite intuitive, but how do we choose the negatives and how many negatives are needed? As we can see in the appendix, there are so many hyperparameters in the appendix and I am not sure to what extent those hyperparameters would affect the performance. \n\nQuestion for the authors:\n1. Compared to some of the previous literature, e.g., IRRR, DPR, the proposed framework seems much more computationally expensive; do you have an idea of how slow the proposed method is compared to those models for both training and inference?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have sufficiently addressed the limitations of their work and I don't think there is a potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5"}, "extra_text": {}}, {"review_id": "S41zqgA9Ax6", "reviewer": "Reviewer_EkjX", "summary": "**Summary**: This paper presents a new retrieval system for multi-hop QA, i.e., answering the questions requires multiple relevant documents. On top of existing dense retrieval techniques, this paper proposes three independent techniques to tackle multi-hop retrieval problems, including a topk multi-vector scoring function, condensed fact selection for query reformulation and a latent method to recover the proper retrieval order during training. This system achieves significant improvements on two recent multi-hop datasets.\n\n**Contribution**:\n1. A strong multi-hop retrieval system that outperforms existing systems by a large margin.\n2. This is the first work showing that dense retrieval can be generalized to more than 2 hops. \n3. Thorough experiment results and ablations. ", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This is a new system with a combinations of some existing techniques or their modifications. For example, the latent hopping order is already explored by IRRR (Qi et al.) However, I think the overall system is valuable given the nice work of combining the best practice and getting significant improvements. The related work is adequately discussed.\n\nQuality: The experiments are solid and the proposed methods are reasonable. The authors are also very clear about the limitations of their work. For instance, the proposed system requires sentence-level evidence annotations.\n\nClarity: Overall, the paper is well-written and easy to follow. For further improvements, the authors could provide more details about the implementations. For example, how many sentence-level facts are appended to the current query and other hyperparameters.\n\nSignificance: The results of this paper are important and valuable. It has pushed the retrieval performance of HotpotQA. Any end-task models should be able to reuse these results. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any negative impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "PJKyfUHiRy5", "reviewer": "Reviewer_ZzDs", "summary": "The paper makes contributions to passage retrieval in the open-domain, multi-hop setting.  It builds on recent work which applied dense passage retrieval iteratively to work for multi-hop tasks (such as MDR and IRRR).  It proposes three modifications to existing work: condensed retrieval, which summarizes retrieved passages at each hop to reduce the search space; focused late interaction, which is a variation on Colbert to consider only top-k scores for each query embedding; and latent hop ordering, which is a learned strategy to order the passages which improves over existing heuristics.  Each of these modifications are relatively minor, but together lead to very strong results on the often used HotpotQA and the newly released HoVer benchmarks.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper builds on recent advances in multi-hop retrieval using dense representations.  The contributions are not ground-breaking, but clear.  The ideas are well executed, with strong results and clearly presented.  Overall this is a solid candidate for publication at NeurIPS.\n\nOriginality:  The contributions consist of 3 improvements to multi-hop dense retrieval.  2 of these (focused late interaction and latent hop ordering) are very minor and can be considered incremental.  The remaining one, condensed retrieval, is a neat and novel idea which incorporates summarization into multi-hop retrieval.  All three contributions are clear, well executed and clearly demonstrated to improve performance.  The end result is a much better, 'polished' system for multi-hop dense retrieval.\n\nSignificance:  Unfortunately 'multi-hop reasoning' (despite the exciting name) suffers from a lack of natural benchmarks and real-world applications, which somewhat limits the significance of advances in this area.  Nevertheless, this is not a weakness of the paper.  The results are strong, and the methods are clear and simple enough for others to adopt.  \n\nQuality:  The experiments are conducted against very strong baselines, and the improvements are convincing.  There are enough ablations to demonstrate the contribution of each component.  \n\nClarity:  The presentation is clear and generally easy to follow.  The authors make it very clear what the current SotA is and exactly what their contributions are.\n\nQuestions/feedback:\n- What is the criteria for claiming \"saturation\" for HotpotQA?  Given no human upper-bound, it would be helpful to make this section more formal.  While many are eager to 'retire' HotpotQA, I don't think simply attaining 90+ accuracy is sufficient to make the claim.  One suggestion could be to look at the remaining examples, and show they are somehow irrelevant (e.g. bad annotation, ambiguous question etc.)\n- What does top-20 passage recall mean exactly when condensing passages?\n- Some of the important implementation / training details can be moved to the main text from the appendix.  Currently the manuscript is pretty light in details (if one doesn't read the appendix).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "084YS9IUVIV", "reviewer": "Reviewer_buZr", "summary": "This paper introduces a new multi-hop reasoning system called Baleen which conducts iterative retrieval and condensing. Baleen repeats the iterative process for pre-defined time steps T (e.g., T=2 for HotpotQA and T=4 for HoVer)  and at every time step, it takes the query from the previous time step and retriever top K passages, and create a new query for the current t step by extracting and condensing the facts (sentences) from the retrieved passages. When they finish retrieval (i.e., t=T), they feed the final query to the reader model to get the final predictions. \nThey conducted experiments on HotpotQA and HoVer. The experimental results on HotpotQA show its competitive passage retrieval performance and they show state-of-the-art results on HoVer, significantly outperforming the baselines. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality.** The multi-hop retrieval and reasoning has been widely studied and considered a challenging task. Although I think the proposed approach itself is solid and technically sound, more discussions on the prior multi-hop retrieval work would be helpful. Several prior work augments search queries for multiple steps with retrieved segments from the context e.g., IRRR (Qi et al., 2020) or GOLDEN (Qi et al., 2021). Another direction is to reformulate the queries in *a latent space* (Multi-step retriever-reasoner by Das et al., 2019 or MUPPET by Feldman and El-Taniv, 2019), and they should be mentioned in the paper as well. \n\n**Quality.** Overall, I think the proposed approach is technically sound and well-motivated. The experimental results show its effectiveness on two multi-hop reasoning datasets. Yet, I have two concerns and questions. First, Baleen assumes that the latent ordering hop setting, where we have gold paragraph annotations but lack the annotation on its order. I think this setting relies on the two target datasets, HotpotQA and HoVer; many QA datasets come without gold paragraph annotations (e.g., TriviaQA), and relying on the existence of the gold paragraph annotations may limit the framework’s applicability to a wider set of datasets. Do you think your approach can work if we do not have gold passage annotations (e.g., heuristically decide gold passages using entity linking or something)? The second concern is on evaluation and comparison with state-of-the-art multi-hop retrievers. Although I understand the main focus of the paper isn’t on improving reader components, I would like to see at least the official supporting fact retrieval performance on HotpotQA, not only unofficial paragraph retrieval scores. The HoVer baseline (TF-IDF + BERT) is a bit weak. Is it possible to test some more recent and strong baselines like SemanticRetrievalMRS (Nie et al., 2019, https://github.com/easonnie/semanticRetrievalMRS) on HoVer? I think they study on both HotpotQA and FEVER so the models and codes should be applied to HoVer as well. \n\n**Clarity.** \nI think the paper is well written and easy to read. \n\n**Significance.** \nThis paper introduces a new approach for multi-hop retrieval and is probably useful for future work in this domain. Although it advances state-of-the-art results on two datasets, I wish the evaluation results with official evaluation metrics on HotpotQA were presented. The ablation results might also provide useful insights. Heuristically constructing reasoning orders or chains have been widely used, but they show that it doesn’t perform well on complex questions.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciate that the authors provide detailed discussions on the limitations and social impacts and I generally agree with the points discussed there and I enjoy reading them. Regarding the hallucinations by giant generative models, some recent retriever-generator frameworks such as Fusion-in-Decoder (Izacard and Grave, 2020) provide attention scores which help us to understand which passages are looked at by the model during answer generation, so I’m not sure if it’s often more ungrounded compared to retriever-extractor frameworks. ", "review_text": "", "extra_scores": {"ethical_concerns": "I don't think the paper has any major ethical issues. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper propose a multi-hop reasoning system called Baleen based on the idea of iterative retrieval. It includes three main ideas: 1) condensed retrieval which summarizes the documents at each hop; focused late interaction which ranks the top-k scores and only includes those for later computation; latent hop ordering which learns to order the passages. The method is evaluated on two datasets, HotpotQA and  HoVer. Both experiments demonstrate the proposed method achieves significant better performance than baselines. 

All reviewers agree on the quality of the work.  The performance is solid. The paper is clear and easy to follow. The authors may want to address the questions raised by reviewers. In addition, the reference part is rather nonstandard. Please cite the correct source, i.e. the official publication should be cited if it is published, instead of the arxiv version. 
Decision: Accept (Spotlight)