Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval

1 Introduction . In open-domain reasoning , a model is tasked with retrieving evidence from a large corpus to answer questions , verify claims , or otherwise exhibit broad knowledge . In practice , such open-domain tasks often further require multi-hop reasoning , where the evidence must be extracted from two or more documents . To do this effectively , a model must learn to use partial evidence it retrieves to bridge its way to additional documents leading to an answer . In this vein , HotPotQA [ 31 ] contains complex questions answerable by retrieving two passages from Wikipedia , while HoVer [ 13 ] contains claims that can only be verified ( or disproven ) by combining facts from up to four Wikipedia passages . Table 1 illustrates this using the claim “ The MVP of [ a ] game Red Flaherty umpired was elected to the Baseball Hall of Fame ” from the HoVer validation set . To verify this claim , a HoVer model must identify facts spread across three Wikipedia pages : Red Flaherty umpired in the World Series in 1955 , 1958 , 1965 , and 1970 . The MVP of the 1965 World Series was Sandy Koufax . Koufax was later elected to the Baseball Hall of Fame . This three-hop claim illustrates three major challenges in multi-hop retrieval . First , multi-hop queries encompass multiple information needs ; the claim above referenced facts from three disparate passages . Second , retrieval errors in each hop propagate to subsequent hops . This can happen if the model directly retrieves information about the Baseball Hall of Fame , confuses Red Flaherty with , say , Robert Flaherty , or singles out the MVP of , say , the 1958 World Series , which Flaherty also umpired . Third , due to the dependency between hops , retrievers must learn an effective sequence of hops , where previously-retrieved clues lead to other relevant passages . These inter-passage dependencies 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can be non-obvious for many-hop problems with three or more passages , and are often left unlabeled , as it can be expensive to annotate one ( or every ) sequence in which facts could be retrieved . These challenges call for highly expressive query representations , robustness to retrieval errors , and scalability to many hops over massive document collections . Existing systems fall short on one or more of these criteria . For instance , many state-of-the-art systems rely on bag-of-words [ 23 ] or single-vector dot-product [ 29 ] retrievers , whose capacity to model an open-domain question is limited [ 17 ] , let alone complex multi-hop queries . Furthermore , existing systems embed trade-offs when it comes to “ hopping ” : they employ brittle greedy search , which limits recall per hop ; or they employ beam search over an exponential space , which reduces scalability to many hops ; or they assume explicit links that connect every passage with related entities , which ties them to link-structured corpora . Lastly , to order the hops , many systems use fragile supervision heuristics ( e.g. , finding passages whose page titles appear in other passages ) tailored for particular datasets like HotPotQA . We tackle these problems with Baleen,1 a scalable multi-hop reasoning system that improves accuracy and robustness . We introduce a condensed retrieval architecture , where the retrieved facts from each hop are summarized into a short context that becomes a part of the query for subsequent hops , if any ( Table 1 ) . Unlike beam search , condensed retrieval allows effective scaling to many hops , and we find that it complements greedy search ( i.e. , taking the best passage per hop ) in improving recall considerably . We then tackle the complexity of queries by proposing a focused late interaction passage retriever ( FLIPR ) , a robust learned search model that allow different parts of the same query representation to match disparate relevant passages . FLIPR inherits the scalability of the vanilla late interaction paradigm of ColBERT [ 16 ] but uniquely allows the same query to exhibit tailored matching patterns against each target passage . Lastly , we devise latent hop ordering , a weak-supervision strategy that uses the retriever itself to select effective hop paths . We first test Baleen on the two-hop HotPotQA benchmark , finding evidence of saturation in retrieval : we achieve 96.3 % answer recall in the top-20 retrieved passages , up from 89.4 % for existing work . We then test Baleen ’ s ability to scale accurately to more hops , reporting our main results using the recent many-hop HoVer task . We build a strong many-hop baseline model that combines recent results from the open-domain question answering [ 17 ] and multi-hop [ 23 ] literatures . This baseline combines ColBERT retrieval and standard greedy search with an ELECTRA [ 5 ] re-ranker , and generalizes typical text-based heuristics to order the hops for training . After verifying its strong results on HotPotQA , we show that it outperforms the official TF-IDF + BERT baseline of HoVer by over 30 points in retrieval accuracy . Against this strong baseline itself , Baleen improves passage retrieval accuracy by another 17 points , raising it to over 90 % at k = 100 passages . Baleen also improves the evidence extraction F1 score dramatically , outperforming even the oracle retrieval + BERT results by Jiang et al . [ 13 ] . Our ablations ( §5.4 ) show that Baleen ’ s FLIPR retriever , condenser architecture , and latent supervision are essential to its strong performance . 1https : //github.com/stanford-futuredata/Baleen In marine biology , “ baleen ” refers to the filter-feeding system that baleen whales use to capture small organisms , filtering out seawater . So too does our system seek to capture relevant facts from a sea of documents . 2 Background and related work . Open-domain reasoning There is significant recent interest in NLP models that can solve tasks by retrieving evidence from a large corpus . The most popular such task is arguably open-domain question answering ( OpenQA ; Chen et al . [ 2 ] ) , which extends the well-studied machine reading comprehension ( MRC ) problem over supplied question–passage pairs to answering questions given only a large text corpus like Wikipedia . Other open-domain tasks span claim verification ( e.g. , FEVER ; Thorne et al . [ 24 ] ) , question generation ( e.g. , with Jeopardy as in Lewis et al . [ 19 ] ) , and open dialogue ( e.g. , Wizard of Wikipedia ; Dinan et al . [ 8 ] ) , among others . Many of these datasets are compiled in the recently-introduced KILT benchmark [ 21 ] for knowledge-intensive tasks . Among models for these tasks , the most relevant to our work are OpenQA models that include learned retrieval components . These include ORQA [ 18 ] , REALM [ 10 ] , and DPR [ 15 ] . Lewis et al . [ 19 ] introduced RAG , which by virtue of a seq2seq architecture can tackle OpenQA as well as other open-domain problems , including claim verification and question generation . Multi-hop open-domain reasoning Most of the open-domain tasks from §2 can be solved by finding one relevant passage in the corpus , often by design . In contrast , a number of recent works explore multi-hop reasoning over multiple passages . These include QAngaroo [ 27 ] and 2WikiMultiHopQA [ 11 ] , among others . While thus “ multi-hop ” , these tasks supply the relevant passages for each example ( possibly with small set of “ distractor ” candidates ) , and thus do not require retrieval from a large corpus . To our knowledge , HotPotQA was the first large-scale open-domain multi-hop task , particularly in its retrieval-oriented “ fullwiki ” setting . HotPotQA catalyzed much follow-up research . However , it is limited to only two-hop questions , many of which may not require strong multi-hop reasoning capabilities ( see Chen and Durrett [ 3 ] ; Wang et al . [ 26 ] ) , softening the retrieval challenges discussed in §1 . Very recently , Jiang et al . [ 13 ] introduced the HoVer many-hop verification dataset , which contains two- , three- , and four-hop examples . HoVer contains just over 18,000 training examples , about 5× smaller than HotPotQA , adding to the challenge posed by HoVer . Multi-hop open-domain models To conduct the “ hops ” , many prior multi-hop systems ( e.g. , Asai et al . [ 1 ] ; Zhao et al . [ 32 ] ) assume explicit links that connect every passage with related entities . We argue that this risks tying systems to link-structured knowledge bases ( like Wikipedia ) or producing brittle architectures tailored for datasets constructed by following hyperlinks ( like HotPotQA ) . Recently , Xiong et al . [ 29 ] and Qi et al . [ 23 ] introduce MDR and IRRR , state-of-the-art systems that assume no explicit link structure . Instead , they use an iterative retrieval paradigm—akin to that introduced by Das et al . [ 6 ] and Feldman and El-Yaniv [ 9 ] —that retrieves passages relevant to the question , reads these passages , and then formulates a new query for another hop if necessary . We adopt this iterative formulation and tackle three major challenges for multi-hop retrieval . ColBERT : late interaction paradigm Most learned-retrieval systems for OpenQA ( §2 ) encode every query and passage into a single dense vector . Khattab and Zaharia [ 16 ] argue that such singlevector representations are not sufficiently expressive for retrieval in many scenarios and introduce late interaction , a paradigm that represents every query and every document at a finer granularity : it uses a vector for each constituent token . Within this paradigm , they propose ColBERT , a state-of-the-art retriever wherein a BERT encoder embeds the query into a matrix of N vectors ( given N tokens ) and encodes every passage in the corpus as matrix of M vectors ( for M tokens per passage ) . The passage representations are query-independent and thus are computed offline and indexed for fast retrieval . During retrieval with query q , ColBERT assigns a score to a passage d by finding the maximumsimilarity ( MaxSim ) score between each vector in q ’ s representation and all the vectors of d and then summing these MaxSim scores . This MaxSim-decomposed interaction enables scaling to massive collections with millions of passages , as the token-level passage vectors can all be indexed for fast nearest-neighbor search . Khattab and Zaharia [ 16 ] find this fine-grained matching to outperform single-vector representations , a finding extended to open-domain question answering by Khattab et al . [ 17 ] . In this work , we propose FLIPR ( §3.1 ) , a retriever with an improved late interaction mechanism for complex multi-hop queries . Whereas existing retrievers , whether traditional ( e.g. , TF-IDF ) or neural [ 18 ; 15 ; 16 ] , seek passages that match all of the query , multi-hop queries can be long and noisy and need to match disparate contexts . FLIPR handles this explicitly with focused late interaction . Relevance-guided supervision Khattab et al . [ 17 ] propose an iterative strategy for weaklysupervising retrievers called relevance-guided supervision ( RGS ) . RGS assumes that no labeled passages are supplied for training the retriever . Instead , every training question is associated with a short answer string whose presence in a passage is taken as a weak signal for relevance . RGS starts with an off-the-shelf retriever and uses it to collect the top-k passages for each training question , dividing these passages into positive and negative examples based on inclusion of the short answer string . These examples are used to train a stronger retriever , which is then used to repeat this process one or two times . We draw inspiration from RGS when designing latent hop ordering : unlike RGS , we do have gold-labeled passages for training but we crucially have multiple hops , whose order is not given . Lastly , our supervision for the first hop shares inspiration with GoldEn [ 22 ] . GoldEn uses a bag-of-words model—and thus effectively term overlap—to identify the “ more easily retrieved ” passage among just two for the two-hop task HotPotQA .

Summary: This paper proposed Baleen a system that improves the accuracy of multi-hop retrieval while learning robustly from weak training signals in the many-hop setting. Baleen generally contains three modules -- Focused Late Interaction, Latent Hop Ordering, and Condenser. Focused Late Interaction is an improved version of ColBERT, in which they only consider $\text{top-}\hat{K}$ partial scores instead of all scores. Latent Hop Ordering proposed a way to supervise the retriever using weak supervision per-hop. The condenser summaries the sentence-level information retrieved per-hop so it could be better used for the next hop. Experimental results on two multi-hop-based datasets, namely HotpotQA and HoVer show that the proposed framework is quite effective for retrieving multi-hop evidence.