Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval

1 Introduction . In open-domain reasoning , a model is tasked with retrieving evidence from a large corpus to answer questions , verify claims , or otherwise exhibit broad knowledge . In practice , such open-domain tasks often further require multi-hop reasoning , where the evidence must be extracted from two or more documents . To do this effectively , a model must learn to use partial evidence it retrieves to bridge its way to additional documents leading to an answer . In this vein , HotPotQA [ 31 ] contains complex questions answerable by retrieving two passages from Wikipedia , while HoVer [ 13 ] contains claims that can only be verified ( or disproven ) by combining facts from up to four Wikipedia passages . Table 1 illustrates this using the claim “ The MVP of [ a ] game Red Flaherty umpired was elected to the Baseball Hall of Fame ” from the HoVer validation set . To verify this claim , a HoVer model must identify facts spread across three Wikipedia pages : Red Flaherty umpired in the World Series in 1955 , 1958 , 1965 , and 1970 . The MVP of the 1965 World Series was Sandy Koufax . Koufax was later elected to the Baseball Hall of Fame . This three-hop claim illustrates three major challenges in multi-hop retrieval . First , multi-hop queries encompass multiple information needs ; the claim above referenced facts from three disparate passages . Second , retrieval errors in each hop propagate to subsequent hops . This can happen if the model directly retrieves information about the Baseball Hall of Fame , confuses Red Flaherty with , say , Robert Flaherty , or singles out the MVP of , say , the 1958 World Series , which Flaherty also umpired . Third , due to the dependency between hops , retrievers must learn an effective sequence of hops , where previously-retrieved clues lead to other relevant passages . These inter-passage dependencies 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can be non-obvious for many-hop problems with three or more passages , and are often left unlabeled , as it can be expensive to annotate one ( or every ) sequence in which facts could be retrieved . These challenges call for highly expressive query representations , robustness to retrieval errors , and scalability to many hops over massive document collections . Existing systems fall short on one or more of these criteria . For instance , many state-of-the-art systems rely on bag-of-words [ 23 ] or single-vector dot-product [ 29 ] retrievers , whose capacity to model an open-domain question is limited [ 17 ] , let alone complex multi-hop queries . Furthermore , existing systems embed trade-offs when it comes to “ hopping ” : they employ brittle greedy search , which limits recall per hop ; or they employ beam search over an exponential space , which reduces scalability to many hops ; or they assume explicit links that connect every passage with related entities , which ties them to link-structured corpora . Lastly , to order the hops , many systems use fragile supervision heuristics ( e.g. , finding passages whose page titles appear in other passages ) tailored for particular datasets like HotPotQA . We tackle these problems with Baleen,1 a scalable multi-hop reasoning system that improves accuracy and robustness . We introduce a condensed retrieval architecture , where the retrieved facts from each hop are summarized into a short context that becomes a part of the query for subsequent hops , if any ( Table 1 ) . Unlike beam search , condensed retrieval allows effective scaling to many hops , and we find that it complements greedy search ( i.e. , taking the best passage per hop ) in improving recall considerably . We then tackle the complexity of queries by proposing a focused late interaction passage retriever ( FLIPR ) , a robust learned search model that allow different parts of the same query representation to match disparate relevant passages . FLIPR inherits the scalability of the vanilla late interaction paradigm of ColBERT [ 16 ] but uniquely allows the same query to exhibit tailored matching patterns against each target passage . Lastly , we devise latent hop ordering , a weak-supervision strategy that uses the retriever itself to select effective hop paths . We first test Baleen on the two-hop HotPotQA benchmark , finding evidence of saturation in retrieval : we achieve 96.3 % answer recall in the top-20 retrieved passages , up from 89.4 % for existing work . We then test Baleen ’ s ability to scale accurately to more hops , reporting our main results using the recent many-hop HoVer task . We build a strong many-hop baseline model that combines recent results from the open-domain question answering [ 17 ] and multi-hop [ 23 ] literatures . This baseline combines ColBERT retrieval and standard greedy search with an ELECTRA [ 5 ] re-ranker , and generalizes typical text-based heuristics to order the hops for training . After verifying its strong results on HotPotQA , we show that it outperforms the official TF-IDF + BERT baseline of HoVer by over 30 points in retrieval accuracy . Against this strong baseline itself , Baleen improves passage retrieval accuracy by another 17 points , raising it to over 90 % at k = 100 passages . Baleen also improves the evidence extraction F1 score dramatically , outperforming even the oracle retrieval + BERT results by Jiang et al . [ 13 ] . Our ablations ( §5.4 ) show that Baleen ’ s FLIPR retriever , condenser architecture , and latent supervision are essential to its strong performance . 1https : //github.com/stanford-futuredata/Baleen In marine biology , “ baleen ” refers to the filter-feeding system that baleen whales use to capture small organisms , filtering out seawater . So too does our system seek to capture relevant facts from a sea of documents . 2 Background and related work . Open-domain reasoning There is significant recent interest in NLP models that can solve tasks by retrieving evidence from a large corpus . The most popular such task is arguably open-domain question answering ( OpenQA ; Chen et al . [ 2 ] ) , which extends the well-studied machine reading comprehension ( MRC ) problem over supplied question–passage pairs to answering questions given only a large text corpus like Wikipedia . Other open-domain tasks span claim verification ( e.g. , FEVER ; Thorne et al . [ 24 ] ) , question generation ( e.g. , with Jeopardy as in Lewis et al . [ 19 ] ) , and open dialogue ( e.g. , Wizard of Wikipedia ; Dinan et al . [ 8 ] ) , among others . Many of these datasets are compiled in the recently-introduced KILT benchmark [ 21 ] for knowledge-intensive tasks . Among models for these tasks , the most relevant to our work are OpenQA models that include learned retrieval components . These include ORQA [ 18 ] , REALM [ 10 ] , and DPR [ 15 ] . Lewis et al . [ 19 ] introduced RAG , which by virtue of a seq2seq architecture can tackle OpenQA as well as other open-domain problems , including claim verification and question generation . Multi-hop open-domain reasoning Most of the open-domain tasks from §2 can be solved by finding one relevant passage in the corpus , often by design . In contrast , a number of recent works explore multi-hop reasoning over multiple passages . These include QAngaroo [ 27 ] and 2WikiMultiHopQA [ 11 ] , among others . While thus “ multi-hop ” , these tasks supply the relevant passages for each example ( possibly with small set of “ distractor ” candidates ) , and thus do not require retrieval from a large corpus . To our knowledge , HotPotQA was the first large-scale open-domain multi-hop task , particularly in its retrieval-oriented “ fullwiki ” setting . HotPotQA catalyzed much follow-up research . However , it is limited to only two-hop questions , many of which may not require strong multi-hop reasoning capabilities ( see Chen and Durrett [ 3 ] ; Wang et al . [ 26 ] ) , softening the retrieval challenges discussed in §1 . Very recently , Jiang et al . [ 13 ] introduced the HoVer many-hop verification dataset , which contains two- , three- , and four-hop examples . HoVer contains just over 18,000 training examples , about 5× smaller than HotPotQA , adding to the challenge posed by HoVer . Multi-hop open-domain models To conduct the “ hops ” , many prior multi-hop systems ( e.g. , Asai et al . [ 1 ] ; Zhao et al . [ 32 ] ) assume explicit links that connect every passage with related entities . We argue that this risks tying systems to link-structured knowledge bases ( like Wikipedia ) or producing brittle architectures tailored for datasets constructed by following hyperlinks ( like HotPotQA ) . Recently , Xiong et al . [ 29 ] and Qi et al . [ 23 ] introduce MDR and IRRR , state-of-the-art systems that assume no explicit link structure . Instead , they use an iterative retrieval paradigm—akin to that introduced by Das et al . [ 6 ] and Feldman and El-Yaniv [ 9 ] —that retrieves passages relevant to the question , reads these passages , and then formulates a new query for another hop if necessary . We adopt this iterative formulation and tackle three major challenges for multi-hop retrieval . ColBERT : late interaction paradigm Most learned-retrieval systems for OpenQA ( §2 ) encode every query and passage into a single dense vector . Khattab and Zaharia [ 16 ] argue that such singlevector representations are not sufficiently expressive for retrieval in many scenarios and introduce late interaction , a paradigm that represents every query and every document at a finer granularity : it uses a vector for each constituent token . Within this paradigm , they propose ColBERT , a state-of-the-art retriever wherein a BERT encoder embeds the query into a matrix of N vectors ( given N tokens ) and encodes every passage in the corpus as matrix of M vectors ( for M tokens per passage ) . The passage representations are query-independent and thus are computed offline and indexed for fast retrieval . During retrieval with query q , ColBERT assigns a score to a passage d by finding the maximumsimilarity ( MaxSim ) score between each vector in q ’ s representation and all the vectors of d and then summing these MaxSim scores . This MaxSim-decomposed interaction enables scaling to massive collections with millions of passages , as the token-level passage vectors can all be indexed for fast nearest-neighbor search . Khattab and Zaharia [ 16 ] find this fine-grained matching to outperform single-vector representations , a finding extended to open-domain question answering by Khattab et al . [ 17 ] . In this work , we propose FLIPR ( §3.1 ) , a retriever with an improved late interaction mechanism for complex multi-hop queries . Whereas existing retrievers , whether traditional ( e.g. , TF-IDF ) or neural [ 18 ; 15 ; 16 ] , seek passages that match all of the query , multi-hop queries can be long and noisy and need to match disparate contexts . FLIPR handles this explicitly with focused late interaction . Relevance-guided supervision Khattab et al . [ 17 ] propose an iterative strategy for weaklysupervising retrievers called relevance-guided supervision ( RGS ) . RGS assumes that no labeled passages are supplied for training the retriever . Instead , every training question is associated with a short answer string whose presence in a passage is taken as a weak signal for relevance . RGS starts with an off-the-shelf retriever and uses it to collect the top-k passages for each training question , dividing these passages into positive and negative examples based on inclusion of the short answer string . These examples are used to train a stronger retriever , which is then used to repeat this process one or two times . We draw inspiration from RGS when designing latent hop ordering : unlike RGS , we do have gold-labeled passages for training but we crucially have multiple hops , whose order is not given . Lastly , our supervision for the first hop shares inspiration with GoldEn [ 22 ] . GoldEn uses a bag-of-words model—and thus effectively term overlap—to identify the “ more easily retrieved ” passage among just two for the two-hop task HotPotQA .

Summary: This paper introduces a new multi-hop reasoning system called Baleen which conducts iterative retrieval and condensing. Baleen repeats the iterative process for pre-defined time steps T (e.g., T=2 for HotpotQA and T=4 for HoVer)  and at every time step, it takes the query from the previous time step and retriever top K passages, and create a new query for the current t step by extracting and condensing the facts (sentences) from the retrieved passages. When they finish retrieval (i.e., t=T), they feed the final query to the reader model to get the final predictions.  They conducted experiments on HotpotQA and HoVer. The experimental results on HotpotQA show its competitive passage retrieval performance and they show state-of-the-art results on HoVer, significantly outperforming the baselines.