Phrase Retrieval Learns Passage Retrieval, Too

1 . Introduction . Dense retrieval aims to retrieve relevant contexts from a large corpus , by learning dense representations of queries and text segments . Recently , dense retrieval of passages [ Lee et al. , 2019 , Karpukhin et al. , 2020 , Xiong et al. , 2021 ] has been shown to outperform traditional sparse retrieval methods such as TF-IDF and BM25 in a range of knowledge-intensive NLP tasks [ Petroni et al. , 2021 ] , including open-domain question answering ( QA ) [ Chen et al. , 2017 ] , entity linking [ Wu et al. , 2020 ] , and knowledge-grounded dialogue [ Dinan et al. , 2019 ] . One natural design choice of these dense retrieval methods is the retrieval unit . For instance , the dense passage retriever ( DPR ) [ Karpukhin et al. , 2020 ] encodes a fixed-size text block of 100 words as the basic retrieval unit . On the other extreme , recent work [ Seo et al. , 2019 , Lee et al. , 2021 ] demonstrates that phrases can be used as a retrieval unit . In particular , Lee et al . [ 2021 ] show that learning dense representations of phrases alone can achieve competitive performance in a number of open-domain QA and slot filling tasks . This is particularly appealing since the phrases can directly serve as the output , without relying on an additional reader model to process text passages . In this work , we draw on an intuitive motivation that every single phrase is embedded within a larger text context and ask the following question : If a retriever is able to locate 1 . Following previous work [ Seo et al. , 2018 , 2019 ] , the term phrase denotes any contiguous text segment up to L words , which is not necessarily a linguistic phrase ( see Section 2 ) . 2 . Our code and models are available at https : //github.com/princeton-nlp/DensePhrases . phrases , can we directly make use of it for passage and even document retrieval as well ? We formulate phrase-based passage retrieval , in which the score of a passage is determined by the maximum score of phrases within it ( see Figure 1 for an illustration ) . By evaluating DensePhrases [ Lee et al. , 2021 ] on popular QA datasets , we observe that it achieves competitive or even better passage retrieval accuracy compared to DPR , without any re-training or modification to the original model . The gains are especially pronounced for top-k accuracy when k is smaller ( e.g. , 5 ) , which helps achieve strong open-domain QA accuracy with a much smaller number of passages as input to a generative model [ Izacard and Grave , 2021a ] . To better understand the nature of dense retrieval methods , we carefully analyze the training objectives of phrase and passage retrieval methods . While the in-batch negative losses in both models encourage them to retrieve topically relevant passages , we find that phrase-level supervision in DensePhrases provides a stronger training signal than using hard negatives from BM25 , and helps DensePhrases retrieve correct phrases , and hence passages . Following this positive finding , we further explore whether phrase retrieval can be extended to retrieval of coarser granularities , or other NLP tasks . Through fine-tuning of the query encoder with document-level supervision , we are able to obtain competitive performance on entity linking [ Hoffart et al. , 2011 ] and knowledge-grounded dialogue retrieval [ Dinan et al. , 2019 ] in the KILT benchmark [ Petroni et al. , 2021 ] . Finally , we draw connections to multi-vector passage encoding models [ Khattab and Zaharia , 2020 , Luan et al. , 2021 ] , where phrase retrieval models can be viewed as learning a dynamic set of vectors for each passage . We show that a phrase filtering strategy learned from QA datasets gives us a control over the trade-off between the number of vectors per passage and the retrieval accuracy . Since phrase retrievers encode a larger number of vectors , we also propose a quantization-aware fine-tuning method based on Optimized Product Quantization [ Ge et al. , 2013 ] , reducing the size of the phrase index from 307GB to 69GB ( or under 30GB with more aggressive phrase filtering ) for full English Wikipedia , without performance degradation . This matches the index size of passage retrievers and makes dense phrase retrieval a practical and versatile solution for multi-granularity retrieval . 2 . Background . Passage retrieval Given a set of documents D , passage retrieval aims to provide a set of relevant passages for a question q . Typically , each document in D is segmented into a set of disjoint passages and we denote the entire set of passages in D as P = { p1 , . . . , pM } , where each passage can be a natural paragraph or a fixed-length text block . A passage retriever is designed to return top-k passages Pk ⊂ P with the goal of retrieving passages that are relevant to the question . In open-domain QA , passages are considered relevant if they contain answers . However , many other knowledge-intensive NLP tasks ( e.g. , knowledgegrounded dialogue ) provide human-annotated evidence passages or documents . While traditional passage retrieval models rely on sparse representations such as BM25 [ Robertson and Zaragoza , 2009 ] , recent methods show promising results with dense representations of passages and questions , and enable retrieving passages that may have low lexical overlap with questions . Specifically , Karpukhin et al . [ 2020 ] introduce DPR that has a passage encoder Ep ( · ) and a question encoder Eq ( · ) trained on QA datasets and retrieves passages by using the inner product as a similarity function : f ( p , q ) = Ep ( p ) > Eq ( q ) . ( 1 ) For open-domain QA where a system is required to provide an exact answer string a , the retrieved top k passages Pk are subsequently fed into a reading comprehension model such as a BERT model [ Devlin et al. , 2019 ] .3 Phrase retrieval While passage retrievers require another reader model to find an answer , Seo et al . [ 2019 ] introduce the phrase retrieval approach that encodes phrases in each document and performs similarity search over all phrase vectors to directly locate the answer . Following previous work [ Seo et al. , 2018 , 2019 ] , we use the term ‘ phrase ’ to denote any contiguous text segment up to L words ( including single words ) , which is not necessarily a linguistic phrase and we take phrases up to length L = 20 . Given a phrase s ( p ) from a passage p , their similarity function f is computed as : f ( s ( p ) , q ) = Es ( s ( p ) ) > Eq ( q ) , ( 2 ) where Es ( · ) and Eq ( · ) denote the phrase encoder and the question encoder , respectively . Since this formulates open-domain QA purely as a maximum inner product search ( MIPS ) , it can drastically improve end-to-end efficiency . While previous work [ Seo et al. , 2019 , Lee et al. , 2020 ] relied on a combination of dense and sparse vectors , Lee et al . [ 2021 ] demonstrate that dense representations of phrases alone are sufficient to close the performance gap with retriever-reader systems . For more details on how phrase representations are learned , we refer interested readers to Lee et al . [ 2021 ] . 3 . Phrase Retrieval for Passage Retrieval . Phrases naturally have their source texts from which they are extracted . Based on this fact , we define a simple phrase-based passage retrieval strategy , where we retrieve passages based on the phrase-retrieval score : f̃ ( p , q ) : = max s ( p ) ∈S ( p ) Es ( s ( p ) ) > Eq ( q ) , ( 3 ) where S ( p ) denotes the set of phrases in the passage p. In practice , we first retrieve a slightly larger number of phrases , compute the score for each passage , and return top k 3 . This is called the retriever-reader approach [ Chen et al. , 2017 ] . unique passages.4 Based on our definition , phrases can act as a basic retrieval unit of any other granularity such as sentences or documents by simply changing S ( p ) ( e.g. , s ( d ) ∈ S ( d ) for a document d ) . Note that , since the cost of score aggregation is negligible , the inference speed of phrase-based passage retrieval is the same as for phrase retrieval , which is shown to be efficient in Lee et al . [ 2021 ] . In this section , we evaluate the passage retrieval performance of phrase-based passage retrieval and how it can contribute to end-to-end open-domain QA . 3.1 Experiment : Passage Retrieval . Datasets We use two open-domain QA datasets : Natural Questions [ Kwiatkowski et al. , 2019 ] and TriviaQA [ Joshi et al. , 2017 ] , following the standard train/dev/test splits for the open-domain QA evaluation . For both models , we use the 2018-12-20 Wikipedia snapshot . To provide a fair comparison , we use Wikipedia articles pre-processed for DPR , which are split into 21-million text blocks and each text block has exactly 100 words . Note that while DPR is trained in this setting , DensePhrases is trained with natural paragraphs.5 Models For DPR , we use publicly available checkpoints6 trained on each dataset ( DPR♦ ) or multiple QA datasets ( DPR♠ ) , which we find to perform slightly better than the ones reported in Karpukhin et al . [ 2020 ] . For DensePhrases , we train it on Natural Questions ( DensePhrases♦ ) or multiple QA datasets ( DensePhrases♠ ) with the code provided by the authors.7 Note that we do not make any modification to the architecture or training methods of DensePhrases and achieve similar open-domain QA accuracy as reported . For phrasebased passage retrieval , we compute Eq . ( 3 ) with DensePhrases and return top k passages . Metrics Following previous work , we measure the top-k passage retrieval accuracy ( Topk ) , which denotes the proportion of questions whose top k retrieved passages contain at least one of the gold answers . To further characterize the behavior of each system , we include 4 . Retrieving 2k phrases is often sufficient for obtaining k unique passages . If not , we try 4k and so on . 5 . We expect DensePhrases to achieve even higher performance if it is re-trained with 100-word text blocks . 6. https : //github.com/facebookresearch/DPR . 7 . DPR♠ is trained on NaturalQuestions , TriviaQA , CuratedTREC [ Baudǐs and Šedivỳ , 2015 ] , and WebQuestions [ Berant et al. , 2013 ] . DensePhrases♠ additionally includes SQuAD [ Rajpurkar et al. , 2016 ] , although it does not contribute to Natural Questions and TriviaQA much . the following evaluation metrics : mean reciprocal rank at k ( MRR @ k ) and precision at k ( P @ k ) . MRR @ k is the average reciprocal rank of the first relevant passage ( that contains an answer ) in the top k passages . Higher MRR @ k means relevant passages appear at higher ranks . Meanwhile , P @ k is the average proportion of relevant passages in the top k passages . Higher P @ k denotes that a larger proportion of top k passages contains the answers . Results As shown in Table 1 , DensePhrases achieves competitive passage retrieval accuracy with DPR , while having a clear advantage on top-1 or top-5 accuracy for both Natural Questions ( +6.9 % Top-1 ) and TriviaQA ( +8.1 % Top-1 ) . Although the top-20 ( and top100 , which is not shown ) accuracy is similar across different models , MRR @ 20 and P @ 20 reveal interesting aspects of DensePhrases—it ranks relevant passages higher and provides a larger number of correct passages . Our results suggest that DensePhrases can also retrieve passages very accurately , even though it was not explicitly trained for that purpose . For the rest of the paper , we mainly compare the DPR♠ and DensePhrases♠ models , which were both trained on multiple QA datasets . 3.2 Experiment : Open-domain QA Recently , Izacard and Grave [ 2021a ] proposed the Fusion-in-Decoder ( FiD ) approach where they feed top 100 passages from DPR into a generative model T5 [ Raffel et al. , 2020 ] and achieve the state-ofthe-art on open-domain QA benchmarks . Since their generative model computes the hidden states of all tokens in 100 passages , it requires large GPU memory ( e.g. , 64 Tesla V100 32GB for training T5 [ Izacard and Grave , 2021a ] ) . We use our phrase-based passage retrieval with DensePhrases to replace DPR in FiD and see if we can use a much smaller number of passages to achieve comparable performance , which can greatly reduce the computational requirements . We train our model with 4 24GB RTX GPUs for training T5-base,8 which are more affordable with academic budgets.9 Results As shown in Table 2 , using DensePhrases as a passage retriever achieves competitive performance to DPR-based FiD and significantly improves upon the performance of original DensePhrases . Its better retrieval quality at top-k for smaller k indeed translates to better open-domain QA accuracy , achieving +6.4 % gain compared to DPR-based FiD 8 . Note that training T5-base with 5 or 10 passages can also be done with 11GB GPUs . 9 . We also keep all the hyperparameters the same as in Izacard and Grave [ 2021a ] except that we accumulate gradients for 16 steps to match the effective batch size of the original work . when k = 5 . To obtain similar performance with using 100 passages in FiD , DensePhrases needs fewer passages ( k = 25 or 50 ) , which can fit in GPUs with smaller RAM .

Summary: This paper explores how dense phrase retrieval (i.e., treating a phrase as the unit of retrieval) compares to dense passage retrieval in the context of QA and specialized document retrieval tasks. To improve phrase retrieval, the authors propose the use of in-passage negatives (i.e., negative phrases taken from a relevant passage) and discuss how this relates to strategies for choosing negative exmaples with passage retrieval. In the evaluation, the proposed approach is more effective than DPR (passage retrieval) for QA and competitive on the other tasks. Additionally, the authors demonstrate that quantization can be used to substantially decrease the phrase index size, which mitigates this potential downside of phrase retrieval.