Phrase Retrieval Learns Passage Retrieval, Too

1 . Introduction . Dense retrieval aims to retrieve relevant contexts from a large corpus , by learning dense representations of queries and text segments . Recently , dense retrieval of passages [ Lee et al. , 2019 , Karpukhin et al. , 2020 , Xiong et al. , 2021 ] has been shown to outperform traditional sparse retrieval methods such as TF-IDF and BM25 in a range of knowledge-intensive NLP tasks [ Petroni et al. , 2021 ] , including open-domain question answering ( QA ) [ Chen et al. , 2017 ] , entity linking [ Wu et al. , 2020 ] , and knowledge-grounded dialogue [ Dinan et al. , 2019 ] . One natural design choice of these dense retrieval methods is the retrieval unit . For instance , the dense passage retriever ( DPR ) [ Karpukhin et al. , 2020 ] encodes a fixed-size text block of 100 words as the basic retrieval unit . On the other extreme , recent work [ Seo et al. , 2019 , Lee et al. , 2021 ] demonstrates that phrases can be used as a retrieval unit . In particular , Lee et al . [ 2021 ] show that learning dense representations of phrases alone can achieve competitive performance in a number of open-domain QA and slot filling tasks . This is particularly appealing since the phrases can directly serve as the output , without relying on an additional reader model to process text passages . In this work , we draw on an intuitive motivation that every single phrase is embedded within a larger text context and ask the following question : If a retriever is able to locate 1 . Following previous work [ Seo et al. , 2018 , 2019 ] , the term phrase denotes any contiguous text segment up to L words , which is not necessarily a linguistic phrase ( see Section 2 ) . 2 . Our code and models are available at https : //github.com/princeton-nlp/DensePhrases . phrases , can we directly make use of it for passage and even document retrieval as well ? We formulate phrase-based passage retrieval , in which the score of a passage is determined by the maximum score of phrases within it ( see Figure 1 for an illustration ) . By evaluating DensePhrases [ Lee et al. , 2021 ] on popular QA datasets , we observe that it achieves competitive or even better passage retrieval accuracy compared to DPR , without any re-training or modification to the original model . The gains are especially pronounced for top-k accuracy when k is smaller ( e.g. , 5 ) , which helps achieve strong open-domain QA accuracy with a much smaller number of passages as input to a generative model [ Izacard and Grave , 2021a ] . To better understand the nature of dense retrieval methods , we carefully analyze the training objectives of phrase and passage retrieval methods . While the in-batch negative losses in both models encourage them to retrieve topically relevant passages , we find that phrase-level supervision in DensePhrases provides a stronger training signal than using hard negatives from BM25 , and helps DensePhrases retrieve correct phrases , and hence passages . Following this positive finding , we further explore whether phrase retrieval can be extended to retrieval of coarser granularities , or other NLP tasks . Through fine-tuning of the query encoder with document-level supervision , we are able to obtain competitive performance on entity linking [ Hoffart et al. , 2011 ] and knowledge-grounded dialogue retrieval [ Dinan et al. , 2019 ] in the KILT benchmark [ Petroni et al. , 2021 ] . Finally , we draw connections to multi-vector passage encoding models [ Khattab and Zaharia , 2020 , Luan et al. , 2021 ] , where phrase retrieval models can be viewed as learning a dynamic set of vectors for each passage . We show that a phrase filtering strategy learned from QA datasets gives us a control over the trade-off between the number of vectors per passage and the retrieval accuracy . Since phrase retrievers encode a larger number of vectors , we also propose a quantization-aware fine-tuning method based on Optimized Product Quantization [ Ge et al. , 2013 ] , reducing the size of the phrase index from 307GB to 69GB ( or under 30GB with more aggressive phrase filtering ) for full English Wikipedia , without performance degradation . This matches the index size of passage retrievers and makes dense phrase retrieval a practical and versatile solution for multi-granularity retrieval . 2 . Background . Passage retrieval Given a set of documents D , passage retrieval aims to provide a set of relevant passages for a question q . Typically , each document in D is segmented into a set of disjoint passages and we denote the entire set of passages in D as P = { p1 , . . . , pM } , where each passage can be a natural paragraph or a fixed-length text block . A passage retriever is designed to return top-k passages Pk ⊂ P with the goal of retrieving passages that are relevant to the question . In open-domain QA , passages are considered relevant if they contain answers . However , many other knowledge-intensive NLP tasks ( e.g. , knowledgegrounded dialogue ) provide human-annotated evidence passages or documents . While traditional passage retrieval models rely on sparse representations such as BM25 [ Robertson and Zaragoza , 2009 ] , recent methods show promising results with dense representations of passages and questions , and enable retrieving passages that may have low lexical overlap with questions . Specifically , Karpukhin et al . [ 2020 ] introduce DPR that has a passage encoder Ep ( · ) and a question encoder Eq ( · ) trained on QA datasets and retrieves passages by using the inner product as a similarity function : f ( p , q ) = Ep ( p ) > Eq ( q ) . ( 1 ) For open-domain QA where a system is required to provide an exact answer string a , the retrieved top k passages Pk are subsequently fed into a reading comprehension model such as a BERT model [ Devlin et al. , 2019 ] .3 Phrase retrieval While passage retrievers require another reader model to find an answer , Seo et al . [ 2019 ] introduce the phrase retrieval approach that encodes phrases in each document and performs similarity search over all phrase vectors to directly locate the answer . Following previous work [ Seo et al. , 2018 , 2019 ] , we use the term ‘ phrase ’ to denote any contiguous text segment up to L words ( including single words ) , which is not necessarily a linguistic phrase and we take phrases up to length L = 20 . Given a phrase s ( p ) from a passage p , their similarity function f is computed as : f ( s ( p ) , q ) = Es ( s ( p ) ) > Eq ( q ) , ( 2 ) where Es ( · ) and Eq ( · ) denote the phrase encoder and the question encoder , respectively . Since this formulates open-domain QA purely as a maximum inner product search ( MIPS ) , it can drastically improve end-to-end efficiency . While previous work [ Seo et al. , 2019 , Lee et al. , 2020 ] relied on a combination of dense and sparse vectors , Lee et al . [ 2021 ] demonstrate that dense representations of phrases alone are sufficient to close the performance gap with retriever-reader systems . For more details on how phrase representations are learned , we refer interested readers to Lee et al . [ 2021 ] . 3 . Phrase Retrieval for Passage Retrieval . Phrases naturally have their source texts from which they are extracted . Based on this fact , we define a simple phrase-based passage retrieval strategy , where we retrieve passages based on the phrase-retrieval score : f̃ ( p , q ) : = max s ( p ) ∈S ( p ) Es ( s ( p ) ) > Eq ( q ) , ( 3 ) where S ( p ) denotes the set of phrases in the passage p. In practice , we first retrieve a slightly larger number of phrases , compute the score for each passage , and return top k 3 . This is called the retriever-reader approach [ Chen et al. , 2017 ] . unique passages.4 Based on our definition , phrases can act as a basic retrieval unit of any other granularity such as sentences or documents by simply changing S ( p ) ( e.g. , s ( d ) ∈ S ( d ) for a document d ) . Note that , since the cost of score aggregation is negligible , the inference speed of phrase-based passage retrieval is the same as for phrase retrieval , which is shown to be efficient in Lee et al . [ 2021 ] . In this section , we evaluate the passage retrieval performance of phrase-based passage retrieval and how it can contribute to end-to-end open-domain QA . 3.1 Experiment : Passage Retrieval . Datasets We use two open-domain QA datasets : Natural Questions [ Kwiatkowski et al. , 2019 ] and TriviaQA [ Joshi et al. , 2017 ] , following the standard train/dev/test splits for the open-domain QA evaluation . For both models , we use the 2018-12-20 Wikipedia snapshot . To provide a fair comparison , we use Wikipedia articles pre-processed for DPR , which are split into 21-million text blocks and each text block has exactly 100 words . Note that while DPR is trained in this setting , DensePhrases is trained with natural paragraphs.5 Models For DPR , we use publicly available checkpoints6 trained on each dataset ( DPR♦ ) or multiple QA datasets ( DPR♠ ) , which we find to perform slightly better than the ones reported in Karpukhin et al . [ 2020 ] . For DensePhrases , we train it on Natural Questions ( DensePhrases♦ ) or multiple QA datasets ( DensePhrases♠ ) with the code provided by the authors.7 Note that we do not make any modification to the architecture or training methods of DensePhrases and achieve similar open-domain QA accuracy as reported . For phrasebased passage retrieval , we compute Eq . ( 3 ) with DensePhrases and return top k passages . Metrics Following previous work , we measure the top-k passage retrieval accuracy ( Topk ) , which denotes the proportion of questions whose top k retrieved passages contain at least one of the gold answers . To further characterize the behavior of each system , we include 4 . Retrieving 2k phrases is often sufficient for obtaining k unique passages . If not , we try 4k and so on . 5 . We expect DensePhrases to achieve even higher performance if it is re-trained with 100-word text blocks . 6. https : //github.com/facebookresearch/DPR . 7 . DPR♠ is trained on NaturalQuestions , TriviaQA , CuratedTREC [ Baudǐs and Šedivỳ , 2015 ] , and WebQuestions [ Berant et al. , 2013 ] . DensePhrases♠ additionally includes SQuAD [ Rajpurkar et al. , 2016 ] , although it does not contribute to Natural Questions and TriviaQA much . the following evaluation metrics : mean reciprocal rank at k ( MRR @ k ) and precision at k ( P @ k ) . MRR @ k is the average reciprocal rank of the first relevant passage ( that contains an answer ) in the top k passages . Higher MRR @ k means relevant passages appear at higher ranks . Meanwhile , P @ k is the average proportion of relevant passages in the top k passages . Higher P @ k denotes that a larger proportion of top k passages contains the answers . Results As shown in Table 1 , DensePhrases achieves competitive passage retrieval accuracy with DPR , while having a clear advantage on top-1 or top-5 accuracy for both Natural Questions ( +6.9 % Top-1 ) and TriviaQA ( +8.1 % Top-1 ) . Although the top-20 ( and top100 , which is not shown ) accuracy is similar across different models , MRR @ 20 and P @ 20 reveal interesting aspects of DensePhrases—it ranks relevant passages higher and provides a larger number of correct passages . Our results suggest that DensePhrases can also retrieve passages very accurately , even though it was not explicitly trained for that purpose . For the rest of the paper , we mainly compare the DPR♠ and DensePhrases♠ models , which were both trained on multiple QA datasets . 3.2 Experiment : Open-domain QA Recently , Izacard and Grave [ 2021a ] proposed the Fusion-in-Decoder ( FiD ) approach where they feed top 100 passages from DPR into a generative model T5 [ Raffel et al. , 2020 ] and achieve the state-ofthe-art on open-domain QA benchmarks . Since their generative model computes the hidden states of all tokens in 100 passages , it requires large GPU memory ( e.g. , 64 Tesla V100 32GB for training T5 [ Izacard and Grave , 2021a ] ) . We use our phrase-based passage retrieval with DensePhrases to replace DPR in FiD and see if we can use a much smaller number of passages to achieve comparable performance , which can greatly reduce the computational requirements . We train our model with 4 24GB RTX GPUs for training T5-base,8 which are more affordable with academic budgets.9 Results As shown in Table 2 , using DensePhrases as a passage retriever achieves competitive performance to DPR-based FiD and significantly improves upon the performance of original DensePhrases . Its better retrieval quality at top-k for smaller k indeed translates to better open-domain QA accuracy , achieving +6.4 % gain compared to DPR-based FiD 8 . Note that training T5-base with 5 or 10 passages can also be done with 11GB GPUs . 9 . We also keep all the hyperparameters the same as in Izacard and Grave [ 2021a ] except that we accumulate gradients for 16 steps to match the effective batch size of the original work . when k = 5 . To obtain similar performance with using 100 passages in FiD , DensePhrases needs fewer passages ( k = 25 or 50 ) , which can fit in GPUs with smaller RAM .

Summary: The paper shows that dense phrase Representations can be better representations for retrieval than the document-level representations. Specifically they show that DensePhrases representations can be used to identify relevant documents _without any fine-tuning_ on the target QA dataset compared to DPR. Additionally they show why phrase representations might outperform passage-level representations (much harder negatives). On non-QA datasets, fine-tuning the query representations can lead to competitive performance too. Finally they also provide a potential solution to reduce the index size of DensePhrases to match that of DPR while maintaining a higher performance.