Contrastive Pre-training for Zero-Shot Information Retrieval

1 INTRODUCTION . Document retrieval is the task of finding relevant documents in a large collection to answer specific queries . This is an important task by itself and a core component of many natural language processing ( NLP ) problems , such as open domain question answering ( Chen et al. , 2017 ) or fact checking ( Thorne et al. , 2018 ) . Traditionally , retrieval systems , or retrievers , leverage lexical similarities to match queries and documents , using , for instance , TF-IDF or BM25 weighting ( Robertson & Zaragoza , 2009 ) . These approaches , based on exact matches between tokens of the queries and documents , suffer from the lexical gap and do not generalize well ( Berger et al. , 2000 ) . By contrast , approaches based on neural networks allow learning beyond lexical similarities , resulting in state-ofthe-art performance on question answering benchmarks , such as MS-MARCO ( Nguyen et al. , 2016 ) or NaturalQuestion ( Kwiatkowski et al. , 2019 ) . As training neural networks requires large amount of data , their strong retrieval results were possible for domains and applications where large training datasets are available . In the case of retrieval , creating these datasets requires manually matching queries to the relevant documents in the collection . This is hardly possible when the collection contains millions or billions of element , resulting in many scenarios where only a few in-domain examples , if any , are available . A potential solution is to train a dense retriever on a large retrieval dataset such as MS MARCO , and then apply it to new domains , a setting refered to as zero-shot . Unfortunately , in this setting dense retrievers are often outperformed by classical methods based on term-frequency , which do not require supervision ( Thakur et al. , 2021 ) . Thus , a natural alternative to transfer learning is unsupervised learning , which raises the following question : is it possible to train dense retrievers without supervision , and match the performance of BM25 ? Training dense retrievers without supervision can be achieved by using a pretext task that approximates retrieval . Given a document , one can generate a synthetic query and then train the network to retrieve the original document , among many others , given the query . The inverse Cloze task ( ICT ) , proposed by Lee et al . ( 2019 ) to pre-train retrievers , uses a given sentence as query and predict the context surrounding it . While showing promising results as pre-training ( Chang et al. , 2020 ; Sachan et al. , 2021a ) , this approach still lags behind BM25 when used as a zero-shot retriever . ICT is strongly related to contrastive learning ( Wu et al. , 2018 ) , which has been widely applied in computer vision ( Chen et al. , 2020 ; He et al. , 2020 ) . In particular , computer vision models trained with the latest contrastive learning methods lead to features well suited to retrieval ( Caron et al. , 2021 ) . In this work , we thus propose to revisit how well contrastive learning performs to train dense retrievers without supervision . We want to investigate how much the recent developments from computer vision , such as MoCo , can improve dense retrievers . Here , our goal is not to develop new techniques , but to determine how far we can go by pushing existing methods to their limits . In this paper , we thus make the following contributions . First , we show that contrastive learning can lead to strong unsupervised retrievers : our model achieves Recall @ 100 results competitive with BM25 on most of the BEIR benchmark . Second , in a few-shot setting , we show that our model benefits from few training examples , and obtains better results than transfering models from large datasets such as MS MARCO . Third , when used as a pre-training method before fine-tuning on MS MARCO , our technique leads to state-of-the-art retrieval performance on the BEIR benchmark . Finally , we perform ablations to motivate our design choices , and show that cropping works better than the inverse Cloze task . 2 RELATED WORK . In this section , we briefly review relevant work in information retrieval , and application of machine learning to this problem . This is not an exhaustive review , and we refer the reader to Manning et al . ( 2008 ) , Mitra et al . ( 2018 ) and Lin et al . ( 2020 ) for a more complete introduction to the field . Term-frequency based information retrieval . Historically , in information retrieval , documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary . Different weighing schemes have been proposed , to determine how important a particular term is to a document in a large dataset . One of the most used weighing scheme is known as TF-IDF , and is based on inverse document frequency , or term specificity ( Jones , 1972 ) . BM25 , which is still widely used today , extends TF-IDF ( Robertson et al. , 1995 ) . A well known limitation of these approaches is that they rely on exact match to retrieve documents . This lead to the introduction of latent semantic analysis ( Deerwester et al. , 1990 ) , in which documents are represented as low dimensional dense vectors . Neural network based information retrieval . Following the successful application of deep learning methods to natural language processing , neural networks techniques were introduced for information retrieval . Huang et al . ( 2013 ) proposed a deep bag-of-words model , in which representations of queries and documents are computed independently . A relevance score is then obtained by taking the dot product between representations , and the model is trained end-to-end on click data from a search engine . This method was later refined by replacing the bag-of-words model by convolutional neural networks ( Shen et al. , 2014 ) or recurrent neural network ( Palangi et al. , 2016 ) . A limitation of bi-encoders is that queries and documents are represented by a single vector , preventing the model to capture fine-grained interactions between terms . Nogueira & Cho ( 2019 ) thus introduced a cross-encoder model , based on a pre-trained BERT model ( Devlin et al. , 2019 ) , which jointly encodes queries and documents . The application of a strong pre-trained model , as well as the cross-encoder architecture , lead to important improvement on the MS-MARCO benchmark ( Bajaj et al. , 2016 ) . The methods described in the previous paragraph were applied to re-rank documents , which were retrieved with a traditional IR system such as BM25 . Gillick et al . ( 2018 ) first studied whether continuous retrievers , based on bi-encoder neural models , could be viable alternative to re-ranking . In the context of question answering , Karpukhin et al . ( 2020 ) introduced a dense passage retriever ( DPR ) based on the bi-encoder architecture . This model is initialized with a BERT network , and trained discriminatively using pairs of queries and relevant documents , with hard negatives from BM25 . Xiong et al . ( 2020 ) further extended this work by mining hard negatives with the model itself during optimization , and trained on the MS-MARCO dataset . Once a collection of documents , such as Wikipedia articles , is encoded , retrieval is performed with a fast k-nearest neighbors library such as FAISS Johnson et al . ( 2019 ) . To alleviate the limitations of bi-encoders , Humeau et al . ( 2019 ) introduces the poly-encoder architecture , where documents are encoded by multiple vectors . Similarly , Khattab et al . ( 2020 ) proposes the ColBERT model , which keeps a vector representation for each term of the queries and documents . To make the retrieval tractable , the term-level function is approximated to first retrieve an initial set of candidates , which are then re-ranked with the true score . In the context of question answering , knowledge distillation has been used to train retrievers , either using the attention scores of the reader of the downstream task as synthetic labels ( Izacard & Grave , 2021 ) , or the relevance score from a cross encoder ( Yang & Seo , 2020 ) . Luan et al . ( 2020 ) compares , theoretically and empirically , the performance of sparse and dense retrievers , including bi- , cross- and poly-encoders . Dense retrievers , such as DPR , can lead to indices that weigh close to 100 GB when encoding document collections such as Wikipedia . Izacard et al . ( 2020 ) shows how to compress such indices , with limited impact on performance , making them more practical to use . Self-supervised learning for NLP . Following the success of word2vec ( Mikolov et al. , 2013 ) , many self-supervised techniques have been proposed to learn representation of text . Here , we briefly review the ones that are most related to our approach : sentence level models and contrastive techniques . Jernite et al . ( 2017 ) introduced different objective function to learn sentence representations , including next sentence prediction and binary order prediction . These objectives were later used in pre-trained models based on transformers , such as BERT ( Devlin et al. , 2019 ) and AlBERT ( Lan et al. , 2019 ) . In the context of retrieval , Lee et al . ( 2019 ) introduced the inverse cloze task ( ICT ) , whose purpose is to predict the context surrounding a span of text . Guu et al . ( 2020 ) integrated a bi-encoder retriever model in a BERT pre-training scheme . The retrieved documents are used as additional context in the BERT task , and the whole system is trained end-to-end in an unsupervised way . Similarly , Lewis et al . ( 2020 ) proposed to jointly learn a retriever and a generative seq2seq model , using self-supervised training . Chang et al . ( 2020 ) compares different pre-training tasks for retrieval , including the inverse cloze task . Constrastive learning was introduced in computer vision by Wu et al . ( 2018 ) , followed by several modifications to improve the training ( He et al. , 2020 ; Chen et al. , 2020 ; Caron et al. , 2020 ) . In the context of natural language processing , Fang et al . ( 2020 ) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation . Different works augmented the masked language modeling objective with a contrastive loss ( Giorgi et al. , 2020 ; Wu et al. , 2020 ; Meng et al. , 2021 ) . Finally , SBERT ( Reimers & Gurevych , 2019 ) uses a siamese network similar to contrastive learning to learn a BERT-like model that is adapted to matching sentence embeddings . Their formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose to use data augmentation to leverage large unaligned textual corpora . Concurrent to this work , Gao & Callan ( 2021 ) have also shown the potential of contrastive learning for information retrieval ; building on the same observation that both tasks share a similar structure . 3 METHOD . In this section , we describe how to train a dense retriever with no supervision . We review the model architecture and then describe contrastive learning - a key component of its training . The objective of a retriever is to find relevant documents in a large collection for a given query . Thus , the retriever takes as input the set of documents and the query and outputs a relevance score for each document . A standard approach is to encode each query–document pair with a neural network ( Nogueira & Cho , 2019 ) . This procedure requires re-encoding every document for any new query and hence does not scale to large collections of documents . Instead , we propose to use a bi-encoder architecture , where documents and queries are encoded independently ( Huang et al. , 2013 ; Karpukhin et al. , 2020 ) . One can compute the relevance score by taking the dot product ( or cosine similarity ) between the document ’ s representation and the representation of the query . More precisely , given a pair of query q and document d , we encode each of them independently using the same model , fθ , parametrized by θ . The relevance score s ( q , d ) is then the dot product of the resulting representations : s ( q , d ) = 〈fθ ( q ) , fθ ( d ) 〉 . In practice , we use a transformer network for fθ to embed both queries and documents . The representation fθ ( q ) ( resp . fθ ( d ) ) for a query ( resp . document ) is obtained by averaging the hidden representations of the last layer . Following previous work on dense retrieval with neural networks , we use the BERT base uncased architecture and refer the reader to Devlin et al . ( 2019 ) for more details .

Summary: This paper proposes to use cropping strategy to train a neural model for IR. It is compared to BM25 and several other neural models, including BERT and the ones using inverse cloze task. The goal of cropping is to train a model that fits better the IR tasks. It is hypothesized that this task is more similar to IR than other pre-training strategies. The method is tested on a set of benchmarks of zero-shot retrieval and MSMARCO with training data. It is shown than the proposed method fine-tuned on MSMARCO can outperform the compared methods, including BM25. When no fine-tuning is used, the method is almost at par with BM25.