Contrastive Pre-training for Zero-Shot Information Retrieval

1 INTRODUCTION . Document retrieval is the task of finding relevant documents in a large collection to answer specific queries . This is an important task by itself and a core component of many natural language processing ( NLP ) problems , such as open domain question answering ( Chen et al. , 2017 ) or fact checking ( Thorne et al. , 2018 ) . Traditionally , retrieval systems , or retrievers , leverage lexical similarities to match queries and documents , using , for instance , TF-IDF or BM25 weighting ( Robertson & Zaragoza , 2009 ) . These approaches , based on exact matches between tokens of the queries and documents , suffer from the lexical gap and do not generalize well ( Berger et al. , 2000 ) . By contrast , approaches based on neural networks allow learning beyond lexical similarities , resulting in state-ofthe-art performance on question answering benchmarks , such as MS-MARCO ( Nguyen et al. , 2016 ) or NaturalQuestion ( Kwiatkowski et al. , 2019 ) . As training neural networks requires large amount of data , their strong retrieval results were possible for domains and applications where large training datasets are available . In the case of retrieval , creating these datasets requires manually matching queries to the relevant documents in the collection . This is hardly possible when the collection contains millions or billions of element , resulting in many scenarios where only a few in-domain examples , if any , are available . A potential solution is to train a dense retriever on a large retrieval dataset such as MS MARCO , and then apply it to new domains , a setting refered to as zero-shot . Unfortunately , in this setting dense retrievers are often outperformed by classical methods based on term-frequency , which do not require supervision ( Thakur et al. , 2021 ) . Thus , a natural alternative to transfer learning is unsupervised learning , which raises the following question : is it possible to train dense retrievers without supervision , and match the performance of BM25 ? Training dense retrievers without supervision can be achieved by using a pretext task that approximates retrieval . Given a document , one can generate a synthetic query and then train the network to retrieve the original document , among many others , given the query . The inverse Cloze task ( ICT ) , proposed by Lee et al . ( 2019 ) to pre-train retrievers , uses a given sentence as query and predict the context surrounding it . While showing promising results as pre-training ( Chang et al. , 2020 ; Sachan et al. , 2021a ) , this approach still lags behind BM25 when used as a zero-shot retriever . ICT is strongly related to contrastive learning ( Wu et al. , 2018 ) , which has been widely applied in computer vision ( Chen et al. , 2020 ; He et al. , 2020 ) . In particular , computer vision models trained with the latest contrastive learning methods lead to features well suited to retrieval ( Caron et al. , 2021 ) . In this work , we thus propose to revisit how well contrastive learning performs to train dense retrievers without supervision . We want to investigate how much the recent developments from computer vision , such as MoCo , can improve dense retrievers . Here , our goal is not to develop new techniques , but to determine how far we can go by pushing existing methods to their limits . In this paper , we thus make the following contributions . First , we show that contrastive learning can lead to strong unsupervised retrievers : our model achieves Recall @ 100 results competitive with BM25 on most of the BEIR benchmark . Second , in a few-shot setting , we show that our model benefits from few training examples , and obtains better results than transfering models from large datasets such as MS MARCO . Third , when used as a pre-training method before fine-tuning on MS MARCO , our technique leads to state-of-the-art retrieval performance on the BEIR benchmark . Finally , we perform ablations to motivate our design choices , and show that cropping works better than the inverse Cloze task . 2 RELATED WORK . In this section , we briefly review relevant work in information retrieval , and application of machine learning to this problem . This is not an exhaustive review , and we refer the reader to Manning et al . ( 2008 ) , Mitra et al . ( 2018 ) and Lin et al . ( 2020 ) for a more complete introduction to the field . Term-frequency based information retrieval . Historically , in information retrieval , documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary . Different weighing schemes have been proposed , to determine how important a particular term is to a document in a large dataset . One of the most used weighing scheme is known as TF-IDF , and is based on inverse document frequency , or term specificity ( Jones , 1972 ) . BM25 , which is still widely used today , extends TF-IDF ( Robertson et al. , 1995 ) . A well known limitation of these approaches is that they rely on exact match to retrieve documents . This lead to the introduction of latent semantic analysis ( Deerwester et al. , 1990 ) , in which documents are represented as low dimensional dense vectors . Neural network based information retrieval . Following the successful application of deep learning methods to natural language processing , neural networks techniques were introduced for information retrieval . Huang et al . ( 2013 ) proposed a deep bag-of-words model , in which representations of queries and documents are computed independently . A relevance score is then obtained by taking the dot product between representations , and the model is trained end-to-end on click data from a search engine . This method was later refined by replacing the bag-of-words model by convolutional neural networks ( Shen et al. , 2014 ) or recurrent neural network ( Palangi et al. , 2016 ) . A limitation of bi-encoders is that queries and documents are represented by a single vector , preventing the model to capture fine-grained interactions between terms . Nogueira & Cho ( 2019 ) thus introduced a cross-encoder model , based on a pre-trained BERT model ( Devlin et al. , 2019 ) , which jointly encodes queries and documents . The application of a strong pre-trained model , as well as the cross-encoder architecture , lead to important improvement on the MS-MARCO benchmark ( Bajaj et al. , 2016 ) . The methods described in the previous paragraph were applied to re-rank documents , which were retrieved with a traditional IR system such as BM25 . Gillick et al . ( 2018 ) first studied whether continuous retrievers , based on bi-encoder neural models , could be viable alternative to re-ranking . In the context of question answering , Karpukhin et al . ( 2020 ) introduced a dense passage retriever ( DPR ) based on the bi-encoder architecture . This model is initialized with a BERT network , and trained discriminatively using pairs of queries and relevant documents , with hard negatives from BM25 . Xiong et al . ( 2020 ) further extended this work by mining hard negatives with the model itself during optimization , and trained on the MS-MARCO dataset . Once a collection of documents , such as Wikipedia articles , is encoded , retrieval is performed with a fast k-nearest neighbors library such as FAISS Johnson et al . ( 2019 ) . To alleviate the limitations of bi-encoders , Humeau et al . ( 2019 ) introduces the poly-encoder architecture , where documents are encoded by multiple vectors . Similarly , Khattab et al . ( 2020 ) proposes the ColBERT model , which keeps a vector representation for each term of the queries and documents . To make the retrieval tractable , the term-level function is approximated to first retrieve an initial set of candidates , which are then re-ranked with the true score . In the context of question answering , knowledge distillation has been used to train retrievers , either using the attention scores of the reader of the downstream task as synthetic labels ( Izacard & Grave , 2021 ) , or the relevance score from a cross encoder ( Yang & Seo , 2020 ) . Luan et al . ( 2020 ) compares , theoretically and empirically , the performance of sparse and dense retrievers , including bi- , cross- and poly-encoders . Dense retrievers , such as DPR , can lead to indices that weigh close to 100 GB when encoding document collections such as Wikipedia . Izacard et al . ( 2020 ) shows how to compress such indices , with limited impact on performance , making them more practical to use . Self-supervised learning for NLP . Following the success of word2vec ( Mikolov et al. , 2013 ) , many self-supervised techniques have been proposed to learn representation of text . Here , we briefly review the ones that are most related to our approach : sentence level models and contrastive techniques . Jernite et al . ( 2017 ) introduced different objective function to learn sentence representations , including next sentence prediction and binary order prediction . These objectives were later used in pre-trained models based on transformers , such as BERT ( Devlin et al. , 2019 ) and AlBERT ( Lan et al. , 2019 ) . In the context of retrieval , Lee et al . ( 2019 ) introduced the inverse cloze task ( ICT ) , whose purpose is to predict the context surrounding a span of text . Guu et al . ( 2020 ) integrated a bi-encoder retriever model in a BERT pre-training scheme . The retrieved documents are used as additional context in the BERT task , and the whole system is trained end-to-end in an unsupervised way . Similarly , Lewis et al . ( 2020 ) proposed to jointly learn a retriever and a generative seq2seq model , using self-supervised training . Chang et al . ( 2020 ) compares different pre-training tasks for retrieval , including the inverse cloze task . Constrastive learning was introduced in computer vision by Wu et al . ( 2018 ) , followed by several modifications to improve the training ( He et al. , 2020 ; Chen et al. , 2020 ; Caron et al. , 2020 ) . In the context of natural language processing , Fang et al . ( 2020 ) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation . Different works augmented the masked language modeling objective with a contrastive loss ( Giorgi et al. , 2020 ; Wu et al. , 2020 ; Meng et al. , 2021 ) . Finally , SBERT ( Reimers & Gurevych , 2019 ) uses a siamese network similar to contrastive learning to learn a BERT-like model that is adapted to matching sentence embeddings . Their formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose to use data augmentation to leverage large unaligned textual corpora . Concurrent to this work , Gao & Callan ( 2021 ) have also shown the potential of contrastive learning for information retrieval ; building on the same observation that both tasks share a similar structure . 3 METHOD . In this section , we describe how to train a dense retriever with no supervision . We review the model architecture and then describe contrastive learning - a key component of its training . The objective of a retriever is to find relevant documents in a large collection for a given query . Thus , the retriever takes as input the set of documents and the query and outputs a relevance score for each document . A standard approach is to encode each query–document pair with a neural network ( Nogueira & Cho , 2019 ) . This procedure requires re-encoding every document for any new query and hence does not scale to large collections of documents . Instead , we propose to use a bi-encoder architecture , where documents and queries are encoded independently ( Huang et al. , 2013 ; Karpukhin et al. , 2020 ) . One can compute the relevance score by taking the dot product ( or cosine similarity ) between the document ’ s representation and the representation of the query . More precisely , given a pair of query q and document d , we encode each of them independently using the same model , fθ , parametrized by θ . The relevance score s ( q , d ) is then the dot product of the resulting representations : s ( q , d ) = 〈fθ ( q ) , fθ ( d ) 〉 . In practice , we use a transformer network for fθ to embed both queries and documents . The representation fθ ( q ) ( resp . fθ ( d ) ) for a query ( resp . document ) is obtained by averaging the hidden representations of the last layer . Following previous work on dense retrieval with neural networks , we use the BERT base uncased architecture and refer the reader to Devlin et al . ( 2019 ) for more details .

Summary: This paper studies pre-training of Siamese Transformer encoders for the zero-shot dense retrieval problem. The author consider contrastive learning loss function to optimize the encoders and investigate variants of ICT for data augmentations. While not much novelty in the techincal sides, the author indeed present solid empirical study on zero-shot retrieval benchmark as well as extension to the few-shot evaluation.