Contrastive Pre-training for Zero-Shot Information Retrieval
Authors: Gautier Izacard
Mathilde Caron
Lucas Hosseini
Sebastian Riedel
Piotr Bojanowski
Armand Joulin
Edouard Grave
Venue: ICLR 2022 Submitted (2022)
Keywords: information retrieval
contrastive pretraining

Abstract: Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking. Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency. Neural retrievers work well on the problems for which they were specifically trained, but they do not generalize as well as term-frequency methods to new domains or applications. By contrast, in many other NLP tasks, conventional self-supervised pre-training based on masking leads to strong generalization with small number of training examples. We believe this is not yet the case for information retrieval, because these pre-training methods are not well adapted to this task. In this work, we consider contrastive learning as a more natural pre-training technique for retrieval and show that it leads to models that are competitive with BM25 on many domains or applications, even without training on supervised data. Our dense pre-trained models also compare favorably against BERT pre-trained models in the few-shot setting, and achieves state-of-the-art performance on the BEIR benchmark when fine-tuned on MS-MARCO.

Under review as a conference paper at ICLR 2022
TOWARDS UNSUPERVISED DENSE INFORMATION RE- TRIEVAL WITH CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking. Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained sate-of-the--rt results on datasets and tasks where large training sets are available. However, they do not transfer well to new domains applications with no training data, and are often outperformed by term-frequency methods such as BM25 which are not supervised. Thus, a natural question is whether it is possible to train dense retrievers without supervision. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers, and show that it leads to strong retrieval performance. More precisely, we show on the BEIR benchmark that our model outperforms BM25 on 8 out of 14 datasets. Furthermore, when few thousands examples are available, we show that fine-tuning our model on these leads to strong improvements compared to BM25. Finally, when used as pre-training before fine-tuning on the MS MARCO dataset, our technique obtains satte-of-the-ann results on the BEIR benchmark.
INTRODUCTION
Document retrieval is the task of finding relevant documents in large collection to answer spe- cific queries. This an important task by itself and a core component of many natural language processing (NIP) problems, such as open domain question answering (Chen et al., 2017) or fact checking (Thorne et al.) 2018). Traditionally, retrieval systems, or retrievers, leverage lexical similari- ties to match queries and documents, using, for instance, TF-ID or BM25 weighting (Robertson) Zaragoza 2009). These approaches, based on exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well (Berger et al. 2000). By contrast, approaches based on neural networks allow learning beyond lexical similarities, resulting in state-of- the-art performance on question answering benchmarks, such as MS MARCO (Nguyen al. 2016)
or NaturalQuestion (Kwiatkowski et al. 2019).
As training neural networks requires large amount of data, their strong retrieval results were possible for domains and applications where large training datasets are available. the case of retrieval, creating these datasets requires manually matching queries the relevant documents in the collection. This is hardly possible when the collection contains millions or billions of element, resulting in many scenarios where only a few in-domain examples, if any, are available. A potential solution is train a dense retriever on a large retrieval dataset such as MS MARCO, and then apply it to new domains, a setting refered to as zero-shot. Unfortunately, in this setting dense retrievers are often outperformed by classical methods based on ferm-frequency, which do not require supervision (Thakur et al. 2021).
Thus, a natural alternative to transfer learning is unsupervised learning, which raises the following question: is it possible to train dense retrievers without supervision, and match the performance of BM25? Training dense retrievers without supervision can be achieved by using a pretext task that approximates retrieval. Given document, one can generate synthetic query and then train the network to retrieve the original document, among many others, given the query. The inverse Cloze task (ICT), proposed by Lee et al. (2019) to pre-train treitrain retrievers, predict the context surrounding it. While showing promising results as pre- tre-training (Chang et al. 2020; Sachan et al. 2021a), this approach still lags behind BM25 when used zero-shot retriever.
given sentence query and

---

Under review as a conference paper at ICLR 2022
ICT is strongly related to contrastive learning (Wu et al. 2018), which has been widely applied in computer vision (Chen et al., 2020 He et al. 2020). In particular, computer vision models trained with the latest contrastive learning methods lead to features well suited to retrieval (Caron et al.) 2021). In this work, we thus propose to revisit how well contrastive learning performs to train dense retrievers without supervision. We want investigate how much the recent developments from computer vision, such as MoCo, can improve dense retrievers. Here, our goal is not to develop new techniques, but to determine how far we can go by pushing existing methods to their limits.
In this paper, we thus make the following contributions. First, we show that contrastive learning can lead to strong unsupervised retrievers: our model achieves @ 100 results competitive with BM25 on most of the BEIR benchmark. Second, in few-shot setting, we show that our model benefits from few training examples, and obtains better results than transfering models from large datasets such as MS MARCO. Third, when used as a pre-training method before fine-tuning on MS MARCO, our technique leads to state-of-the-an retrieval performance on the BEIR benchmark. Finally, we perform ablations to motivate our design choices, and show that cropping works better than the inverse Cloze task.
2 RELATED WORK
In this section, we briefly review relevant work in information retrieval, and application of machine learning to this problem. This is not an exhaustive review, and refer the reader to Manning et al.) (2008). Mitra et al. (2018) and Lin et al. (2020) for a more complete introduction to the field.
Term- -requeery based information retrieval. Historically, in information retrieval, documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary. Different weighing schemes have been proposed, to determine how important a particular term is to a document in a large dataset. One of the most used weighing scheme is known as TF-ID, and is based on inverse document frequency, or term specificity (Jones) 1972). BM25, which is still widely used today, extends TF-IDF (Robertson et al. 1995). A well known limitation of these approaches is that they rely on exact match to retrieve documents. This lead to the introduction of latent semantic analysis (Deerwester et al. 1990), in which documents are represented as low dimensional dense vectors.
Neural nefwork based information retrieval. Following the successful application of deep learn- ing methods to natural language processing, neural networks techniques were introduced for informa- tion retrieval. Huang et al. (2013) proposed deep bag-of-words model, in which representations of queries and documents are computed independently. A relevance score is then obtained by taking the dot product between representations, and the model is trained end-to-end on click data from a search engine. This method was later refined by replacing the bag-of-words model by convolutional neural networks (Shen et al. 2014) or recurrent neural network (Palangi et al. 2016). A limitation of bi-encoders is that queries and documents are represented by single vector, preventing the model to capture fine-grained interactions between terms. NNouueira & Cho (2019) thus introduced a cross-encoder model, based pre-trained BERT model (Devlin al.) 2019), which jointly encodes queries and documents. The application of a strong pre-trained model, as well as the cross-encoder architecture, lead important improvement on the MS-MARCO benchmark (Bajaj et al. 2016).
The methods described in the previous paragraph were applied to re-rank documents, which were retrieved with traditional IR system such as BM25. Gillick et al. (2018) first studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking. In the context of question answering, Karpukhin al. (2020) introduced a dense passage retriever (DPR) based on the bi-encoder architecture. This model is initialized with BERT network, and trained discriminatively using pairs of queries and relevant documents, with hard negatives from BM25. XXiong et al. (2020) further extended this work by mining hard negatives with the model itself during optimization, and trained on the MS-MARCO dataset. Once a collection of documents, such Wikipedia articles, is encoded, retrieval is performed with a fast k-nearest neighbors library such as FAISS Johnson et al. (2019). To alleviate the limitations of bi-encoders, Hummau et al. (2019) introduces the polddencoder architecture, where documents are encoded by multiple vectors. Similarly, Khattab et al. (2020) proposes the ColBERT model, which keeps a vector representation for each term of the queries and documents. To make the retrieval tractable, the term-level function
2

---

Under review as a conference paper at ICLR 2022
is approximated to first retrieve an initial set of candidates, which are then re-ranked with the true score. In the context of question answering, knowledge distillation has been used to train retrievers. either using the attention scores of the reader of the downstream task synthetic labels (Izacard & Grave, ). or the relevance score from a cross encoder (Yang & Seo 2020). Luan et al. (2020) compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross- and poly-encoders. Dense retrievers, such as DPR, can lead to indices that weigh close to 100 GB when encoding document collections such as Wikipedia, [Izcard et al. (2020) shows how to compress such indices, with limited impact on performance, making them more practical to use.
Self-supervieed learning for NLP. Following the success of word2vec (Mikolov al. 2013), many self-supervised techniques have been proposed to learn representation of text. Here, we briefly review the ones that are most related to our approach: sentence level models and contrastive techniques. Jernite et al. (2017) introduced different objective function learn sentence representations, including next sentence prediction and binary order prediction. These objectives were later used in pre-trained models based on transformers, such as BERT (Devlin et al., 2019) and AIBERT (Lan et al. 2019). In the context of retrieval, [ee et al. (2019) introduced the inverse cloze task (ICT), whose purpose is to predict the context surrounding span of text. Guu al. (2020) integrated bi-encoder retriever model in a BERT pre-training scheme. The retrieved documents are used as additional context in the BERT task, and the whole system trained end-to-end an unsupervised way. Similarly, Lewis et al. (2020) proposed to jointly learn a retriever and generative seq2seq model, using self-supervised training. Chang al. (2020) compares different pre-training tasks for retrieval, including the inverse cloze task. Constrastive learning was introduced in computer vision by Wu et al. (2018), followed by several modifications to improve the training (He et al.] [020 Chen et al.] [220;] Caron al., 2020). In the context of natural language processing, Fang et al. (2020) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation. Different works augmented the masked language modeling objective with contrastive loss (Giorgi et al. 2020 Wu et al. 2020
Meng et al.] 2021). Finally, SBERT (Reimers & Gurevych| (019) uses a siamese network similar to contrastive learning learn BERT-like model that is adapted to matching sentence embeddings. Their formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose
data augmentation to leverage large unaligned textual corpora. Concurrent to this work, Gao & Callan ( have also shown the potential of contrastive learning for information retrieval; building on the same observation that both tasks share a similar structure.
METHOD
In this section, we describe how to train a dense retriever with no supervision. We review the model architecture and then describe contrastive learning a key component of its training.
The objective of a retriever is to find relevant documents in a large collection for a given query. Thus, the retriever takes as input the set of documents and the query and outputs a relevance score for each document. A standard approach is to encode each query-document pair with neural network (Nogueira & Cho, 2019). This procedure requires re-encoding every document for any new query and hence does not scale large collections of documents. Instead, we propose to use a bi-encoder architecture, where documents and queries are encoded independently (Huang et al. 2013) Karpukhin et al.] 2020). One can compute the relevance score by taking the dot product (or cosine similarity) between the document's representation and the representation of the query. More precisely, given a pair of query q and document d, we encode each of them independently using the same model, fe. parametrized by 0. The relevance score s(g. d) is then the dot product of the resulting representations:
s(g. d) = (folg).
In practice, we use a transformer network for fe to embed both queries and documents. The representation fo(g) (resp. fo(d) for query (resp. document) is obtained by averaging the hidden representations of the last layer. Following previous work on dense retrieval with neural networks, we use the BERT base uncased architecture and refer the reader to Devlin al. (201)) for more details.
3

---

Under review as a conference paper at ICLR 2022
3.1
UNSUPERVISED TRAINING ON UNALIGNED DOCUMENTS
In this section, we describe our unsupervised training pipeline. We briefly review the loss function traditionally used in contrastive learning. We then discuss obtaining positive pairs from single text document, critical ingredient for this training paradigm.
3.1.1 CONTRASTIVE LEARNING
Contrastive learning is an approach that relies on the fact that every document is, in some way, unique. This signal is the only information available in the absence of manual supervision. The resulting algorithm learns by discriminating between documents, using a contrastive loss (Wu et al. 2018). This loss compares either positive (from the same document) negative (from different documents) pairs of document representations. More formally, given positive pair of representations (q, k+) and set of negative pairs (q, k.)i the contrastive InfoNCE loss is defined
(xp(s(q,k,)/3
C(q,k4) = En= exp(s(q,
where r a temperature parameter. This loss encourages the relevance score of similar examples to be high and that of dissimilar examples to be low. Another interpretation of this loss function is the following: given the query representation q. the goal is to recover, or retrieve, the representation h4 corresponding to the positive document, among all the negatives k, In the following, refer the left-hand side representations in the score as queries and the right-hand side representations as keys.
3.1.2 BUILDING POSITIVE PAIRS FROM SINGLE DOCUMENT
crucial element of contrastive learning how to build positive pairs from a single input. In computer vision, this step relies on applying two independent data augmentations to the same image, resulting in two "views" that form positive pair (Wu et al., 2018; Chen et al. 2020p. While we consider similar independent text transformation, also explore dependent transformations designed to reduce the correlation between views.
Inverse Cloze Task is data augmentation that generates two mutually exclusive views of document, introduced in the context of retrieval by Lee et al. (2019). The idea is to take a span of tokens to form one view, and its complementary to form the other view. More precisely, given sequence of text (w0,.... Wa), ICT samples a span b where S a < S n,, uses the tokens of the span as the query and the rest of the tokens the document (or key). In the original implementation by Lee et al. (2019) the span corresponds to a sentence, and is kept in the document 10% of the time encourage lexical matching.
Independent cropping is a common independent data augmentation used for images where views are generated independently by cropping the input. In the context of text, cropping is equivalent to sampling a span of tokens. This strategy thus samples independently two spans from a document to form positive pair. As opposed to the inverse Cloze task, in cropping both views of the example correspond to contiguous subsequence of the original data. A second difference between cropping and ICT is the fact that the task is symmetric: both the queries and documents follow the same distribution. Independent cropping encouraging the network to learn exact matches between the query and document, in a way that is similar lexical matching methods like BM25. In practice, we can either fix the length of the span for the query and the key, or sample them.
also lead to overlap between the two views of the data, hence
Additional data augmentation. Finally, we also consider additional data augmentations such as random word deletion, random word replacement or random word masking. We use these perturbation in addition to random cropping.
3.1.3 BUILDING LARGE SET OF NEGATIVE PAIRS
An important aspect of contrastive learning is to maintain large set of negative pairs. Most standard frameworks differ from how the negatives are handled, and we briefly describe two of them, SimCLR and MoCo, that we use in this work.
4

---

Under review as a conference paper at ICLR 2022
Negative pairs within batch. A first solution, called SimCLR (Chen et al.) 2020 is generate the negative pairs by using the other examples from the same batch: each example in a batch is transformed twice to generate positive pairs, and we generate negative pairs by using the views from the other examples in the batch. In that case, the gradient is back-propagate representations of both the queries and the keys. A downside of this approach is that it requires extremely large batch sizes to work well. This technique was also considered when using ICT to pre-train retrievers by Lee et al. (2019).
through the
Negative pairs across batches. An alternative approach is to store representations from previous batches in a queue and use them as negative examples in the loss (Wu et al. 2018). This allows for smaller batch size but slightly changes the loss by making it asymmetric between "queries" (one of the view generated from the elements of the current batch), and "keys" (the other generated views, as well as the elements stored in the queue). Gradient is only backpropagated through the "queries", and the representation of the "keys" are considered as fixed. In practice, the features stored in the queue from previous batches comes form previous iterations of the network. This leads to drop of performance when the network rapidly changes during training. Instead, He et al.] (2020) proposed to generate representations of keys from second network that updated more slowly. This approach, called MoCo, considers two networks: one for the keys, parametrized by Og, and one of the query, parametrized by fy. The parameters of the query network are updated with backpropagation and stochastic gradient descent, like in SimCLR, while the parameters of the key network, or Momentum encoder, is updated from the parameters of the query network by using a exponential moving average:
0₂ + m0 + (1 - mm))0-
where is the momentum parameter that takes its value in [0, 1].
4 EXPERIMENTS
In this section, we empirically evaluate contrastive learning way to train dense retrievers. First, we compare our best models to the state-of-the on competitive retrieval benchmarks for both zero-shot and few-shot retrieval. Then, we provide ablation study motivate the technical choices leading to our best retriever, shedding some light on the important ingredients of our approach. We give technical details about models in Appendix AA.
4.1 BEIR BENCHMARK
The BEIR benchmark, introduced by Thakur et al. (2021), contains 18 retrieval datasets with a focus on diversity. Each dataset is made of a set of queries, the corresponding relevant documents and a large collection of documents to retrieve from. These datasets correspond to nine different retrieval tasks, such as fact checking, question answering, entity retrieval or citation prediction. Moreover, the benchmark covers large number of domains, such as Wikipedia, news articles, scientific publications or social media posts. Finally, there is also diversity in terms documents and queries length, with queries ranging from 3 to 190 words and documents from 11 to 630 words. Most datasets from the BEIR benchmark do not contain a training set, and the focus of the benchmark is zero-shot retrieval. However, most machine learning based retrievers are still trained supervised data, such as the large scale retrieval dataset MS-MARCO (Bajaj et al. 2016). Following standard practice, we report metrics this benchmark, the nDGC@ 10 and the Recall 100. These two metrics complementary and both important: nDCG focuses on the ranking of the top 10 retrieved documents, and is good at evaluating documents returned to human, for example in a search engine. On the other hand, the Recall @ 100 is relevant to evaluate retrievers that are included in marchine learning systems, such as question answering. Indeed, such models can process hundreds of documents, and ignore the ranking of these documents [Izaaard & Grave 2020).
4.2 BASELINES
First, we compare our approach to the BM25 ranking function, which does not require any training data. However, the relevance of individual terms depends on the inverse document frequency, which is computed over each document collection independently. In that sense, the ranking function used in the BM25 baseline is adapted each dataset on which retrieval is performed. We also compare
5

---

Under review as a conference paper at ICLR 2022
BM25
ColBERT
SPARTA
docTSQuery
DPR
ANCE
GenQ
TAS-B
Ours
TREC-COVID
65.6
67.7
53.8
71.3
33.2
65.4
61.9
48.1
61.0
NFCorpus
32.5
30.5
30.1
32.8
18.9
23.7
31.9
31.9
33.2
NQ
32.9
52.4
39.8
39.9
47.4
44.6
35.8
46.3
50.2
HoipoiQA
60.3
59.3
49.2
58.0
39.1
45.6
53.4
58.4
64.5
FIQA
23.6
31.7
19.8
29.1
11.2
29.5
30.8
30.0
28.8
ArguAna
31.5
23.3
27.9
34.9
17.5
41.5
49.3
42.9
46.0
Tóuche- 2020
36.7
20.2
17.5
34.7
13.1
24.0
18.2
16.2
25.9
Quora
78.9
85.4
63.0
80.2
24.8
85.2
83.0
83.5
85.4
CQADupStack
29.9
35.0
25.7
32.5
15.3
29.6
34.7
31.4
32.2
DBPedia
31.3
39.2
31.4
33.1
26.3
28.1
32.8
38.4
38.8
Scidoes
15.8
14.5
16.2
7.7
14.3
14.9
16.0
Fever
75.3
77.1
59.6
71.4
56.2
66.9
66.9
70.0
77.7
Climate FEVER
21.3
18.4
8.2
20.1
14.8
19.8
17.5
22,8
23,4
SciFact
66.5
67.1
58.2
67.5
31.8
50.7
64.4
64.3
68.1
Average
43.0
44.4
35.5
44.4
25.5
40.5
42.5
42.8
46.5
Best
5
2
Table 1: BEIR Benchmark. We report nDCG@ 10 on the test sets of the datasets from the BEIR benchmark for bi-encoder methods without re-ranker. We also report the average ("Average") and number of datasets where a method is the best ("Best on") over the entire BEIR benchmark (excluding three datasets because their licence). We evaluate retrievers after pre-training on unsupervised data with our contrastive learning and finetuning on MS-MARCO ("Ours")). Bold is the best overall.
to machine learning based retrievers, which can be classified in three categories: sparse, dense and iate-interaction For sparse methods, we compare doc''Squery (Nogueira et al. 2019), which is document expansion technique based on the T5 model (Raffel et al. 2019)). The second sparse model we use as baseline is SPARTA (Zhao et al., 2020 which computes sparse representations of documents using BERT model. More precisely, the ranking function of SPARTA is based on interactions between terms of the document and the query. Because the representations of the query terms non-contextualized, the scores be pre-computed when indexing the database. For dense methods, we use DPR (Karpukhin et al.) 2020) and ANCE (Xiong et al.) 2020) as baselines, which bi-encoder models trained supervised data such NaturalQuestions MS-MARCO. We also include the TAS-B technique (Hofstätter et al. (021). which performs distillation from a cross-encoder bi-encoder model. Finally, compare to GenQ, which uses generative query model synthetic query-document pairs, and is applied to each dataset. Thus, different GenQ model is used for each dataset of the BEIR benchmark. For tate-interactio methods, we use ColBERT (Khattabl et al. 2020), which computes pairwise scores between contextualized representations of the query and document. To make this approach scalable, this scoring function is only applied on documents obtained by applying approximation of the full score. Interestingly, it should be noted that BM25 a very competitive baseline the zero-shot setting, outperforming all dense retrievers on average over the BEIR benchmark. Only the re-ranking models and the query expansion (docT5query), which also based on BM25, obtain better results than BM25.
4.3 RESULTS
In T we report the nDCG@ 10 on the BEIR benchmark for different retrievers. The correspond- ing Recall @ 100 performances can be found in Table appendix. We individually report results on each dataset as well as the average ("Avg.") over 14 datasets of the BEIR Benchmark (excluding for license reasons), We omit retrievers that use re-ranker during inference since it is orthogonal to our focus and can be applied to any method.
We observe that contrastive pre-training leads to strong performance among retrievers: model obtains state-of-the art performance when finetumed on the MS- MARCO dataset. should be noted that finetuning procedure MS-MARCO is simpler than for other dense retrievers: a simple strategy for negative mining and do not use distillation. Our model would probably also benefits from improvements proposed by these retrievers, but this is beyond the scope of this paper.

---

Under review as a conference paper at ICLR 2022
finetuning
SciFact
NFCorpus
FiQA
Table 2:
Few-shot retrieval. nDCG@ 10 on the test sets of benchmarks when finetuned on small training set. We compare BERT and our pre-training with and without the finetuning step on MS-MARCO. Note that our unsu- pervised pre-training alone leads to better performance than BERT finetuned on MS-MARCO.
Dataset size
809
2,590
5,500
BM25
66.5
32.5
23.6
BERT
75.2
29.9
26.1
Ours
84.0
33.9
31.5
BERT
MS-MARCO
80.9
33,2
30.9
Ours
MS-MARCO
84.1
34.9
34.4
REALM
SimCSE
BM25
Ours
Recall@)100
50
Trec-COVID
Touche-2020
NFCorpus
Soidocs
DBPedia
Climate- Climate-fever fever
COaDupStack
FIQA
MSMARCO
HotpoiQA
NQ
ArguAna
Scifact
Fever
Quora
Avg.
Figure 1: Zero-shot retrieval. We compare our pre-training without using any annotated data to REALM, SimCSE and BM25. For SimCSE we report results the model using RoBERTa large. REALM uses annotated entity recognition data for training. We highlight that our unsupervised pretraining on par with BM25 but on datasets.
We also illustrate the benefit of our dense retriever compared to BM25 by evaluating it in a few shot setting, where have access a small number training queries and relevant documents to finetune locally the retriever. This setting is common in practice, and lexical based methods, like BM25, cannot benefit leverage the small training sets adapt its weights. In Table [ we report the nDCG@ 10 results on the three datasets from the BEIR benchmark associated with the smallest training sets. These training sets range from 809 to almost 5,500 queries. We observe that on these small datasets, our pre-training leads to significantly better results than BERT pre-training only, even when BERT model finctuned on MS-MARCO, Our pre- pre-trained model also outperforms BM25, showing the advantage of dense retriever over lexical methods in the low-shot setting.
Finally, we compare the performance fully unsupervised models, i.e., without finetuning on MS-MARCO other annotated data. In Table 8 we report the retrieval performance of model on two standard question answering benchmarks: NaturalQuestions (Kwiatkowski et al.) 2019) and TriviaQA (Joshi et al. 2017). In Figure we report the performance of unsupervised models the BEIR benchmark. Interestingly, we observe that, in this setting, our pre-training can lead to a model with competitive retrieval performance compared to BM25. Our approach outperforms previously proposed unsupervised dense retrievers. On question answering benchmarks it also outperforms strong BM25 baseline (Ma et al. 202T by 3% for the Recall 100, and is on par with BM25 on TriviaQA for the Recall @ 100. On datasets of the BEIRbenchmark, we observe that our unsupervised retriever is competitive with BM25 on every datasets, but TREC-COVID and Tóuche-2020. These results show the potential of contrastive learning to develop fully unsupervised dense retrievers.
5 ABLATION STUDIES
In this section, we investigate the influence of different design choices on our method. In these ablations, all the models are pre-trained Wikipedia for 200k gradient steps, with batch size of
7

---

Under review as a conference paper at ICLR 2022
2,048 (on 32 GPUs). Each fine-tuning MS-MARCO takes 20k gradient steps with batch size of 512 (on 8 GPUs), using AdamW and no hard negatives.
MoCo vs. SimCLR. First, we compare the two contrastive pre-training methods: MoCo and SimCLR. As in SimCLR, the number of negative examples is equal to the batch size, we train models with a batch size of 4,096 and restrict the queue in MoCo the same number of elements. This experiment measures the effect of using of momentum encoder for the keys instead of the same network as for the queries. Using a momentum also prevents from backpropagating the gradient through the keys. We report results, with fine-tuning MS-MARCO in Table We observe that the difference of performance between the two methods small, especially after fine-tuning on MS-MARCO. This validates choice of MoCo our contrastive learning framework, since scales to a larger number of negative examples without the need to increase the batch size.
NFCorpus
NQ
FIQA
ArguAna
Quora
DBPedia
SeilDoes
FEVER
AVG
MoCo SimCLR
31.4
43.8
26.3
38.4
83.5
40.1
73.6
42.8
29.5
39.5
23.6
38.3
83.7
36.1
16.1
73.4
41.6
Table 3: MoCo SimCLR. In this table, report nDCG@ 10 the BEIR benchmark for SimCLR and MoCo, after fine-tuning on the MS-MARCO dataset.
Number of negative examples. Next, study the influence of the number of negatives used in the contrastive loss, by varying the queue size of the MoCo algorithm. We consider values ranging from 2,048 131,072, and report results Figure We see that on average over the BEIR benchmark, increasing the number of negatives leads to better retrieval performance, especially in the unsupervised setting. However, we note that this effect is not equally strong for all datasets.
MS MARCO
TREC COVID
NFCorpus
NaturalQuestions
HotpotQA
40
50
32
50
70
30
40
28
30
40
60
20
30
26
20
30
40
50
10
20
22
10
30
241
g22
213
244
2
2
222 2 g24
211
22
2
2 222
2
24
2 2
FiQA
ArguAna
Touche-2020
Quora
DBPedia
45
85
25
40
20
80
40
20
16
75
30
15
35
14
70
20
10
30
65
241
22
213
244
2
207
21
222
nlو all g!7
2 2l2
22
2
2
2
20 22 2
24
2/2 2
SCIDOCS
FEVER
CLIMATE-FEVER
SciFacts
Average
18
80
25
70
45
16
70
20
65
40
14
12
60
15
60
35
30
10
50
10
55
24t 222
21
212 233
2!7
241 g
24
2
211 212
Without MSMARCO
With MSMARCO
Figure 2: Impact of the number of negatives. We report nDCG@ 10 as a function of the queue size, with and without fine-tuning on MS-MARCO. We report numbers using MoCo framework where the keys for the negatives are computed with the momentum encoder and stored in a queue.
Data augmentations. Third, we compare different ways to generate pairs positive examples from single document chunk of text. In particular, we compare random cropping, which leads to pairs with overlap. and the inverse cloze task, which was previously considered to pre-train retrievers. Interestingly,
shown in Table 4 the random cropping strategy strongly outperforms the inverse cloze task in our setting. We argue that overlap between the two views the data is beneficial, it allows the network leam lexical matching, similarly to BM25. We also investigate
8

---

Under review as a conference paper at ICLR 2022
whether additional data perturbations, such random word deletion replacement, are beneficial for retrieval. After fine-tuning on MS-MARCO, seems that such data perturbations slightly degrade the performance of dense retrievers.
NFCorpus
NQ
ArguAna
Quora
DBPedia
SciDocs
FEVER
Overall
ICT
30.5
37.5
32.6
82.3
33.1
14.6
67.9
38.8
Crop
31.5
44.1
39,5
84.2
39.6
16.3
74.8
43.4
Crop + delete
31.6
42.3
39.2
84.6
39.7
16.2
71.8
42.6
Crop replace
30.5
43.2
42.6
84.6
39.2
16.0
73.8
42.8
Table 4: Impact of data augmentions. nDCG@ 10 after fine-tuning on MS-MARCO.
Training data. Finally, we study the impact the pre-training data on the performance of our retriever, by training on Wikipedia, CCNet mix of both sources of data. We report results in T and observe that there is no clear winner between the two data sources. Unsurprisingly, pre-training Wikipedia leads to better performance datasets for which documents also come from Wikipedia, such as NaturalQuestions or FEVER. On the other hand, on datasets from different domains than Wikipedia such FiQA ArguAna, training on the better results. To get the best of both worlds, we consider to strategies to mix the two data sources. In the "50/50%" strategy, examples are sampled uniformly across domain, meaning that half the batches are from Wikipedia and the other half from CCNet. In the "uniform" strategy, examples are sampled uniformly over the union of the dataset. Since CCNet is significantly larger than Wikipedia, this means that most of the batches are from CCNet.
diverse CCNet data leads to
NFCorpus
NQ
FiQA
ArguAna
Quora
DBPedia
SciDoes
FEVER
Overall
Wiki
31.5
44.1
26.8
39.5
84.2
39.6
16.3
74.8
43.4
CCNet
31.8
39.9
31.6
43.3
84.5
36.1
16.0
66.4
41.8
Uniform
32.6
40.6
31.0
42.3
84.8
37.2
15.9
66.5
42.6
50/50%
32.5
41.4
29.4
40.4
84.6
39.0
16.3
75.4
43.9
Table 5: Training data. We report nDCG@ 10 after fine-tuning on MS-MARCO.
Impact of fine-tuning on MS-MARCO. opposed the impact of our fine-tuning MS-MARCO, to the BERT base uncased model. We report results in Table 6 and observe that when applied to BERT, our fine-tuning leads results that are lower than the state-of-the-ard Hence, we believe that most of the improvement compared to the state-of-the-arr retrievers can be attributed to our contrastive pre-training strategy.
To isolate the impact of our pre-training method as
propose apply the fine-tuning
NFCorpus
NQ
FIQA
ArguAna
Quora
DBPedia
SciDocs
FEVER
Overall
BERT
28.2
44.6
25.9
35.0
84.0
34.4
13.0
69.8
42.0
Ours
33.2
50.2
28.8
46.0
85.4
38.8
16.0
77.7
46.5
Table Finetuning. We report nDCG@ 10 after fine-tuning BERT and our model on MS-MARCO.
DISCUSSION
In this work, we propose to explore the limits of contrastive pre-training to learn dense text retrievers. We use the MoCo technique, which allows us to train with a large number negative examples. We make several interesting observations: first, while the inverse Cloze task has been introduced in the context of retrieval, we show that independent random cropping leads to better results. Second, show that neural networks trained without supervision using contrastive learning exhibits good retrieval performance. Based on these observations, we trained a model (albeit not tfee---rrt--tttttt performance the BEIR benchmark when fine-tuned MS-MARCO. that obtains

---

Under review as a conference paper at ICLR 2022
REFERENCES
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms mareo: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611 11696,,,,6,, 2016.
Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. Bridging the lexical chasm: statistical approaches findiigg In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 192-199, 2000.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.00993 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sule-vvie vision transformers. arXiv preprint arXiv:2104.14290 2021.
Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-basee based large-scale retrieval. arXiv preprint arXiv:2002.03912 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia answer
open-domain questions. In Proc. ACI., 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on muchine learning, pp. 1597-1607. PMLR, 2020.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41 (6):391-407, 1990.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766 2020.
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108, 05540, 2021.
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. End-to-end retrieval continuous space. arXiv preprint arXiv: 10f::::11110000000 2018.
John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.0363 03666999.,,,9, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wes Wei Chang. Realm: Retrieval- augmented language model pre-training- arXiv preprint arXiv:2002.08909 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Sebastian Hofstätter, Sheng- ciently teaching an effective dense retriever with balanced topic aware sampling. arXiv preprint (oXiv:200400666. 2021.
Sheng-Chich Chich Lin, Jheng- Hong Yang. Jimmy Lin, and Allan Hanbury. Effi-
Po-Sen Huang, Xiaodong He, Jianfeng Gao. Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338, 2013.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Transformer architectures and pre- pre-training strategies for fast and accurate multi multi-sentence scoring. arXiv preprint arXiv:1905.01 2019.
10

---

Under review as a conference paper at ICLR 2022
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020.
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. In International Conference on Learning Representations, 2021. URL https
7/openrevi ew net./ www.oormdddTTe.. - 6wysdb
Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave, memory efficient baseline for open domain question answering. arXiv preprint crXiv:0012.15156. 2020.
Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsuper- vised sentence representation learning, arXiv preprint arXiv: /v0vv:/705.00557 2017.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions Big Data, 2019.
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. ACL, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04996, 2020.
Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiv:2102.11600, 2021.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1909. 1942, 2019.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proc. ACL, 2019.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing, arXiv preprint arXiv:2006.15000 2020.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained transformers for text ranking: Bert and beyond. arXiv preprint arXiv:2010.06440 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. arXiv preprint arXiv:2005.00181, 2020.
Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. A replication study of dense passage retriever, 2021.
Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan. Introduction to information retrieval. Cambridge university press, 2008.
11

---

Under review as a conference paper at ICLR 2022
Yu Meng, Chenyan Xiong, Payal Bajaj. Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Coco-Im: Correcting and contrasting text sequences for language model pretraining. arXiv preprint arXiv:2102.08473, 2021.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. pp. 3111-3119, 2013.
Bhaskar Mitra, Nick Craswell, et al. An introduction to neural information retrieval. Foundations and Trends@ in Information Retrieval, 13(1):1-126, 2018.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprchension dataset. In CoCo@ NIPS, 2016.
Rodrigo Nogueira and Kyunghyun Cho. 191.00090 2019.
Passage re-ranking with bert. arXiv preprint
Rodrigo Nogueira, Jimmy Lin, and Al Epistemic. From doc2query to doctttttquery. Online preprint, 2019.
Hamid Palangi. Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song. and Rabab Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(4))694707, 2016.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. (at.:/99000083 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond.
Now Publishers Inc, 2009.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Bancoock-Bcaulice Mike Gatford, et al. Okapi at TREC-3. NIST Special Publication Sp, 1995.
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L Hamilton, and Bryan Catanzaro. End-to-end training of neural retrievers for open-domain question answering. arXiv preprint arXiv:2101.00408 2021a.
Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering, 2021b.
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd international conference world wide web, pp. 373-374, 2014.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 2021.
James Thome,, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05330 05355 2018.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary. Francisco Guzmán, Armand Joulin, and Edouard Grave CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, 2020.
Zhirong WU, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018
12

---

Under review as a conference paper at ICLR 2022
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:0012.1566,, 2020.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.0088 0000008 2020.
Sohee Yang and Minjoon Seo. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010.10999, 2020.
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lec. Sparta: Efficient open- pomn-n question answering via sparse transformer matching retrieval. arXiv preprint arXiv:2009. 13013, 2020.
13

---

Under review as a conference paper at ICLR 2022
BM25
ColBERT
SPARTA
docTSQuery
ANCE
GenQ
TAS-B
Ours
TREC-COVID
49.8
46.4
40.9
54.1
21.2
45.7
45.6
38.7
42.2
NFCorpus
25.0
25.4
24.3
25.3
28.0
28.0
28.6
NQ
76.0
91.2
78.7
83.2
88.0
83.6
86.2
90.3
92.4
HotpoiQA
74.0
74.8
65.1
70.9
59.1
57.8
67.3
72.8
78.6
FIQA
53.9
60.3
44.6
59.8
34.2
58.1
61.8
59.3
59.6
ArguAna
91.4
89.3
97.2
75.1
97.8
94.2
97.7
Touché- 2020
53.8
43.9
38.1
55.7
30.1
45.8
45.1
43.1
30.8
CQADupStack
60.6
62.4
52.1
63.8
40.3
57.9
65.4
62.2
63.9
Quora
97.3
98.9
89.6
98.2
47.0
98.7
98.8
98.6
98.9
DBPedia
39.8
46.1
41.1
36.5
34.9
31.9
43.1
49.9
53.6
SCIDOCS
35.6
34.4
29.7
36.0
33.2
33.5
37.2
FEVER
93.1
93.4
84.3
91.6
84.0
90.0
92.8
93.7
95.1
Himate-FEVEER
43.6
44.4
22.7
42.7
39.0
44.5
45.0
53,4
56,0
SciFact
90.8
87.8
86.3
91.4
72.7
81.6
89.3
89.1
92.2
Avg.
63.4
64.3
56.2
64.7
47.7
60.0
64.2
64.8
66.2
Best
3
0
9
Table 7: BEIR Benchmark, We report Recall @ 100 on the test sets of the datasets from the BEIR benchmark for bi-encoder methods without re-ranker. We also report the average ("Avg.") and number of datasets where a method is the best ("Best on") over the entire BEIR benchmark (excluding three datasets because their licence). We evaluate retrievers after pre-training on unsupervised data with our contrastive learning and finetuning on MS-MARCO ("Ours"). Bold is the best overall.
A TECHNICAL DETAILS
Contrastive pre-training. For the model with fine-tuning on MS-MARCO (respectively, without fine-tuning), we use the MoCo algorithm He et al. (2020) with a queue of size 65536 (resp. (37,)) momentum value of 0.999 (resp. 0.9995) and temperature of 0.05. We use the random cropping data augmentation, with documents 256 tokens and span sizes sampled between 10% and 50% (resp. 5% and 50%) of the document length. We also apply word deletion with probability of 20%. We optimize the model with the Adam W (Loshchilov Hutter 2019) optimizer, with learning rate of 10 (resp. 5 10 50, batch size of 2,048 and 500,000 (resp. 200.000) steps. We initialize the network with the publicly available BERT base uncased model. For training data, we use a data from Wikipedia (resp. mix between Wikipedia and CC-net data (Wenzek al. 2020). where half the batches are sampled from each source).
Fine-tuning on MS-MARCO. When fine-tuning our model on MS-MARCO, use the ASAM optimizer won et al. 2021), with a learning rate of 10-5 and batch size of 1024 with a temperature of 0.05, also used during pretraining. We train an initial model with random negative examples for 20000 steps, mine hard negatives with this first model, and re-train a second model with those. Each query associated with a gold document and negative document, which a random document in the first phase and hard negative 10% of the time in the second phase. All documents from a batch are used as negatives for each query.
NaturalQuestions
TriviaQA
R@1
R@5
R@20
R@
R 100
Inverse Cloze Task Sachan al. 2021b
12.6
32.3
50.9
66.8
19.2
40.2
57.5
73.6
Masked salient spuns (Sachin 2021b
20.0
41.7
59.8
31.7
53.3
68.2
79.4
BM25 Ma et al. 202
62.9
78.3
76.4
83,2
Ours
21.9
47.2
67.2
81.3
33.9
59.5
74.2
83.2
Supervised topline: DPR (Karpukhin et al. 2020
78.4
85.4
79.4
85.0
Table 8: Recall @k the test sets of NaturalQuestions and TriviaQA. For Inverse Cloze Task and Masked Salient Spans we report the results of Sachan et al. (2021b). The Masked Salient Spans model uses annotated entity recognition data.
14


Reviews:
[{"review_id": "jCqsLchZBhX", "reviewer": "Reviewer_cPwk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- A cropping-based positive pair construction method is proposed and compared with the ICT-based method.\n- The zero-shot evaluation and comparison on the BEIR benchmark look good.\n\nCons:\n- Using contrastive learning in dense IR is not new. Most technical pieces mentioned here (SimCLR, MoCo, ICT, etc.) were used somewhere else before. \n- For the fine-tuning setting (e.g. Table 6), it could be better if the baseline dense IR approaches (mentioned in Table 1) can be included as well, instead of just comparing with BERT.\n ", "summary_of_the_paper": "This paper proposed a contrastive learning-based method for zero-shot dense IR. Based on the contrastive learning framework, 3 positive pair construction methods and 2 negative pair construction methods are used and compared. Evaluations are performed on the BEIR benchmark, with the best average result achieved compared to a list of IR (both sparse and dense) baselines in the zero-shot setting.", "summary_of_the_review": "Duo to the novelty and insufficient experiment, I think the paper should be further refined and improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bcEtF5b9Z4h", "reviewer": "Reviewer_tnYn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- the paper writing is clear and easy to follow\n- solid empircal comparison\n\n## Weaknesses\n- lack of techincal novelty. The contrastive loss for learning Siamese Transformers is not new. Creating pseudo (query, doc) pairs for the constrastive loss is also studied in ICT.  \n- the ablation study of truely zero-shot (w/o fine-tuning with MS-MARCO) evaluation on BEIR is missing.\n- the author should also compare with other pre-training frameworks in Figure 1, including ICT, REALM, and SimCSE.\n- from Figure 1, it seems like the the proposed model is worse than BM25 in terms of the average Recall@100\n- for reproducibility, the author should release the experiment code, pre-training datasets and the pre-trained models\n\n\n", "summary_of_the_paper": "This paper studies pre-training of Siamese Transformer encoders for the zero-shot dense retrieval problem. The author consider contrastive learning loss function to optimize the encoders and investigate variants of ICT for data augmentations. While not much novelty in the techincal sides, the author indeed present solid empirical study on zero-shot retrieval benchmark as well as extension to the few-shot evaluation.", "summary_of_the_review": "This paper studies pre-training of Siamese Transformer encoders for the zero-shot dense retrieval problem. While the empirical results seem promising, there are very little techincal novelty in this work. Since this paper concerns pre-training tasks for retrieval, the author should also show the results of ICT and REALM for the Figure 1 setup. At the moment, I think this paper is marginally below the acceptance threshold. If the author can address my concern about proper ablation comparisons, I am very happly to update the recommendation scores. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NygoBnumsqA", "reviewer": "Reviewer_NWVD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem for neural models for IR: what unsupervised training task is suitable for an IR model? The existing strategies such as masked language model and ICT fit some of the NLP tasks, but not necessarily the IR task. This paper proposes to use cropping to train a model for IR.\nThe proposed strategy is interesting and intuitively more relevant to IR tasks. The experiments show that this is indeed better than the other training strategies.\nThe paper is well written, and contains a set of experiments on several datasets. The experimental results are convincing.\nOne broader question is on the idea of replacing BM25 in the zero-shot learning scenario. As the experiments show, when no training data (aligned query-documents) is available, the proposed method can just catch up with BM25. In other words, the cropping training strategy can just reach the level of BM25, but with heavy pre-training. It does not seem reasonable to try to compete with an inexpensive method such as BM25 by a heavy model. Instead, the neural model should seek to complement BM25, i.e. capture other matching signals.\nFrom this perspective, trying to beat BM25 in a zero-shot learning scenario may not be useful in practice.", "summary_of_the_paper": "This paper proposes to use cropping strategy to train a neural model for IR. It is compared to BM25 and several other neural models, including BERT and the ones using inverse cloze task. The goal of cropping is to train a model that fits better the IR tasks. It is hypothesized that this task is more similar to IR than other pre-training strategies.\nThe method is tested on a set of benchmarks of zero-shot retrieval and MSMARCO with training data. It is shown than the proposed method fine-tuned on MSMARCO can outperform the compared methods, including BM25. When no fine-tuning is used, the method is almost at par with BM25.\n", "summary_of_the_review": "The proposed method is interesting and the experiments show it effectiveness in the scenario of zero-shot learning and few-shot learning.\nHowever, the question is about the necessity to build a complexe neural model to compete with much more efficient BM25 model in the zero-shot learning situation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "143C1z-kdE", "reviewer": "Reviewer_F8YA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed model has some merits but a lot of weaknesses, which I elaborate next.\n\n**Weaknesses**:\nThe major weaknesses in the paper are:\n\nIntroduction Section:\n1. The authors need to assign much more credit to the previous papers on unsupervised retrieval using contrastive methods. It is sad to see that the authors have not properly cited even the original ICT paper which was the first to propose the zero-shot neural text retrieval model and several important follow up works ([1], [2], [3]) in the Introduction section. The authors should pay careful attention to how they are positioning their work in relation to the previous well-known work on this topic. It’s appalling to see this style of writing that leads to improper credit assignment. I am sorry to say but writing the paper in a way where you intentionally or unintentionally hide the contributions of the previous work on the topic misguides the reader towards the contributions of the paper and is not in the spirit of a top-tier conference like ICLR.\n\n2. From the Introduction section, the paper positions itself as a new unsupervised contrastive representation learning algorithm in the field of neural retrieval. My suggestion to you would be to position the paper as a followup to the ICT and subsequent works. And then explain clearly the new features in your proposed model when compared against a strongly trained ICT and followup models in references ([1], [2], [3]). You can make a table in the Introduction section listing the new introduced aspects in the proposed model and compare them in a rigorous manner with the previous work.\n\nRelated Work Section:\n\n3. In the 4th paragraph, it is wrongly mentioned that Karpukhin et al. 2020 in their DPR paper introduced the dense retriever trained with BERT initialization. This is not correct. It was first done by the ICT paper and later DPR applied their model for supervised contrastive training.\nMinor edits: (Izacard and Grave, 2021) and (Yang and Seo, 2020) appear two times in the same sentence. My suggestion would be to cite all the important work in unsupervised dense retrieval such as [1, 2, 3] etc. and then contrast your work with the results of previous work.\n\nOther issues in writing\n\n4. Section 3.1.2 first paragraph: A method from computer vision literature is presented for the two views of an image but its citation is not provided.\n\nMethods Section \n\n5. Section 3.1.2: ICT paragraph: There are a couple of errors in how the ICT model is described. First, it should be clearly mentioned that the the view 1 is one sentence while the view 2 is all the remaining sentences in the paragraph. Please refer to Section 4 of the ICT paper for full details (https://arxiv.org/pdf/1906.00300.pdf). Second, to enable lexical matching in ICT, the view 1 is removed with a probability of 0.9 and is kept otherwise. Again, please see Section 4 in the original paper for more details and Section 9.2 for the analysis of why this is important during ICT training.\n\n6. Section 3.1.3: *Negative pairs within a batch*: this method for contrastive learning was first used in ICT and subsequent work such as [2, 3]. *Negative Pairs Across Batches*: This method (MuCo) has also been investigated for dense retriever training by [4]. These papers should be cited here. Also, [4] reports some issues \n\nResults Section:\n\n7. Section 4.3: (Second last paragraph)  The better performance of contrastively trained models when compared to BERT in both zero-shot and few-shot settings has already been investigated earlier. For example, [3] compares the zero-shot retrieval performance for both BERT and ICT in Table 3 and for few-shot performance in Figure 3. This should be mentioned here.\n\n8. I feel that the inclusion of zero-shot results Figure 1 in the paper doesn’t really add much value until compared to a couple of previous baselines such as that of ICT. As the code is not submitted with the paper, I can't verify if the implementation of ICT is correct or not. My suggestion is that you should first make sure the implementation of ICT is correct, then train it with the same settings as your proposed approach of Independent Cropping. Also include the results of ICT in Table1, Table2, and Figure1. This will help the reader to better appreciate the results in the paper.\n\n**Strengths**\n\n1. The proposed method of Independent Cropping for unsupervised retrieval training seems novel and often outperforms strong BM25 baseline on the BEIR benchmark.\n\n2. The results presented in Table 3 are important as it demonstrates that MoCo can be leveraged to train models instead of SimCLR in a memory efficient manner.\n\n3. If the ICT implementation is correct, then the results in Table 4 would be very insightful. Then, the authors should really dig deep into the reasons why the model performance is better than ICT when keeping all the training settings the same such as the number of training steps, batch size, etc.\n\n\n**Other Questions**\n\n1. It is mentioned that you averaged the token-level representations to compute query and document representations. Did you compare how much the results differed when you just took their [CLS] token representations?\n\n2. Could you also report the zero-shot Recall@20,100 results in Table 4 without finetuning of MS-MARCO? \n\n3. For qualitative analysis, can the authors present some examples on NQ when their model shows better top-1 recall than ICT due to a better lexical match? \n\nReferences:\n\n[1] Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yim- ing Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In International Conference on Learning Representations.\n\n[2] Luan, Yi, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. \"Sparse, Dense, and Attentional Representations for Text Retrieval.\" Transactions of the Association for Computational Linguistics.\n\n[3] Sachan, Devendra Singh, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, and Bryan Catanzaro. 2021. \"End-to-end training of neural retrievers for open-domain question answering.\" In ACL 2021.\n\n[4] Yang, Nan, Furu Wei, Binxing Jiao, Daxing Jiang, and Linjun Yang. 2021. \"xMoCo: Cross momentum contrastive learning for open-domain question answering.\" In ACL 2021.\n", "summary_of_the_paper": "This paper presents a contrastive learning approach for unsupervised text retrieval. As the approach is unsupervised, similar to the Inverse Cloze Task (ICT) approach, the authors propose to do independent cropping of a paragraph to generate a pair of \"pseudo-question\" and \"pseudo-document\", which is used to train the model. The model is trained using contrastive learning following the MoCo algorithm. Experiments are performed on the recent BEIR benchmark for zero-shot and few-shot retrieval evaluation. The proposed approach often performs better than the BM25 algorithm on zero-shot retrieval on the BEIR benchmark. There are also some ablations presented to analyze different aspects of model training such as the effect of number of negative examples, training data, data augmentations etc.\n ", "summary_of_the_review": "The paper proposes the approach of independent cropping of a paragraph into two segments for unsupervised training of the dense retriever. The authors argue that independent cropping enables the model to possess lexical matching ability, a property which improves retrieval performance.  \n\nFirst, I feel that the writing in the Introduction and Related Work sections needs major improvements, specially that of crediting and citing the previous work in the field of unsupervised contrastive training. I have also provided my suggestions for the same under Weakness section. Second, I feel that the authors should correctly implement a ICT following the original paper and perform training using the same settings as done for their model. Third, instead of comparing with BM25, the emphasis should be on comparing results with ICT and providing an in-depth understanding of why or why not their proposed model works better. \n\nDue to these shortcomings, in its current form, the paper falls substantially short of the ICLR publication bar. \nHowever, I would request the authors to provide their responses to the points raised in the Weakness section and under the Other Questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Good premise:  What unsupervised training supports IR?  This is a key question for IR and is a focus for papers in TREC 2019 Deep Learning Track, for instance.  Also, historically, empirical work in the IR community is a very high standard. 

One reviewer says the contrastive loss for learning Siamese Transformers is not new, and prior experimental work was listed.  Several reviewers suggested extensions to the empirical work, some of which was subsequently done.  Results are "promising" according to one reviewer, but not strong enough.  Another reviewer says a different use context is needed since its hard to compete with efficient BM2t in its own terms.  The authors made some good changes to their paper: updated intro and related work, extended results and discussion, but the 4 reviewers remained in agreement, a reject.  However, some reviewers felt this was a good contribution, so with further empirical work and polish it should be good.
Decision: Reject