Unsupervised Dense Information Retrieval with Contrastive Learning
Authors: Gautier Izacard
Mathilde Caron
Lucas Hosseini
Sebastian Riedel
Piotr Bojanowski
Armand Joulin
Edouard Grave
Venue: Accepted by TMLR ()

Abstract: Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new applications with no training data, and are outperformed by unsupervised term-frequency methods such as BM25. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings. On the BEIR benchmark our unsupervised model outperforms BM25 on 11 out of 15 datasets for the Recall@100. When used as pre-training before fine-tuning, either on a few thousands in-domain examples or on the large MS~MARCO dataset, our contrastive model leads to improvements on the BEIR benchmark. Finally, we evaluate our approach for multi-lingual retrieval, where training data is even scarcer than for English, and show that our approach leads to strong unsupervised performance. Our model also exhibits strong cross-lingual transfer when fine-tuned on supervised English data only and evaluated on low resources language such as Swahili. We show that our unsupervised models can perform cross-lingual retrieval between different scripts, such as retrieving English documents from Arabic queries, which would not be possible with term matching methods.

Published in Transactions on Machine Learning Research (08/2022)
Unsupervised Dense Information Retrieval with
Contrastive Learning
Gautier Izacard 2
gizacard@fb. com mathilde wife...coooo hoss 0ifb com sriedel@ff. com bojanouski@ft com ajoulin @fb. com egrave
Mathilde Caron 0,00
Lucas Hosscini
Sebastian Ricde19.4
Piotr Bojanowski
Armand Joulin
Edouard Grave
Mela Al Research, Ecole normale supérieure, PSL University, Inria,
Université Grenoble Alpes, University College London
Reviewed on OpenReview: https: openreview. net/ forum? id- JKNIIXX7770
Abstract
Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as alternative to classical sparse methods based term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new applications with no training data, and are outperformed by unsupervised term-frequency methods such as BM25. In this work, we explore the limits contrastive learning as a way to train unsupervised dense retrievers and show that leads to strong performance in various retrieval settings. On the BEIR benehmark unsupervised model outperforms BM25 11 out of 15 datasets for the Recall@100. When used pro-training before finc-tuning, either in-domain examples on the large MS MARCO dataset, our contrastive model leads to improvements on the BEIR benchmark. Finally, we evaluate our approach for multi- lintial retrieval, where training data is even scarcer than for English, and show that our approach leads to strong unsupervised performance. Our model also exhibits strong cross- cross-lingus lingual transfer when fine-tuned on supervised English data only and evaluated on low resources language such Swahili. We show that our unsupervised models can perform lingull retrieval between different scripts, such retrieving English documents from Arabic queries, which would not be possible with term matching methods.
few thousands
Introduction
Document retrieval is the task of finding relevant documents in a large collection to answer specific queries. This is an important task by itself and a core component to solve many natural language processing (NLP) problems, such as open domain question answering (Chen al., 2017a) or fact checking (Thorne al., 2018). Traditionally, retricval systems, or retrievers, leverage lexical similaritics match queries and documents, using, for instance, TF-IDF or BM25 weighting (Robertson & Zaragoza, 2009). These approaches, based near-exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well (Berger et al., 2000). By contrast, approaches based on neural networks allow learning beyond lexical similarities, resulting in state-of-the-art performance on question answering benchmarks, such as MS MARCO (Nguyen et al., 2016) or NaturalQuestions (Kwiat et al., 2019).
The strong retrieval results of neural networks have been possible for domains and applications where large training datasets are available. In the case of retrieval, creating these datasets requires manually matching queries to the relevant documents in the collection. This is hardly possible when the collection

---

Published in Transactions on Machine Learning Research (08/2022)
contains millions or billions of elements, resulting in many scenarios where only few in-domain examples, if any, are available. A potential solution is to train dense retriever on a large retrieval dataset such as MS MARCO, and then apply it to new domains, a setting referred to as zero-shot. Unfortunately, in this setting, dense retrievers are often outperformed by classical methods based on term-frequency, which do not require supervision (Thakur et al., 2021). Moreover large annotated datasets are generally not available in languages other than English. Thus, using large collections of supervised data not suitable to train multilingual retrieval systems.
A natural alternative to transfer learning is unsupervised learning, which raises the following question: is it possible to train dense retrievers without supervision, and match the performance of BM25% Training dense retrievers without supervision can be achieved by using an auxiliary task that approximates retrieval. Given a document, one can generate a synthetic query and then train the network to retrieve the original document, among many others, given the query. The inverse Cloze task (ICT), proposed by Lee et al. (2019) to pro-train retrievers, uses a given sentence as a query and prodicts the context surrounding it. While showing promising results pre-training (Chang el. al., 2020; Sachan et al., 2021), this approach still lags behind BM25 when used as zero-shot retriever. ICT is strongly related to contrastive learning (Wu et al., 2018), which has been widely applied in computer vision (Chen et al., 2020; He et al., 2020). In particular, computer vision models trained with the latest contrastive learning methods led to features well suited to retrieval (Caron et al., 2021). We thus propose to revisit. how well contrastive learning performs to train dense retrievers without supervision.
In this paper, we make the following contributions. First, we show that contrastive learning can lead strong unsupervised retrievers: our model achieves Recall@100 results competitive with BM25 most of the BEIR benchmark. Second, in a few-shot setting, show that our model benefits from few training examples, and obtains better results than transferring models from large datasets such as MS MARCO. Third, when used as a pre-training method before fine-tuning on MS MARCO, our technique leads to strong performance on the BEIR benchmark. We perform ablations to motivate our design choices, and show that cropping works better than the inverse Cloze task. Finally we train a multilingual dense retriever with rastivevive learning and show that achieves state-of-the-art performance.
Code and pre-trained models are available here: http:s///ttaa..ooccccccooorrrrrs
2 Related work
In this section, we briefly review relevant work in information retrieval, and application of machine learning to this problem. This is not an exhaustive review, and we refer the reader to Manning et al. (2008), Mitra et al. (2018) and Lin et al. (2020) for a more complete introduction to the field.
ferm-frequenee based information retrieval. Historically, in information retrieval, documents and queries are represented as sparse vectors where each element of the vectors corresponds to a term of the vocabulary. Different weighing schemes have been proposed, to determine how important a particular term is to document a large dataset. One of the most used weighing scheme is known as TF-IDF, and is basod on inverse document frequency, or term specificity (Jones, 1972). BM25, which is still widely used today, extends TF-IDF (Robertson et al., 1995). A well known limitation of these approaches is that they rely on near-exact match to retrieve documents. This led to the introduction of latent semantic analysis (Deerwester et al., 1990), in which documents are represented as low dimensional dense vectors.
Neural network based information retrieval. Following the successful application of deep learning methods to natural language processing, neural networks techniques were introduced for information retrieval. Huang et al. (2013) proposed a deep bag-of-words model, in which representations of queries and documents are computed independently. A relevance score is then obtained by laking the dot product between representations, and the model is trained end-to to-end on click data from a search engine. This method was later refined by replacing the bag-of-words model by convolutional neural networks (Shen et al., 2014) or recurrent neural network (Palangi et al., 2016). A limitation of bi-encoders is that queries and documents are represented by a single vector, preventing the model to capture fine-grained interactions between terms. Nogueira & Cho
2

---

Published in Transactions on Machine Learning Research (08/2022)
(2019) introduced a cross-encoder model, based on the BERT model (Devlin et al., 2019), which jointly encodes queries and documents. The application of strong pre-trained model, as well as the cross-encoder architecture, lead to important improvement on the MS MARCO benchmark (Bajaj et al., 2016).
The methods described in the previous paragraph were applied to re-rank documents, which were retrieved with a traditional TR system such as BM25. Gillick et al. (2018) first studied whether continuous retrievers, based on bi-encoder neural models, could be viable alternative to re-ranking. In the context of question answering, Karpukhin et al. (2020) introduced dense passage retriever (DPR) based the bi-encoder architecture. This model is initialized with a BERT network, and trained discriminatively using pairs of queries and relevant documents, with hard negatives from BM25. Xiong et al. (2020) further extended this work by mining hard negatives with the model itself during optimization, and trained on the MS MARCO dataset. Once a collection of documents, such as Wikipedia articles, is encoded, retrieval is performed with a fast k-nearest neighbors library such as FATSS Johnson et al. (2019). To alleviate the limitations of bi-encoders, Humcau et al. (2019) introduces the poly-encoder architocture, where documents are encoded by multiple vectors. Similarly, Khattab et al. (2020) proposes the ColBERT model, which keeps veclor representation for each term of the queries and documents. To make the retrieval tractable, the term-level function is approximated to first retrieve an initial set of candidates, which are then re-ranked with the true score. In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Tzacard & Grave, 2020a), or the relevance score from a cross encoder (Yang & Seo, 2020). Luan et al. (2020) compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross- and poly-encoders. Dense retrievers, such as DPR, can lead to indices weighing nearly 100GB when encoding document collections such as Wikipedia. Izacard et al. (2020) shows how to compress such indicos, with limited impact on performance, making them more practical to use.
Self-supervised learning for NLP. Following the success of word2vec (Mikolov el al., 2013), many self-supervised techniques have been proposed to learn representation of text. Here, we briefly review the ones that are most related to approach: sentence level models and contrastive techniques. Jernite et al. (2017) introduced different objective functions to learn sentence representations, including next sentence prediction and sentence order prediction. These objectives were later used in pre-trained models based on transformers, such as BERT (Devlin et al., 2019) and AIBERT (Lan et al., 2019). In the context of retrieval, Lee et al. (2019) introduced the inverse cloze task (ICT), whose purpose is to predict the context surrounding a span of text. Guu et al. (2020) integrated a bi-encoder retriever model in a BERT pre-training scheme. The retrieved documents are used as additional context in the BERT task, and the whole system is trained end-to-end in an unsupervised way. Similarly, Lewis et al. (2020) proposed to jointly learn a retriever and generative seq2seq model, using self-supervised training. Chang et al. (2020) compares different pre-training tasks for retrieval, including the inverse cloze task. In the context of natural language processing, Fang et al. (2020) proposed to apply MoCo where positive pairs of sentences are obtained using back-translation. Different works augmented the masked language modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al., 2020; Meng et al., 2021). SBERT (Reimers & Gurevych, 2019) uses a Siamese network similar to contrastive learning to learn a BERT-like model that is adapted to matching sentence emboddings. Thoir formulation is similar to our work but requires aligned pairs of sentences to form positive pairs while we propose to use data augmentation to leverage large unaligned textual corpora. Concurrent to this work, Gao & Callan (2021) have also shown the potential of contrastive learning for information retrieval; building on the same observation that both tasks share similar structure. Spider (Ram et al., 2021), a contemporary work, uses spans appearing multiple times in a document to create pseudo examples for contrastive learning in order to train unsupervised retrievers. Finally, Chen et al. (2021) train a dense retriever to imitate unsupervised lexical-basee methods. This improves performance on a range of tasks and achieves stato-of-tho-ar results when combining the resulting dense rieve with Contriever, our model pre-trained with contrastive learning.
3 Method
In this section, we describe how to train a dense retriever with no supervision. We review the model architecture and then describe contrastive learning key component of its training.
3

---

Published in Transactions on Machine Learning Research (08/2022)
The objective of retriever is to find relevant documents in large collection for a given query. Thus, the retriever takes as input the set of documents and the query and outputs a relevance score for each document. A standard approach is to encode each query-document pair with a neural network (Nogueira & Cho, 2019). This procedure requires re-encoding every document for any new query and hence does not scale to large collections of documents. Instead, we follow standard approaches (Huang et al., 2013; Karpukhin 2020) in information retrieval and use bi-encoder architecture where documents and queries are encoded independently. The relevance score between query and document is given by the dot product between their representations after applying the encoder. More precisely, given a query 9 and document d, encode each of them independently using the same model, fo, parameterized by 0. The relevance score s(g. d) between a query and document d is then the dot product of the resulting representations:
al.,
s(g, d) = (o(()))
In practice, we use a transformer network for So to embed both queries and documents. Alternatively, two different encoders can be used to encode queries and documents respectively as in DPR (Karpukhin et al., 2020). Empirically, we observed that using the same encoder, such as in Xiong et al. (2020) and Reimers & Gurevych (2019), generally improves robustness in the context of zero-shot transfer or few-shot learning, while having no impact on other settings. Finally, the representation fo(g) (resp. fo(d) for query (resp. document) is obtained by averaging the hidden representations of the last layer. Following previous work on dense retrieval with neural networks, we use the BERT base uncased architecture and refer the reader to Devlin et al. (2019) for more details.
3.1 Unsupervised training unaligned documents
In this section, we describe our unsupervised training method. We briefly review the loss function traditionally used in contrastive learning and also used in ICT (Lee et al., 2019). We then discuss obtaining positive pairs from a single text document, a critical ingredient for this training paradigm.
3.1.1 Contrastive learning
Contrastive learning is approach that relies the fact that every document is, in some way, unique. This signal is the only information available in the absence of manual supervision. A contrastive loss is used to learn by discriminating ween documents. This loss compares either positive (from the same document) or negative (from different documents) pairs of document representations. Formally, given a query q with an associated positive document k4, and pool of negative documents (k,))=1.K, the contrastive InfoNCE loss is defined as:
exp(s(g, k4)/T)
C(g,k4) (xx(s(g,k44//) E ex)(s(q,k://T)
(1)
where a temperature parameter. This loss encourages positive pairs have high scores and negative pairs to have low scores. Another interpretation of this loss function is the following: given the query representation g, the goal is to recover, retrieve, the representation 44 corresponding to the positive document, among all the negatives k; In the following, we refer to the left-hand side representations in the score queries and the rinh side representations as keys.
3.1.2 Building positive pairs from a single document
A crucial clement of contrastive learning is how to build positive pairs from single input. In computer vision, this step relies applying two independent data augnentatiios to the same image, resulting in two "views" that form positive pair (Wu et al., 2018; Chen et al., 2020). While we consider similar independent text transformation, we also explore dependent transformations designed to reduce the correlation between views.
data augmentation that generates two mutually exclusive views of a document, introduced in the context of retrieval by Lee al. (2019). The first view obtained by randomly sampling a span of tokens from a segment of text, while the complement of the span forms the second view. Specifically, given sequence of text (w)),-.,,.)) ICT samples a span (Way..., Wb), where S b S n, uses the
Inverse Cloze Task
4

---

Published in Transactions on Machine Learning Research (08/2022)
100
REALM
SimCSE
BM25
Contriever
Recall
50
Trec-COVID
Touche-2020
NFCorpus
Seidoes
DBPedia
Climate-fever
CQADupStack
FIQA
MSMARCO
HotpoiQA
NQ
ArguAna
Scifact
Fever
Quora
Avg.
Figure Unsupervised retrieval. We compare our pre-training without using any annotated data to REALM (Guu et al., 2020), SimCSE (Gao al., 2021) and BM25. For SimCSE we report results of the model using RoBERTa large. REALM uses annotated entity recognition data for training. We highlight that our unsupervised pro-training is par with BM25 but on 2 datasets.
tokens of the span as the query and the complement (Wi,..., Wa- Wb+1.... w,) as the key. In the original implementation by Lee et al. (2019) the span corresponds to a sentence, and is kept in the document 10% of the time to encourage lexical matching. The Inverse Cloze Task closcly related to the Cloze task which uses the span complement (wI, -1- Wb+1,-., Wa) as the query.
Independent cropping is a common independent data angmentation used for images where views are generated independently by cropping the input. In the context of text, cropping is equivalent to sampling a span of tokens. This strategy thus samples independently two spans from a document to form a positive pair. As opposed to the inverse Cloze task, cropping both views of the example correspond to contiguous subsequence of the original data. A second difference between cropping and ICT the fact that independent random cropping is symmetric: both the queries and documents follow the same distribution. Independent cropping also lead to overlap between the two views of the data, hence encouraging the network to learn exact matches between the query and document, in way that is similar to lexical matching methods like BM25. In practice, we can either fix the length of the span for the query and the key, or sample them.
Additional data augmentation. Finally, also consider additional data augmentations such as random word deletion, replacement or masking. We use these perturbations in addition to random cropping.
3.1.3 Building large set of negative pairs
An important aspect of contrastive learning is to sample large set of negatives. Most standard frameworks differ from each other in terms of how the negatives handled, and we briefly describe two of them, in-batch negative sampling and MoCo, that we use in this work.
Negatives within a batch. first solution is generate the negatives by using the other examples from the same batch: each example in batch is transformed twice to generate positive pairs, and we generate negatives by using the views from the other examples in the batch. We will refer to this technique as "in-batch negatives". In that case, the gradient is back- back-propagated through the representations of both the queries and the keys. A downside of this approach is that it requires extremely large batch sizes to work well Chen et al. (2020), with Qu et al. (2021) reporting improvement in the context of information retrieval up to 8192 negatives. This mothod has been widely used train information retrieval models with supervised data Chen et al. (2017b); Karpukhin al. (2020) and was also considered when using ICT pre-train retrievers by Lee et al. (2019).
5

---

Published in Transactions on Machine Learning Research (08/2022)
Negative pairs across batches. An alternative approach is to store representations from previous batches in queue and use them negative examples in the loss (Wu et al., 2018). This allows for smaller batch size but slightly changes the loss by making it asymmetric between "queries" (one of the view generated from the elements of the current batch), and "keys" (the elements stored in the queue). Gradient is only backpropagated through the "queries", and the representation of the "keys" are considered as fixed. In practice, the features stored in the queue from previous batches comes form previous iterations of the network. This leads to drop of performance when the network rapidly changes during training. Instead, He et al. (2020) proposed to generate representations of keys from a second network that is updated more slowly. This approach, called MoCo, considers two networks: one for the keys, parametrized by 6s, and one of the query, parametrized by 0g- The parameters of the query network are updated with backpropagation and stochastic gradient descent, similarly to when using in-batch negalives, while the parameters of the key netk or Momentum encoder, is updated from the parameters of the query network by using a exponential moving average:
02 + m0 + (1 -1-110 (1-m))ag
(2)
where m is the momentum parameter that takes its value in [o, 1].
4 Experiments
In this section, we ompirically evaluate our best retriever trained with contrastive learning, called Contriever (contrastive retriever), which uses MoCo with random cropping. We use a contrastive learning procedure that differs from ICT (Lee et al., 2019) mainly in three aspects. First, positive pairs are sampled using random cropping and tokens from each element of the pair are deleted with a probability of 10%. Second use MoCo where negatives consists of elements from previous batches stored in queue. This allows to scale to a large number of negatives. Third we use data from Wikipedia and CCNet. (Wenzek et al., 2020) for training. Ablation studics motivating these technical choices are performed in Scction 6. More technical details about our model are given in Appendix A.I.
4.1 Datasets
Contriever is trained with contrastive learning on documents sampled from mix between Wikipedia data and CCNet data (Wenzek et al., 2020), where half the batches are sampled from each source.
First, we evaluate our model on two question answering datasets: NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). For both dalasets, we use the open domain versions as introduced by Lee al. (2019), and the English Wikipedia dump from Dec. 20, 2018 as the collection of documents retrieve from. We report the top-k retrieval accuracy, i.e. the number of questions for which at least one of the top-k passages contain the answer.
Second, we use the BEIR benchmark, introduced by Thakur et al. (2021), which contains 18 retrieval datasets, corresponding to nine tasks, such as fact checking or citation prediction, and covering different domains, such as Wikipedia or scientific publications. Most datasets from BEIR do not contain a training set, and the focus of the benchmark is zero-shot retrieval. However, most machine learning based retrievers are still trained on supervised data, such as the large scale retrieval dataset MS MARCO (Bajaj et al., 2016). Following standard practice, report two metrics this benchmark: nDCG@10 and Recall@ 100. The nDCG@10 focuses the ranking of the top 10 retrieved documents, and is good at evaluating rankings returned to humans, for example in a scarch engine. On the other hand, Recall@100 is relevant to evaluate retrievers that are used in machine learning systems, such as question answering. Indeed, such models can process hundreds of documents, and ignore their ranking (Izacard & Grave, 2020b). While nDCG@10 is the main metric of BEIR, we are more interested in the Recall@100 to evaluate bi-encoders, as our goal to develop retrievers that can be used in ML systems. Moreover, in many settings, retrieved documents can be re-ranked with a more powerful model such as cross-encoder, thus improving the nDCG@10.
6

---

Published in Transactions on Machine Learning Research (08/2022)
Unsupervised recall@k on the test sets of NaturalQuestions and TriviaQA. For Inverse Cloze Task and Masked Salient Spans report the results of Sachan et al. (2021). The Masked Salient Spans model uses annotated named entity recognition data. For BM25 report the results of Ma et al. (2021)
Table 1:
NaturalQuestions
TriviaQA
R@5
R@20
R@100
R@5
R@20
R@000
Inverse Cloze Task (Sachan et al., 2021) Masked salient spans (Sachan et al., 2021)
32.3
50.9
66.8
40.2
57.5
73.6
41.7
59.8
74.9
53.3
68.2
79.4
BM25 (Ma et al., 2021)
62.9
78.3
76.4
83.2
Contriever
47.8
67.8
82.1
59.4
74.2
83.2
supervised model: DPR (Karpukhin et al., 2020)
78.4
85.4
79.4
85.0
supervised model: FiD-KD (Izacard & Grave, 2020a)
73.8
84.3
89.3
77.0
83.6
87.7
4.2 Baselines
First, we compare Contriever to BM25, which does not require supervision. On QA datasets, we compare to dense retrievers trained with ICT and the Masked Salient Spans from Sachan et al. (2021). On BEIR, we consider the retriever from REALM (Guu et al., 2020), and RoBERTa large fine-tuned with SimCSE (Gao et al., 2021), as unsupervised dense retrievers. We also compare to ML-based retrievers trained MS MARCO, classified in three categories: sparse, dense and late-interaction. For sparse methods, we compare to Splade v2 (Formal et al., 2021), which computes sparse representations of documents with BERT pre-trained model. For dense methods, we use DPR (Karpukhin et al., 2020) and ANCE (Xiong et al., 2020), which are bi-encoders trained on supervised data such as NaturalQuesstins or MS MARCO. We also compare to TAS-B (Hofstätter al., 2021), which performs distillation from a cross-encoder a bi-encoder, and GenQ, which creates synthetic query-document pairs with generative model. For late-interaction, use ColBERT (Khattab et al., 2020), which computes pairwisc scores between contextualized representations of queries and documents, well as cross-encoder used to re-rank docunents retrieved with BM25.
4.3 Results
First, we compare the performance of fully unsupervised models, i.c., without fine-tuning on MS MARCO or other annolated data. In Table 1, we report the retrieval performance on two question answering dalasets: NaturalQuestions (Kwiatkowski al., 2019) and TriviaQA (Joshi et al., 2017). Here, our model is competitive with strong BM25 bascline (Ma et al., 2021), ), for example leading to 3 points improvement for the recall@100 NaturalQuestions. It also outperforms previously proposed dense retrievers which were trained with ICT salient span masking. In Figure 1 we report the recall@100 performance of unsupervised models on the BEIR benchmark. Interestingly, we observe that in this setting, Contriever is competitive compared to BM25 on all datasots, but TREC-COVID and Tóucho-2020. In particular, it obtains better performance than BM25 on 11 out of 15 datasets from the benchmark for the recall@100. Contriever also ourperfoms previously proposed unsupervised dense retrievers, which obtains lower performance than BM25 in general. For the nDCG@10, which puts more emphasis on the very first retrieved documents, while Contriever largely closes the gap between unsupervised retrievers and BM25, it is still outperformed by BM25 as reported in Table 11. The difference is mainly due to the fact that BM25 largely outperforms Contriever on two datasets with specific features: Tree-COVID and T62che-2020. Tree-COVID is an information retrieval dataset related to COVID. However data used to train Contriever were collected before the COVID out break, thus they may not be adapted. Tóuche-202) 2020 contains long documents, which does not seem to be very well supported by dense neural retrievers: even after supervised training, models are still lagging behind BM25. Overall, these results show the potential of contrastive learning to train fully unsupervised dense retrievers.
.GenQ thus leads one different model each dataset.
7

---

Published in Transactions on Machine Learning Research (08/2022)
Table BEIR Benchmark. We report nDCG@10 on the test sets from the BEIR benchmark for bi-encoder
methods without re-ranker. We also report the average and number of datasets where a method is the best
("Best on") over the entire BEIR benchmark (excluding three datasets because of their licence). Bold is the
best overall. MS MARCO is excluded from the average. "CE" refers to cross-encoder.
BM25
CE
DPR
ANCE
TAS-B
Gen-Q
ColBERT
Splade v2
Ours
Ours-
MS MARCO
22.8
41.3
17.7
38.8
40.8
40.1
40.7
47.0
Tree-COVID
65.6
75.7
33.2
65.4
48.1
61.9
67.7
71.0
59.6
70.1
NFCorpus
32.5
35.0
18.9
31.9
30.5
33.4
32.8
34.4
NQ
32.9
53.3
47.4
44.6
46.3
35.8
52.4
52.1
49.8
57.7
HotpotQA
60.3
70.7
39.1
45.6
58.4
53.4
59.3
68.4
63.8
71.5
FIQA
23.6
34.7
11.2
29.5
30.0
30.8
31.7
33.6
32.9
36.7
ArguAna
31.5
31.1
17.5
41.5
42.9
49.3
23.3
47.9
44.6
41.3
Touche-2020
36.7
27.1
13.1
24.0
16.2
18.2
20.2
36.4
23.0
29.8
CQADupStack
29.9
37.0.
15.3
29.6
31.4
34.7
35.0
34.5
37.7
Quora
78.9
82.5
24.8
85.2
83.5
85.4
86.5
82.4
DBPedia
31.3
40.9
26.3
28.1
38.4
32.8
39.2
43.5
41.3
47.1
Scidocs
15.8
16.6
7.7
12.2
14.9
14.3
14.5
15.8
16.5
17.1
FEVER
75.3
81.9
56.2
66.9
70.0
66.9
77.1
78.6
75.8
81.9
Climate-FENV FEVER
21.3
25.3
14.8
19.8
22.8
17.5
18.4
23.5
23.7
25.8
Scifact
66.5
68.8
31.8
50.7
64.3
64.4
67.1
69.3
67.7
69.2
Avg. w/o CQA
44.0
49.5
26.3
41.3
43.7
45.1
50.6
47.3
51.2
Avg.
43.0
48.6
25.5
40.5
42.8
42.5
44.4
46.6
50.2
Best on
0
Table 3: Few-shot retrieval. Test nDCG@10 after training on small in-domain training set. We compare BERT and our model, with and without an intermediate fine-tuning step on MS MARCO. Note that our unsupervised pre-training alone outperforms BERT with intermediate MS MARCO fine-tuning.
Additional data
SciFact
NFCorpus
FIQA
# queries
729
2,590
5,500
BM25
66.5
32.5
23.6
BERT
75.2
29.9
26.1
Contriever
84.0
33.6
36.4
BERT
MARCO
80.9
33.2
30.9
Contriever
MS MARCO
84.8
35.8
38.1
Next, we report nDCG@10 on the BEIR benchmark for different retricvers trained on MS MARCO Table (recall@100 can be found in Table 10 of appendix). We individually report results on each dataset as well as the average over 14 datasets of the BEIR Benchmark (excluding for license reasons). We observe that when used as pre-training, contrastive learning leads strong performance: contriever obtains the best results among dense bi-encoder methods for the nDCG@10 and is state-of-the-art for the recall@ 100 (improving the average recall@100 from 65.0 to 67.1). This strong recall@10 100 performance can be further exploited by using cross-cncoder to re-rank the retricved documents: this leads to the state-of-the-art on 8 datascts of the BEIR benchmark for the nDCG@10, as well as on average. It should be noted that our fine-tuning procedure on MS MARCO simpler than for other retrievers, as we use a simple strategy for negative mining and do not use distillation. Our model would probably also benefits from improvements proposed by these retrievers, but this is beyond the scope of this paper.
Finally, illustrate the benefit of our retriever compared to BM25 in few-shot setting, where have access to small number of in-domain retricval examples. This setting is common in practice, and lcxical based methods, like BM25, cannot leverage the small training sets to adapt its weights. In Table 3, we report nDCG@10 on three datasets from BEIR associated with the smallest. training sets, ranging from 729 to
We the existing as-marco-Ho Maaarco-inni.--L----.- cross-encoder model perform the re-ranking.

---

Published in Transactions on Machine Learning Research (08/2022)
5,500 queries. We observe that on these small datasets, our pre-training leads to better results than BERT pre-training, even when BERT is fine-tuned on MS MARCO as intermediate step. Our pre-trained model also outperforms BM25, showing the advantage of dense retriever over lexical methods in the few-shot setting. More details are given in Appendix A.3.
5 Multilingual retrieval
In this section, we illustrate another advantage of learning unsupervised dense retrievers, when performing multi-lingual retrieval. While large labeled datasets are available in English, allowing to train strong dense retrievers (as shown in previous sections), this is unfortunatlly not the case for lower resources languages. Here, we show how unsupervised training is promising direction. First, show that our approach leads to strong performance, either in full unsupervised setting, or by fine-tuning a multi-lingual model on English data such as MS MARCO. Second, we remonstrale that our model can also perform cross-lingual retrieval, by retrieving English documents from other languages queries. Unsupervised retrievers based on lexical matching, such as BM25, would obtain poor performance, espocially for pairs of languages with different. scripts such as English and Arabic.
5.1 Multilingual pre-training
Our multilingual model, called mCCnttieeeer, is jointly pre-trained on 29 languages. The multilingual pre- training largely follows the method discussed in previous sections, but it differs by few particularities related to the pre-training data and the hyperparameters used. The model is initialized with the multilingual version of BERT, mBERT, trained on 104 languages. For the pre-training data, we consider the langnages contained in CCNet ( (Wenzek et al., 2020) that are also part of our evaluation datasets. This results in a training set containing the CCNet data for 29 languages detailed in Table 12. During pre-training, examples are uniformly sampled over languages, i.e. the probability that training sample comes from a specific language is the same for all languages. We observed that incrcasing the number of languages considered for pro-training generally deteriorates performance reported in Appendix B.3 similarly to what has been observed for multilingual masked language models (Conneau et al., 2019). We pre-trained our multilingual mContriever with a queue size of 32768. This generally improves stability, and is able to compensate for the additional instabilities observed in the multilingual setting. More detailed hyperparameters are given in Appendix B.1.
5.2 Fine- Fine-tuning
Large labeled datasets for information retrieval are generally available only in English. Tt therefore natural to investigate whether large monolingual datasets can be leveraged for multilingual retrieval. We consider fine-tuning pre-trained mContriever model on MS MARCO. This generally improves performance in all languages. The model trained on MS MARCO can be further fine-tuned on Mr. TyDi achieving state-of-the-art performance on this dataset. Further details regarding fine-tuning are given in Appendix B.2.
5.3 Evaluation
We evaluate the performance of our pre-trained model with and without fine-tuning on English data on two different. benchmarks. First, we consider Mr. TyDi (Zhang al., 2021), a multilingual information retrieval benchmark dorived from TyDi QA (Clark et al., 2020). Given question, the goal is find relevant documents in pool of Wikipedia documents in the same language.
In Mr. TyDi the pool of documents is restricted to the language of the query. Being able to retrieve relevant documents from another language is desirable to leverage large souree of information that may no be available in all languages. In order to evaluate cross-lingual retrieval performance we derive an evaluation setting from MKQA (Longpre et al., 2020). Given a question in a specific language, we perform retrieval in English Wikipedia, and evaluate the English answer is in the retrieved documents. The MKQA dataset makes this possible by providing the same questions and answers in 26 languages. We remove unanswerable questions, questions accepting binary yes/ne no answer and questions with long answers from the original MKQA dataset.
9

---

Published in Transactions on Machine Learning Research (08/2022)
Table 4: Multilingual retrieval on Mr. TyDi. We report MRR@1O and Recall@100 on the sets of Mr. TyDi. mContriever fine-tuned MS MARCO compared against its counterparts without contrastive pre-training using similar fine-tuning recipe, referred to as mBERT + MS MARCO, well as model initialized with XLM-R referred to as XLMMR + MS MARCO. We also report the results after fine-tuning on Mr. TyDi for the model trained on MS MARCO.
ar
bn
em
fi
id
ja
ko
IU
SW
te
th
avg
MRR
100
BM25 (Zhang et al., 2021)
36.7
41.3
15.1
28.8
38.2
21.7
28.1
32.9
39.6
41.7
33.3
mDPR (Zhang ct al., 2021)
26.0
25.8
16.2
11.3
14.6
18.1
21.9
18.5
7.3
10.6
13.5
16.7
Hybrid (Zhang et al., 2021)
49.1
53.5
28.4
36.5
45.5
35.5
36.2
42.7
40.5
42.0
49.2
41.7
mBERT + MS MARCO
34.8
35.1
25.7
29.6
36.3
27.1
28.1
30.0
37.4
39.6
20.3
31.3
XLM-R + MS MARCO
36.5
41.7
23.0
32.7
39.2
24.8
32.2
29.3
35.1
54.7
38.5
35.2
mContriever
27.3
36.3
9.2
21.1
23.5
19.5
22.3
17.5
38.3
22.5
37.2
25.0
MS MARCO
43.4
12.3
27.1
35.1
42.6
32.4
34.2
36.1
51.2
37.4
40.2
38.4
Mr. Tydi
72.4
67.2
56.6
60.2
63.0
54.9
55.3
59.7
70.7
90.3
67.3
65.2
Recall@i 100
BM25 (Zhang ot al., 2021)
80.0
87.4
55.1
72.5
84.6
65.6
79.7
66.0
76.4
81.3
85.3
74.3
mDPR (Zhang et al., 2021)
62.0
67.1
47.5
37.5
46.6
53.5
49.0
49.8
26.4
35.2
45.5
47.3
Hybrid (Zhang et al., 2021)
86.3
93.7
69.6
78.8
88.7
77.8
70.6
76.0
78.6
87.5
80.9
mBERT + MS MARCO
81.1
88.7
77.8
74.2
81.0
76.1
66.7
77.6
74.1
89.5
57.8
76.8
XLM-R + MS MARCO
79.9
84,2
73.1
81.6
87.4
70.9
71.1
74.1
73.9
91.2
89.5
79.7
mContriever
82.0
89.6
48.8
79.6
81.4
72.8
66.2
68.5
88.7
80.8
90.3
77.2
+ MS MARCO
88.7
91.4
77.2
88.1
89.8
81.7
78.2
83.8
91.4
96.6
90.5
87.0
Mr. Tydi
94.0
98.6
92.2
92.7
94.5
88.8
88.9
92.4
93.7
98.9
95.2
93.6
This results in an evaluation set of 6619 queries. It should be noted that methods based on term matching such as BM25 are intrinsically limited in this cross-lingual retrieval setting because similar terms in different. languages may not match.
5.4 Baselines
On Mr. TyDi (Zhang et al., 2021) we report results from the original paper. This includes BM25 baseline, model fine-tuned on NaturalQuestions using the DPR pipeline, and an hybrid model combining the two.
On our cross-lingual evaluation benchmark derived from MKQA, we consider the retriever of the CORA question answering pipeline (Asai et al., 2021), trained on a combination of datasets containing the English NaturalQuestions and the cross-lingual XOR-TyDi QA, with data augmentation based on translation.
Additionally, to isolate the effect of contrastive pre-training, we also compare mContriever fine-tuned MS MARCO to its counterparts without contrastive pre-training, initialized from mBERT. This model is referred as mBERT + MSMARCO in tables. We also report results obtained by fine-tuning XLM-R (Conneau et al., 2019) on MS-MARCO. For both models we use the same sapeer-arramters used to finc-tune mContriover on MS MARCO except for the temperature, additional details are reported Appendix B.2.
5.5 Results
We report results on Mr. TyDi in Table 4. The effectiveness of the multilingual pre-training appears clearly as the pre-trained model fine-tuned on MS MARCO achieve significantly beller performance than its counterparts without pre-training when fine-tuned using the same pipeline. Interestingly fine-tuning on these English-only datasets improves performance on all languages. Morcover our unsupervised mContriever outperforms BM25 for the Recall@ 100, and after fine-tuning on MS MARCO achieves state-of taaf-thh-art performance for this metric. Performance be further improved by fine-tuning the train set of Mr. TyDi. This appears to be
10

---

Published in Transactions on Machine Learning Research (08/2022)
Table 5: Cross-lingual retrieval on MKQA. We report the average on all languages included in MKQA for the Recall@100 and the Recall and report the Recall 100 for subsct of languages. Complete results are reported in Table 13 and Table 14 of appendix.
Avg. R@200
Avg. R@100
en
ar
ja
ko
he
de
CORA (Asai et al., 2021)
49.0
59.8
75.6
44.5
47.0
45.5
69.2
48.3
68.1
mBERT + MS MARCO
45.3
57.9
74.2
44.0
51.7
48.2
63.9
46.8
59.6
XLM-R + MS MARCO
46.9
59.6
73.4
42.5
53.2
49.6
63.4
46.9
61.1
mContriever
31.4
49.2
65.3
43.0
47.1
44.8
37.2
44.7
49.0
+ MS MARCO
53.9
65.6
75.6
53.3
60.4
55.4
70.0
59.6
66.6
Table 6: MoCo vs. in-batch negatives. In this table, report nDCG@10 on the BEIR benchmark for in-batch negatives and MoCo, without fine-tuning on the MS MARCO dataset.
NFCorpus
NQ
FIQA
ArguAna
Quora
DBPedia
SciDocs
FEVER
AVG
MoCo In-batch negatives
26.2
13.1
13.7
33.0
69.5
20.0
11.9
57.6
30.1
24.2
21.6
13.0
33.7
74.9
17.9
13.6
56.1
31.9
particularly important for the MRR@100 metric which put emphasis on the quality of the first documents retrieved. For this metric our unsupervised model still lagging behind BM25.
Results on the cross-lingual retrieval benchmark derived from MKQA are reported in Table 5, with per language details for the Recall@100 and Recall@20 reported in Table 13 and Table 14 of appendix. Interestingly using only supervised training data in English, our mContriever fine-tuned on MS MARCO outperforms the CORA retriever. Also, similarly to the results reported on Mr. TyDi, adding multilingual contrastive pre-training before fine-tuning on MS MARCO improves performance over its counterpart without pre-training. On MKQA, evaluation is performed by lowercasing both queries and documents, we observed that this improves performance. This does not impact the CORA retriever which is based on an uncased version of mBERT.
6 Ablation studies
In this section, we investigate the influence of different design choices on our method. In these ablations, all the models are pre-trained on English Wikipedia for 200k gradient steps, with a batch size of 2,048 (on 32 GPUs). Each fine-tuning on MS MARCO takes 20k gradient steps with a batch size of 512 (on 8 GPUs), using AdamW and no hard negatives.
MoCo vs. in-batch negatives. First, we compare the two contrastive pre-training methods: MoCo and in-batch negatives. As in-batch negatives, the number of negative examples equal to the batch size, we train models with a batch size of 4,096 and restrict the queue in MoCo to the same number elements. This experiment measures the effect of using of momentum encoder for the keys instead of the same network for the queries. Using momentum also prevents from backpropagating the gradient through the keys. We report results, without fine-tuning MS MARCO in Table 6. We observe that the difference of performance between the two methods is small, especially after fine-tuning on MS MARCO, We thus propose to use MoCo as our contrastive learning framework, since it scales to a larger number of negative examples without the need to increase the batch size.
Number of negative examples. Next, we study the influence of the number of negalives used in the contrastive loss, by varying the queue size the MoCo algorithm. We consider values ranging from 2,048 to 131,072, and report results Figure We see that on average over the BEIR benchmark, incrcasing the number of negatives leads to better retrieval performance, espocially in the unsupervised setting, However, note that this effect not equally strong for all datasets.
11

---

Published in Transactions on Machine Learning Research (08/2022)
MS MARCO
TREC COVID
NFCorpus
NaturalQuestionn
HotpotQA
40
50
32
50
30
40
30
28
40
30
50
20
30
24
26
20
40
10
20
22
10
30
211
2
2
2
2
2
2
2
2
2
2
2
2 2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
FIQA
ArguAna
Touche-2020
Quora
DBPedia
30
45
85
20
40
2.5
40
18
80
20
16
75
30
35
15
70
20
10
30
65
211
2
2
2
21h
2
2
21
2
2
2
2
2
2
2
2
2
2
2
2
217
2
2
2
2
2
2
2
2
2
2
2
2
27
SCIDOCS
80
FEVER
255
C.IMATB-FEVER
70
SciFacts
Average
70
20
65
40
14
60
15
60
35
12
30
10
50
10
55
241
24
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
212
2
2
2
Without MSMARCO
With MSMARCO
Figure 2: Impact of the number of negatives. We report nDCG@10 as function of the queue size, with and without fine-tuning MS MARCO. We report numbers using the MoCo framework where the keys for the negatives are computed with the momentum encoder and stored in queue.
Data augmentations. Third, we compare different ways to generate pairs of positive examples from a single document or chunk of text. In particular, we compare random cropping, which leads to pairs with overlap, and the inverse cloze task, which was previously considered to pre-train retrievers. Interestingly, as shown in Table 7, the random cropping strategy outperforms the inverse cloze task in our setting. We believe that random cropping, leading to the identical distributions of keys and queries, leads more stable training with MoCo compared to ICT. This might explains part of the difference of performance between the two methods. We also investigate whether additional data perturbations, such random word deletion replacement, are beneficial for retrieval.
Training data. Finally, we study the impact of the pre-training data on the performance of our retriever, by training on Wikipedia, CCNet mix of both sources of data. We report results in Table 8, and observe that there is clear winner between the two data sources. Unsurprisingly, training the more diverse CCNet data leads to strong improvements on datasets from different domains than Wikipedia, such as FiQA or Quora. On the other hand, on a dataset like FEVER, training on Wikipedia Icads to better results. To get the best of both worlds, we consider two strategies to mix the two data sources. In the "50/50%" strategy, examples are sampled uniformly across domain, meaning that half the batches are from Wikipedia and the other half from CCNet. In the "uniform" strategy, examples are sampled uniformly over the union of the dataset. Since CCNet is significantly larger than Wikipedia, this means that most of the batches are from CCNet.
Impact of fine-tuning on MS MARCO. To isolate the impact of pre-training from the impact of fine-tuning on MS MARCO, we apply the same fine-tuning to the BERT base uncased model. We report results Table 9, and observe that when applied to BERT, our fine-tuning leads results that are lower than the of Hence, we believe that most of the improvement compared to the state-of the-art. reevess be attributed to our contrastive pre-training strategy.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Table Impact of data augmentions. We report nDCG@10 without fine-tuning on MS MARCO.
NFCorpus
NQ
ArguAna
Quora
DBPedia
SciDocs
FEVER
Overall
ICT
23.2
19.4
31.6
27.6
21.3
10.6
55.6
25.9
Crop
27.6
17.7
35.6
75.4
21.0
13.3
64.5
32.2
Crop + delete
26.8
20.8
35.8
77.3
21.5
14.0
67.9
33.8
Crop + replace
27.7
18.7
36.2
75.6
22.0
13.0
66.8
32.9
Table 8 Training data.
We report nDCG@10 without fine-tuning MS MARCO.
NFCorpus
NQ
FIQA
ArguAna
Quora
DBPedia
SciDoes
FEVER
Overall
Wiki
27.6
17.7
16.3
35.6
75.4
21.0
13.3
64.5
33.0
CCNet
29.5
25.8
26.2
35.2
80.6
20.5
14.9
60.9
34.9
Uniform
31.0
19.4
25.1
37.8
80.4
21.5
14.7
59.8
33.9
50/50%
31.5
18.6
23.3
36.2
79.1
22.1
13.7
64.1
34.7
Table 9: Fine-tuning. We report nDCG@10 after fino-tuning BERT and our model on MS MARCO.
NFCorpus
NQ
FiQA
ArguAna
Quora
DBPedia
SeiDocs
FEVER
Overall
BERT
28.2
44.6
25.9
35.0
84.0
13.0
69.8
42.0
Contriever
33.2
50.2
28.8
46.0
85.4
38.8
16.0
77.7
46.5
7 Discussion
In this work, we propose explore the limits of contrastive pro-training to learn dense text retrievers. We use the MoCo technique, which allows to train models with a large number of negative examples. We make several interesting observations: first, we show that neural networks trained without supervision using contrastive learning exhibit good retrieval performance, which are competitive with BM25 (albeit not state-of-theeart). These results can be further improved by fine-tuning on the supervised MS MARCO dataset, leading to strong results, in particular for re0all@100. Based on that observation, we use cross-encoder to re-rank documents retrieved with our model, leading to new state-of-the-ars on the competitive BEIR benchmark. We also performed extensive ablation experiments, and obscrved that independent random cropping scems be a strong alternative to the inverse Cloze task for training retrievers.
References
Akari Asai, Xinyan Yu, Jungo Kasai, and Hannanch Hajishirzi. One question answering model for many languages with cross-lingual dense passage retrieval. CoRR, abs/2107.11977, 2021. URI. https://arxiv. org/aba /s//bb2/200.. 11976. 10, 11
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension datasct. arXiv preprint arXiu:1611.0922 2016. 3,
Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. Bridging the lexical chasm: Proceedings of the 23nd annual international ACM SIGIR
statistical approaches to answer-finding conference on Research and development in information relrieval, pp. 192 199, 2000.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.1420, 2021.
2
Wei-Cheng Chang, Felix X Yu, Yin- Men Chang, Yiming Yang, and Sanjiv Kumar. Pro-training tasks for embedding- large-scale retrieval. arXio preprint arXi::2002.0800 039322, 2020. 3
13

---

Published in Transactions on Machine Learning Research (08/2022)
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia answer open-domain questions. In Proc. ACT, 2017a.
Ting Chen, Yizhou Sun, Yue Shi, and Liangjic Hong. On sampling strategies for neural network-based collaborative filtering- arXiv preprint vrrXix1770.000 07881, 2017b. 5
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR,
2020. 2, 4, 5
Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. Salient phrase aware dense retrieval: Can dense retriever imitate a sparse one?, 2021. URL
3
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi QA: A benchmark for informatiion-seeknng question answering in lypologically diverse languages. CoRR, abs/2008 hbs/2003.05002, 2020. URL https: wttp://wwww..bbb//...00050 05002. 9
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. CoRR, abs/ /////9911.01111 2019. URL 9, 10, 19
02116.
Scott Decrwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American sociely for information science, 41(6))391-400) 1990.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019. 3, 4
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self ceff-supervised learning for language understanding. arXiv preprint orxi::205. 12766, 2020. 3
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXio:2109. 2021. 7
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint "5i:2108.0554. 2021. 3
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings.
arXiv preprint ar Xix:21104, 00::2104.08821, 2021. 5, 7
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. End-to-end retrieval in continuous space. arXiv preprint arXie:1811.08000 2018. 3
John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval-augmented
language model pro-training. arXiv preprint +Xix:200200809, 2020. 3, 5,
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020. 2, 6, 17
Scbastian Hofstätter, Sheng-Chic Chich Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. arXio preprint arXiv:2104.06967 2021.
Po-Sen Huang, Xiaodong Hc, Jianfeng Gao, Li Deng, Alex Acero, and Larry Hock. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference Information & Knowledge Management, pp. 2333 2338, 2013. 4
14

---

Published in Transactions on Machine Learning Research (08/2022)
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. ar preprinl arXiv:1905.0196: 2019. 3
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering, 2020a. URL https: //axxiv. http://ari...or/aa///0/011. httg//bs//012.04584... 3, 7
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint 0122022, 2020b. 6
Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. A memory efficient baseline for open domain question answering. arXiv preprint orXiv:2012.151: 2020. 3
Yacine Jernite, Samuel Bowman, and David Sontag. basee objectives for fast unsupervised sentence representation learning. arXiv preprint urXiv:1705.0055: 2017. 3
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019.
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documenlation, 1972. 2
Mandar Joshi, Eunsol Choi, Danicl S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge datasel for reading comprehension. In Proc. ACL, 2017. 6, 7
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2044. 04906, 2020. 3, 4, 5, 7
Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert, 2020. 3,
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019. 1, 6, 7
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: adaptive sharpness-awarr aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiu:2102. 1600, 2021. 17
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXie preprint xrXiv:1909.11900 2019. 3
Kenton Loe, Ming- Wei Chang, and Kristina Toutanova. Latent retricval for weakly supervised open domain question answering. In Proc. ACL, 2019. 2, 3, 4, 5, 6
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. arXiv preprint 20x6:20066. 15020, 2020.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained transformers for text ranking: Bert and beyond. 0at:22010064467 2020. 2
arXiv preprint
Shayne Longpre, Yi Lu, and Joachim Daiber. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. CoRR, abs/2007.15200) 15207, 2020. URT. httph: ttrg/abs//000
9
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 17, 18
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. arXio preprint arXiv22005.00181 2020.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. A replication study of dense passage retriever, 2021.
Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan. Introduction to information retrieval. Cambridge university press, 2008. 2
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Coco-Im: Correcting and contrasting text sequences for language model pretraining. arxiv preprint arXiv:2102.08/77 2021.
Tomas Mikolov, Tya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionallyy. In Advances in neural information processing systems, pp- 3111-31 11119,119, 2013. 3
Bhaskar Mitra, Nick Craswell, et al. An introduction to neural information retrieval. Foundations and Trends@ in Information Retrieval, 13(1):1-126, 2018.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS, 2016. 1
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXio preprint arXio:1901.0405 2019. 2, 4
Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions Audio, Speech, and Language Processing, 24(4):694-707, 2016. 2
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang- Rocket An optimized training approach to dense passage retrieval for open- domain question answering. In Proceedings of the 2021 Conference of the North American Chopter of the Association for Computational Linguislics: Human Longuage Technologies, pp. 5835-5841 5847, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 100/6633/1//2 w0t0v122211la/a wwwcl-main.nni.... URL attps://aclanthol.. www.:/loooyyrgg/221111// www.:/.............aaaaaa.aaaaaa maacl-main. 466.
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. Learning to retrieve passages without supervision, 2021. URL https://arxiv org/abs. 2abs//21.00778. 3
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXix:1908. 10084, 2019. 3,
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance frumework: BM25 and beyond. Now Publishers Inc, 2009.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Bancockk-eeallenn Mike Gatford, et al. Okapi at TREC-3. NIST Special Publication Sp, 1995.
Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open- open-domain question answering, 2021. 2,
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd international conference world wide web, pp. 373-374, 2014. 2
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heteroge- nous benchmark for zero-shot evaluation of information retricval models. arXiv preprint 2xXX::214400666. 2021. 2, 6, 18, 19
James Thorne, Andreas Vlachos, Christos Christodonlopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355, 2018.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Guillaume Wenzek, Marie-Ame Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CONet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, 2020. 6, 9, 17
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733 3742, 2018. 2, 4,
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXto:2012.15400 2020.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprinl orii:2007.00800, 2020. 3, 4,
Sohee Yang and Minjoon Seo. Is retriever merely an approximator of reader? arXiv preprint Xrx: 10999, 2020.
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi lingtll benchmark for dense retrieval. CoRR, abs/ 0ts/2108..8777,, 2021. URL
08787 9, 10
A Technical details for Contriever
A.1 Contrastive pre-training
For the model with fine-tuning on MS MARCO, we use the MoCo algorithm He et al. (2020) with queue of size 131,072, momentum value of 0.9995 and temperature of 0.05. We use the random cropping data augmentation, with documents of 256 tokens and span sizes sampled between 5% and 50% of the document length. Documents are simply random piece of text sampled from mix between Wikipedia and CCNet data (Wenzek et al., 2020), where half the batches are sampled from each source. We also apply loken deletion with a probability of 10%. We optimize the model with the AdamW (Toshchilov & Hutter, 2019) optimizer, with learning rate of 5. 10-5, batch size of 2,048 and 500,000 steps. We initialize the network with the publicly available BERT base uncased model.
A.2 Fine-tuning on MS MARCO
For the fine-tuning on MS MARCO we do not use the MoCo algorithm and simply use in-batch negatives. We use the ASAM optimizer (Kwon al., 2021), with learning rate of 10-5 and a batch size of 1024 with a temperature of 0.05, also used during pre-training. We train an initial model with random negative examples for 20000 steps, mine hard ivgatives with this first model, and re-train a second model with those. Each query is associated with a gold document and a negative document, which is a random document in the first phase and a hard negative 10% of the time in the second phase. For each query, all documents from the current batch aside of the gold document are used as negatives.
A.3 Few-shot training
For the few-shot evaluation presented in Table 3, we train for 500 epochs on each dataset with a batch size of 256 with in-batch random negatives. We evaluate performance performance on the development sot every 100 gradient updates and perform early stopping based on this metric. For SciFact, we hold out randomly 10% of the training data and them development set, leading to train set containing 729 samples.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Table 10: BEIR Benchmark. We report the recall 100 on the test from the BEIR benchmark for bi-encoder methods. We report the capped recall 100 Trec-COVID following the original BEIR setup. Note that using cross-encoder to re-rank the top-100 documents do not change the recall@ 100, hence, we do not include these methods in this table. We also report the average and number of datasets where method is the best ("Best on") over the entire BEIR benchmark (excluding three datasets because of their licence). Bold is the best overall. On Crvc we report the capped Recall see Thakur et al. (2021) for more details. MS MARCO is excluded from the average.
BM25
DPR
ANCE
TAS-B
Gen-Q
ColBERT
Splade v2
Ours
MS MARCO
65.8
55.2
85.2
88.4
88.4
86.5
89.1
Trec-COVID
49.8
21.2
45.7
38.7
45.6
46.4
12.3
40.7
NFCorpus
25.0
20.8
23.2
28.0
25.4
27.7
30.0
NQ
76.0
88.0
83.6
90.3
86.2
91.2
93.0
92.5
HotpotQA
74.0
57.8
72.8
67.3
74.8
82.0
77.7
FIQA
53.9
34.2
58.1
59.3
61.8
60.3
62.1
65.6
ArguAna
94.2
75.1
93.7
94.2
97.8
91.4
97.2
97.7
Touche-2022 2020
53.8
30.1
45.8
43.1
45.1
43.9
35.4
29.4
CQADupStack
60.6
40.3
57.9
62.2
65.4
62.4
66.3
Quora
97.3
47.0
98.7
98.6
98.8
98.9
98.7
99.3
DBPedia
39.8
34.9
31.9
49.9
43.3
46.1
57.5
54.1
Scidoes
35.6
21.9
26.9
33.5
33.2
34.4
36.4
37.8
Fever
93.1
84.0
90.0
93.7
92.8
93.4
95.1
94.9
Climate-fever
43.6
39.0
44.5
45.0
44.4
52.4
57.4
Scifact
90.8
72.7
81.6
89.1
89.3
87.8
92.0
94.7
Avg. w/o CQA
63.6
48.3
60.1
65.0
64.2
64.5
64.8
67.1
Avg.
63.4
47.7
60.0
64.8
64.2
64.3
67.0
Best
0
0
1
0
7
B Multilingual retrieval with mContriever
B.1 Hyperparameters for multilingual contrastive pre-training
The pre-trained mConttrever model is pro-trained for 500,000 steps with quene of size 32768, and temperature of 0.05 and momentum value of 0.999. We optimize the model with the AdamW (Loshchilov & Hutter, 2019) optimizer, with learning rate of 5 10-5, The learning rate follows a linear warmup for 20,000 steps followed by linear decay until the end of training. Languages used for pre-training are detailed Table 12.
B.2 Hyperparameters for multilingual fine-tuning
We fine-tune mCoottriver using in-batch negatives, Adamw optimizer (Loshchilov & Hutter, 2019), a learning rate of 10-5, and batch size 1024 samples with a temperature т of 0.03. On MS MARCO and Mr. TyDi the model is trained for 20k gradient steps. We notice overfitting on NaturalQuestions, and thus reduced the training to 1k gradient steps. We use a warmup of 1000 gradient steps with lincar decay afterwards in all mined on Mr. TyDi with the model trained MS MARCO. We did not observe significant improvements using hard negatives MS MARCO and NaturalQuestions.
cases. Hard negatives
For the fine-tuning on MS MARCO of the models initiliazed from mBERT (resp. XLM-R) without contrastive pre-training, we use a temperat ure ure - of (resp. 5). We tried lemperatures in {10,5, 2, 1, 0.1, 0.05) and chose the one leading to the best performance. We observed a decrease in performance for lower temperatures. We fino-tuned mContriever with 0.05 following the temperature used during pre-training. We followed the temperature 7 0.05 used for the training of Contriever, and did not test other temperatures for the contrastive pre-training of the multilingual model, mContriever.
18

---

Published in Transactions on Machine Learning Research (08/2022)
Unsupervised retrieval. Performance of unsupervised methods on the BEIR datasets. We report the capped recall Trec-COVID following the original BEIR setup. For SimCSE report results of the model using RoBERTa large. REALM uses annotated entity recognition data for training, On Treo-COVID we report the capped Recall@100 see Thakur et al. (2021) ) for more details.
Table 11:
Model (-)
BM25 BERT
SimCSE REALM
Contriever
Dataset (4)
Recall 100
MS MARCO
65.8
3.5
33.6
52.6
67.2
Tree- COVID
49.8
10.6
26.8
8.1
17.2
NFCorpus
25.0
6.7
18.2
23.0
29.4
NQ
76.0
14.3
42.9
58.1
77.1
HotpotQQ
74.0
15.8
42.7
56.1
70.4
FiQA-2018
53.9
6.9
41.0
28.0
56.2
ArguAna
94.2
59.1
95.2
73.1
90.1
Tóuche-2020
53.8
3.0
18.6
11.5
22.5
CQADupStack
48.9
61.4
Quora
97.3
74.6
97.9
92.7
98.7
DBPedia
39.8
7.1
21.5
33.0
45.3
SCIDOCS
35.6
11.3
23.0
23.1
36.0
Fever
93.1
13.6
50.8
93.6
Climate-fever
43.6
12.8
44.8
42.3
44.1
SciFact
90.8
35.2
75.3
83.8
92.6
Avg.
63.6
19.0
45.4
46.9
60.1
Best
2
10
NDCG@ 10
MS MARCO
22.8
0.6
8.8
15.2
20.6
Trre-COViL
65.6
16.6
38,6
20.1
27.4
NFCorpus
32.5
2.5
14.0
21.1
31.7
NQ
32.9
2.7
12.6
15.2
25.4
HotpotQA
60.3
4.9
23.3
40.5
48.1
F1QA-2018
23.6
14.8
9.7
24.5
ArguAna
31.5
23.1
45.6
22.8
37.9
Touche-2020
36.7
11.6
19.3
CQADupStack
29.9
2.5
20.2
13.5
28.4
Quora
78.9
3.9
81.5
71.6
83.5
DBPedia
31.3
3.9
13.7
22.7
29.2
SCIDOCS
15.8
2.7
7.4
9.0
14.9
FEVER
75.3
4.9
20.1
42.9
68.2
Climate-fever
21.3
4.1
17.6
14.3
15.5
Scilact
66.5
9.8
38.5
47.1
64.9
Avg.
41.7
8.7
24.6
25.1
36.0
Best
12
0
1
2
B.3
Curse multilinguality
We tried to pre-train models on different sets of languages. We generally observed performance deterioration when scaling to more languages similarly to what has been observed for general multilingual masked language models Conneau et al. (2019). In Table 15 we report results on Mr. TyDi with model pre-trained the 11 languages of Mr. TyDi versus the model used in the rest of the paper which has been pre-trained on 29 languages including the 11 languages of Mr. TyDi as detailed Table 12. We also report performance of these models after training on MS MARCO, eventually followed by further fine-tuning on Mr. TyDi. It appears that the performance of the unsupervised model and the performance after fine-tuning on MS MARCO are better for the model pre-trained only on languages. The difference is mitigated after mne-tuning on Mr. TyDi.
19

---

Published in Transactions on Machine Learning Research (08/2022)
Table 12: List of languages used for multilingual retrieval.
bn Bengali
da Danish
de German
Language Pre-training
Arabic
English
Spanish
TyDi
X
MKQA
hm Hungarian
Language Pre-training
Finnish
French
Hebrew
Italian
Indonesian
Mr. TyDi
/
x
MKQA
km
ko
al
Language Pre-training Mr. TyDi
Japanese
Khmer
Korean
Malay
Dutch
Norwegian
x
x
x
MKQA
pl
pt Russian
ru
Sv Swedish
sw Swahili
te
Language Pre-training TyDi
Polish
Portugese
Telugu
X
MKQA
zh-en Chinese (Simplified)
zh-hk
zh-tw Chinese (Traditional)
Language
Thai
Turkish
Vietnamese
Chinese (Hong Kong)
Pre-training
Mr. TyDi
X
X
MKQA
Table 13: Recall@100 on MKQA for cross-lingual retrieval in the setting described in Section 5.3.
avg
fi
ko
he
th
da
de
Ir
CORA
59.8
75.6
44.5
61.3
47.0
45.5
69.2
65.0
44.4
68.9
68.1
70.2
mBERT + MS MARCO
57.9
63.9
51.7
63.7
59.6
XLM-R MARCO
59.2
42.4
37.7
48.6
62.9
67.5
61.5
66.9
60.9
Contriever
49.2
43.0
43.1
44.8
51.8
37.2
34.5
44.7
51.4
49.3
49.0
MS MARCO
65.6
53.3
66.6
55.4
70.0
70.8
63.5
72.0
66.6
70.1
pl
pi
hu
m8
km
no
Lr
zh-cn
xh- hhk
zh-tw
CORA
68.3
72.0
65.6
67.9
59.5
61.2
67.9
35.6
68.3
61.5
52.00
52.8
52.8
mBER + MS MARCO
64.1
66.7
59.0
61.9
57.5
58.6
62.8
32.9
63.2
56.0
58.4
59.3
59.3
XLM-R MS MARCO
58.1
66.4
61.0
62.0
60.1
62.4
66.1
46.6
65.9
60.6
55.8
55.5
55.7
Contriever
56.7
61.7
44.4
54.5
47.7
45.1
56.7
27.8
50.2
44.3
54.3
51.9
52.5
MS MARCO
70.3
71.4
68.8
68.5
66.7
67.8
71.6
37.8
71.5
68.7
64.1
64.5
64.3
Table 14: @ecall@20 on MKQA for cross-lingual retrieval in the setting described in Section 5.3.
avg
ko
th
de
CORA
49.0
68.5
31.7
49.7
34.1
33.1
46.5
60.3
58.1
36.8
33.6
59.4
58.5
61.6
mBERT + MS MARCO
45.3
65.5
30.2
35.9
41.7
31.5
44.3
52.4
30.5
32.6
38.5
52.5
46.6
53.8
XLM-R MS MARCO
46.7
64.5
29.0
45.1
39.7
31.9
45.9
51.4
56.1
32.5
49.4
55.8
45.3
50.5
Contriever
31.4
50.2
26.6
26.7
29.4
27.9
32.7
20.7
37.6
22.2
31.1
31.2
31.2
30.7
MS MARCO
53.9
67.2
40.1
55.1
46.2
41.7
52.3
59.3
60.0
45.6
52.0
62.0
54.8
59.3
it.
m]
pl
hu
vi
ms
km
fr
zh-cn
sh-hk
zh-tw
CORA
58.2
63.5
54.3
58.4
47.6
49.8
57.6
24.8
58.8
49.1
38.6
40.5
39.6
mBER MS MARCO
52.1
55.3
45.6
49.5
44.6
16.9
49.9
21.5
51.3
42.7
44.6
45.3
45.5
XLM-R MS MARCO
45.4
51.5
48.5
47.3
51.0
33.4
53.7
48.7
42.4
42.4
42.0
Contriever
38.6
45.1
25.1
37.6
28.3
27.3
39.6
15.7
33.2
26.5
35.0
32.7
32.5
MS MARCO
59.4
60.9
58.1
56.9
55.2
55.9
60.9
26.2
61.0
56.7
50.9
51.9
51.2
20

---

Published in Transactions on Machine Learning Research (08/2022)
Table 15: Performance dilution for multilingual retrievers. We report MRR@100 and R@100 on the test set of Mr. TyDi after pre-training on two different sets of languages, one containing the 11 languages of Mr. TyDi which is included in the set of 29 languages used to train mContriever used in the rest of the paper. We also report results of these models after fine-tuning on MS MARCO, potentially followed by a final fine-tuning stage on Mr. TyDi.
ar
bn
en
fi
ja
ko
ru
SW
th
avg
MRR@100
11 languages
28.9
38.6
10.2
22.9
26.2
21.7
26.3
20.5
39.0
20.2
40.6
26.8
29 languages
27.3
36.3
9.2
21.1
23.5
19.5
22.3
17.5
38.3
22.5
37.2
25.0
MS MARCO
11 languages
44.0
41.1
26.8
38.3
43.4
34.7
37.1
37.6
55.3
32.1
45.8
39.7
29 languages
43.4
42.3
27.1
35.
42.6
32.4
34.2
36.1
51.2
37.4
40.2
38.4
Mr. TyDi
11 languages
73.5
66.8
55.9
60.4
62.7
53.8
55.8
60.6
69.5
89.8
68.7
65.2
29 languages
72.4
67.2
56.6
60.2
63.0
54.9
55.3
59.7
70.7
90.3
67.3
65.2
R@100
11 languages
83.4
89.2
55.2
81.5
83.6
75.6
72.4
75.7
88.9
70.0
91.9
78.9
29 languages
82.0
89.6
48.8
79.6
81.4
72.8
66.2
68.5
88.7
80.8
90.3
77.2
MS MARCO
11 languages
89.6
91.9
78.7
89.0
91.2
83.4
80.2
86.0
92.6
95.9
92.8
88.3
29 languages
88.7
91.4
77.2
88.1
89.8
81.7
78.2
83.8
91.4
96.6
90.5
87.0
Mr. TyDi
11 languages
94.2
98.2
93.3
93.6
94.7
89.1
87.1
92.3
94.5
98.9
96.9
93.9
29 languages
94.0
98.6
92.2
92.7
94.5
88.8
88.9
92.4
93.7
98.9
95.2
93.6
21


Reviews:
[{"review_id": "8ShG0gjone", "reviewer": "Reviewer_aLmD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nIt proposes and develops an interesting unsupervised method for dense retrieval based on contrastive learning that is inspired by data augmentation methods in vision. The novelty with respect to prior work by Lee et al. (2019) is the way positive examples are built from a document, namely by sampling two different text spans allowing for overlap as opposed to contiguous, mutually exclusive spans.\n\nThe empirical investigation is thorough and covers a range of experiments ranging from few-shot retrieval, zero-shot and multilingual retrieval with competitive results in all cases. This provides solid evidence for the effectiveness of the approach and it also provides ablation experiments that justify the design choices.  The comparison includes previous data augmentation approaches for similar tasks based on inverse cloze tasks.\n\nWriting is clear and the general positioning with respect to prior work is quite good. It is important to emphasize classical baselines for these tasks and improving over them is important and should be of interest to the relevant retrieval community.\n\n**Weaknesses** \n\nConceptually the proposed idea is similar to existing techniques and it could be viewed as incremental. For instance, the bi-encoder architecture is pretty common, with contrastive loss, and negative handling. \n\nImprovements in multilingual retrieval appear to be based on pretraining on 29 languages which can in principle be applied to mBERT. Even though the gap between them is big, it is not clear if the improvement is due to the additional data or the superiority of the learning contrastive objective.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Recent studies on document retrieval have shown that dense pre-trained retrievers on a large retrieval dataset tend to be outperformed by classical frequency-based methods when tested on new domains and such datasets are hard to collect for different languages. This paper proposes to train dense retrievers in an unsupervised way (without relying on labeled data) using contrastive learning on an auxiliary task that approximates retrieval. Evaluation on few-shot settings shows competitive performance compared to pretraining on large labeled datasets during pretraining. It also appears to be the first example of reaching competitive performance to classical methods when pre-trained on large datasets.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The description should emphasize the novel aspects and make clear what parts are new compared to key most related prior work such as Lee et al. (2019) beyond the empirical results.\n\nThe method section has some phrasing that suggests existing ideas are proposed for the first time e.g. using the independent encoding of the query and document which is a bit misleading (e.g. sBERT). For contrastive learning, I'd suggest citing the ICT paper which used a similar objective, and then pointing out the differences.\n\nFor the multilingual experiment, the authors should clarify whether the mBERT baseline is pre-trained on the same data as the proposed method. If not, I would suggest adding this comparison for completeness. \n\nIn experiments that require finetuning, it would be interesting to report what is the parameter count for each method where applicable to get a better understanding of the differences.\n\n", "broader_impact_concerns": "The work relies on big pre-trained models for which there are ethical concerns regarding discrimination, exclusion, misinformation, malicious uses, and so on. Previously they have been used for document retrieval but it would be good to add a broader impact statement to discuss such concerns in this area.  ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FjE77QlMgU", "reviewer": "Reviewer_CTNi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strength: \n- Impressive empirical results on important tasks. \n- Simple, well-motivated method that works well.\n\n* Weakness:\n- There are some issues with clarity in writing, especially with regard to experimental set-ups, see detailed comments in \"requested changes\" section. \n- Comparison with related work/baselines can be much clearer. The work is evaluated in *many* different settings, which is definitely the strength of the paper, but the comparison is often not very clear.. (see requested changes section again).\n- Lack of study on computational efficiency: it’d be very helpful to add a discussion on efficiency — both in terms of speed (both training and inference time) and memory requirements. One of the major bottlenecks of building/using retrieval systems is their large memory footprint. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* The paper presents a new, high-performing dense passage retriever. \n* The proposed method is relatively simple, built upon previously introduced techniques for self-supervised learning. From what I understand, two ingredients that make this model work better than the previous iteration of unsupervised retriever -- (1) using MoCo (Momentum Contrast) training to enable large negative batch efficiently and (2) using independent cropping instead inverse cloze task\n* The model is evaluated extensively -- on the English QA benchmark, IR benchmark (BEIR), and multilingual QA retrieval benchmark dataset, and shows impressive performance across the board. \n* The code/model will be publicly available, and I anticipate many people will use this for future work. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The experimental setup in this paper is not very clear overall -- few things that need to be clarified:\n* The setting of the training data construction setting was not super clear to me. In the end, did you use Crop + delete? Did you also use ICT? What was the final positive training set look like, and how they are sampled? \n* I don't understand “+CE” parts on Table 2  — how do you train the cross-encoder here? \n---\n* I'm not very convinced that crop + delete is better because it provides symmetric views -- I think this could potentially be because it encourages more lexical match. I wonder if increasing the proportion of positive training examples where the query sentence is kept (currently set to 10%) can improve the performance of ICT. As this is one of the main technical differences with prior work, a bit more rigorous study would be helpful. \n--\n* As mentioned in \"weakness\", adding a discussion on the computational aspect would be great -- although I assume at inference time the computational costs would be similar to other dense retrieval models.\n* Is inverse cloze task / masked salient spans baselines (in table 1) comparable in terms of the amount of training data used?\n--\nA few missing related work:\n* Learning to Retrieve Passages without Supervision: https://arxiv.org/pdf/2112.07708.pdf  --> this is particularly relevant. it's very recent work (which, actually cites results from this paper), but the idea of unsupervised retrieval is very related and the paper will benefit from discussing it together. Overall, I feel this paper didn't really do a great job describing baseline/comparison systems clearly... For example, salient span masking comparison model was introduced a bit out of the blue when discussing baseline/results, maybe mention it a bit when introducing positive training example generation would be helpful. \n* Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? https://arxiv.org/pdf/2110.06918.pdf \n\n\nMore minor comments... \n* The supervised results in Table 1 should be updated with more recent numbers, for example.. https://arxiv.org/pdf/2104.08253.pdf\n* Why not use nDCG metric for multilingual studies? \n* XORQA-Retrieval would be a good benchmark dataset to evaluate cross-lingual retrieval.\n* The writing of the introduction can be improved — it’s not super clear how ICT and your approach differ from reading the introduction. \n* The second paragraph in the introduction “As training neural networks requires large amount of data” —> no longer true for various NLP tasks which can exploit pre-trained language models. \n* I feel abstract is a bit overclaiming — on average it still underperforms BM25. \n* I would prefer to have an ablation study earlier than later, probably after when you introduce various design choices. \n* On page 5, might be good to provide concrete numbers of necessary batch sizes —  “extremely large batch sizes to work well”. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pc7VczTGeX", "reviewer": "Reviewer_dx1v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The work achieves very strong results and advances the state of the art on an important task: being able to to perform high-quality retrieval across domains and languages \n* The experiments and approach are described in sufficient detail, although sometimes the information takes a while a to find\n* There are good ablations to tease apart the importance of the points of departure wrt prior work (with a small suggestion), and strong baselines from prior work are selected\n\n## Weaknesses\n\n* There is an emphasis on comparing to the state of the art and demonstrating strong performance and it takes effort to find the place where differences to prior work and main factors for the performance differences are described\n\n* The impact of example formulation (in comparison to ICT) is not fully ablated -- I would suggest an additional experiment on that\n\n* When other methods are compared to in figures and Tables, the differences to the proposed method are not articulated \n\n* There are a number of typos which I will list in the requested changes section, some technical imprecision, and imprecision in the claims\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* The paper shows that unsupervised (or self-supervised) neural bi-encoder retrievers can do better than previously thought and can be on par with BM25 as measured on the BEIR benchmarks, especially on Recall@100\n\n* A new method for unsupervised contrastive learning of a retriever, previously proposed in the vision community, is applied to text retrieval, and shown to outperform contrastive learning variants used for text retriever training before (e.g. ICT)\n\n* Departures from prior work in terms of training set, batch sizes, and definition of positive/negative examples are ablated and show that MoCo enables the use of larger batch sizes while not hurting performance, which together with cropping to construct examples and use of a different pre-training dataset lead to improvements across many domains\n\n* The paper also shows that the value of unsupervised pre-training can be higher than the value of training on out-of-domain supervised data\n\n* New state-of-the art results on a subset of the BEIR benchmark are achieved, by further finetuning on MSMARCO and re-ranking with a cross-encoder. The comparison is a fair comparison against prior work also using such labeled data and re-rankers. \n\n* Experiments on multilingual retrieval and cross-lingual retrieval are also very strong. Unsupervised contrastive pre-training of the retriever is shown to bring significant value for multilingual retrieval ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "## More substantial\n\n* I would suggest to emphasize the lessons from the ablations more and refer to the ablations section from earlier parts of the paper, because the ablations are the main learnings regarding the impact of differences from prior work. Also when claiming improvements over SOTA, mention the main factors behind those improvements.\n\n* when comparing data augmentation methods in Table 7, I suggest to evaluate ICT with in-batch negatives as well. The authors mention that random cropping is better for MoCo which assumes similar distributions of queries and documents, but we are not sure whether the improved performance of cropping is due solely to the interaction with MoCo\n\n* Gao & Callan 2021  is referred to as concurrent work, but I don't think a difference of 6 months would count as concurrent; please describe differences and learnings compared to Gao & Callan\n\n* P.3 “Instead, we propose to use a bi-encoder architecture, where documents and queries are encoded independently” . This statement seems to imply bi-encoders are newly presented in this work, rephrase to something like “We follow (cite) to use a bi-encoder instead ..”\n\n* Eq (1) on p.4 – the denominator should include the positive example but the description indicates it only sums over a pool of negative documents.\n\n* Mention the pre-training dataset briefly in the main part of the text, e.g. in 4.1 where I currently see only evaluation datasets. \n\n## Less substantial (typos)\n\nSome example typos are below\n\n* P. 1 “billions of element” → billions of elements\n* P.2 “uses a given sentence as query and predict the context surrounding it” → “uses a given sentence as a query and predicts the context surrounding it”\n* P. 3 “Indices weigh 100GB” \n* P.5 “We use these perturbation” → We use these perturbations\n* P.12 “neural networks .. exhibits good ..” → “neural networks .. exhibit good ..”\n\n\n\n\n", "broader_impact_concerns": "No concern. The work advances the state of the art toward multilingual information access.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers all agreed that this submission should be accepted. Please make the updates you discussed with the reviewers, taking particular heed to clarify novelty w.r.t. existing papers and update the multilingual experiments.
Decision: Accept (poster)