How to Train Your  Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval
Authors: Sheng-Chieh Lin
Akari Asai
Minghan Li
Barlas Oguz
Jimmy Lin
Yashar Mehdad
Wen-tau Yih
Xilun Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Generalizable Dense Retrieval
Data Augmentation
Progressive Training
TL;DR: We propose data augmentation method and a progressive training strategy to train a BERT-base-sized dense retriever, which achieves state-of-the-art effectiveness in both supervised and zero-shot evaluations.

Abstract: Various techniques have been developed in recent years to improve dense retrieval (DR), such as unsupervised contrastive learning and pseudo-query generation. 
Existing DRs, however, often suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, which some argue was due to the limited model capacity.
We contradict this hypothesis and show that a generalizable DR can be trained to achieve high accuracy in both supervised and zero-shot retrieval without increasing model size.
In particular, we systematically examine the contrastive learning of DRs, under the framework of Data Augmentation (DA).
Our study shows that common DA practices such as query augmentation with generative models and pseudo-relevance label creation using a cross-encoder, are often inefficient and sub-optimal.
We hence propose a new DA approach with diverse queries and sources of supervision to progressively train a generalizable DR.
As a result, DRAGON, our Dense Retriever trained with diverse AuGmentatiON, is the first BERT-base-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations and even competes with models using more complex late interaction.

How to Train Your DRAGON:
Diverse Augmentation Towards Generalizable Dense Retrieval
Sheng-Chieh Lin'; Akari Asaï², Minghan Li', Barlas Oguz², Jimmy Lin', Yashar Mehdad Wen-tau Yih and Xilun Chen
University of Waterloo', University of Washington', Meta AI
(s2691in, m69221,,,6991.... jimmylin @wwaterloo..oooooooo ca, akari @cs wwwwwwwtoh.wn.ooommo edu (barlaso, mehdad, scottyih, xilun @meta. com
Abstract
(BEIR maboct: aDCGAOO
50
CIBERT
Various techniques have been developed in re- cent years to improve dense retrieval (DR),
Zeroshot enllationn
46
such as unsupervised contrastive learning and pseudo-query generation. Existing DRs, how- ever, often suffer from effectiveness tradeoffss between supervised and zero-shot retrieval, which some argue was due to the limited model capacity. We contradict this hypothesis and show that generalizable DR be trained to achieve high accuracy in both supervised and zero-shot retrieval without increasing model size. In particular, systematically exam- ine the contrastive learning of DRs, under the framework of Data Augmentation (DA). Our study shows that common DA practices such as query augmentation with generative mod- els and pseudo-relevance label creation using a cross-encoder, are often inefficient and sub- optimal. We hence propose a new DA approach with diverse queries and sources of supervision to progressively train a generalizable DR. As result, DRAGON, our Dense Retriever trained with diverse AuGmentatiON, is the first BERT- base-sized DR to achieve state-of-the-art effec- tiveness in both supervised and zero-shot eval- uations and even competes with models using more complex late interaction.
CL-RBD
44
Rockatjii
12
Thmmue iivel
COTMAE
Voll recter ediced
35
37
Supervised ovaluation (MS MARCO Dev: RRg
Figure 1: Supervised versus zero-shot effectiveness comparison among existing state-of of-the--rt retrievers. All models use BERT-base-sized (1 10M parameters) backbone except for GTR-XXL (4.8B parameters).
to design more expressive representations to cap- ture more fine-grained token-level information; e.g.. SPLADE+ (Formal et al.. 2022) and Col- BERTv2 (Santhanam et al., 2022b) in Figure 1. However, these designs add complexity and latency to retrieval systems (Mackenzie et al., 2021).
By contrast, dense retrieval (DR) is a simpler bi- encoder retriever that maps queries and documents into low-dimensional vectors and computes text similarity through simple dot product. Top-k re- trieval can be performed directly using ANN search libraries (Johnson et al., 2021; Guo al., 2020). Recently, various methods have been proposed improve DR effectiveness while keeping its simple architecture, such as pre-training (Lee et al., 2019;
1 Introduction
Bi-encoder based neural retrievers allow doc- uments to be pre-computed independently of queries and stored, enabling end-to-end retrieval among huge corpus for downstream knowledge- intensive tasks (Karpukhin et al., 2020; Reimers and Gurevych, 2019). Recently, Thakur et al. (2021b) show that bi-encoder retrievers still derperform BM25 in real-world scenarios, where training data scarce. One potential solution is
Chang et al., 2020), query augmentation (Oguz et al., 2022), and distillation (Ren et al., 2021: Zeng et al., 2022). For example, pre-training on MS MARCO corpus improves accuracy in the fully supervised setting while leveraging other corpora can improve transfer in the zero-shot setting. How- ever, improvement in one setting is only achieved at the expense of the other. Figure/ plots exist- ing state-of-the-an DR models with respect to their effectiveness on these two axes, which presents a
"This work is done during Sheng-Chhe -Chnh''s internship Meta. I Xilun and Sheng-Chir Chieh contributed equally to work. 'The code and model checkpoints are available at: ht tps: //gl thub. com/ facebook esearch, /dpr- scale

---

clear tradeoff between supervised and zero-shot effectiveness (the blue line). The only exception. GTR-XXL (Ni al., 2022), breaks the effective- ness tradeoff at the expense of efficiency (i.e., query encoding), which leverages very large pre-trained models with 40 times more parameters. This effec- tiveness tradeoff prompts some to hypothesize that we have fully exploited the capacity of BERT-base- sized DR model (Ni et al., 2022) and explore how cleverly increase model parameters without sac- rificing retrieval efficiency. For example, the recent work (Wang et al., 2022; Dai et al., 2022) proposes to train one expert dense retriever for each specific scenario, resulting in slow adaptation to real-world applications (Asai et al., 2023).
Table I: Categorization of existing DR models by their approaches to data augmentation.
Model
Qry Aug.
Label Aug.
Corpus
RockerAAVZ (een etal. 2021)
MARCO
CL-DRD (Zeng 2022)
cot condndeneer (Gan and Callan, 2022)
MARCO
Conrrever floaand
2021)
cuppiig
Wiki+
COCO-DR (Yu at al., 2022)
BEIR
(Wang 2022)
GenQ
PTR (Dai sial., 2022)
DRAGON
cropping Gien)
reinevers
MARCC
diverse relevance labels from multiple retrievers directly. Thus, we propose simple strategy to pro- gressively augment relevance labels which guides dense retrievers to learn diverse relevance signals more effectively.
Our final model is trained on 28 million aug- mented queries consisting of two types (cropped sentences and synthetic queries), well pro- gressive relevance label augmentation using di- verse (sparse, dense, and multi-vector) retriev- ers. As shown in Figure 1, DRAGON, Dense Retriever trained with diverse AuGmentatiON, is the first dense retriever to break the supervised and zero-shot effectiveness tradcoff without increasing model size retrieval complexity; e.g., GTR-XXL, SPLADE++ and ColBERTv2.
In this work, we contradict this hypothesis and show that a generalizable DR can indeed be trained to achieve state-of-the-an art effectiveness in both su- pervised and zero-shot evaluations without increas- ing model size. To this end, we first investigate the important factors contributing to the recent progress of DR. For example, DR seems to gain zero-shot transfer capability from pre- pre-training on large-scale and diverse training queries (Izacard et al., 2021; Yu et al., 2022) while knowledge dis- tillation can improve the supervision quality by au- tomatically identifying relevant passages which not labeled by humans (Ren et al., 2021; Zeng et al., 2022). To better understand these approaches, we devise a unified framework of data augmentation (DA) for contrastive learning. Under the frame- work, the previous work can be viewed as DA with different recipes of query augmentation and rele- vance label augmentation shown in Table 1. The DA framework also helps design comprehensive studies for better DR training.
We summarize our contributions follows: (I) We conduct a systematic study of DR training un- der the lens of data augmentation, which provides some surprising but key insights into training a generalizable dense retriever; (2) We propose a progressive label augmentation strategy to guide a dense retriever to learn the diverse but complex rel- evance labels; (3) DRAGON, our BERT-base-Ste sized DR, reaches state-of-the-ard retrieval effectiveness in both supervised and zero-shot evaluations.
Background
Guided by detailed empirical exploration along the space of our DA framework, find the fol- lowing: (1) for relevance label augmentation, we identify that the key to training generalizable dense retriever is to create diverse relevance labels for each query, for which we use multiple retriev- ers instead of a strong cross encoder; (2) with such diverse relevance labels, dense retrievers can be trained effectively using cheap and large-scale aug- mented queries (c.g., cropped sentences from corpus) instead of the more expensive neural gen- erative queries. This finding opens the door to further building cheap but useful training data in scale for DR in the future. Finally, we find that it is suboptimal for a dense retriever to learn the
In this section, we first introduce the retrieval task and contrastive learning for dense retrieval. We then provide a unified framework for understand- ing recent approaches to improve dense retrieval training as instances of data augmentation.
2.1 Training Dense Retrieval Models
Given query 4, our task is to retrieve a list of documents to maximize some ranking metrics such as nDCG or MRR. Dense retrieval (DR) based on pre- pre-trained transformers (Devlin et al., 2018; Raf- fel et al., 2020) encodes queries and documents low dimensional vectors with a bi-encoder architec- ture and uses the dot product between the encoded

---

vectors as the similarity score:
DR training with the positive passages predicted by cross encoders. These knowledge distillation approaches further improve training data quality through label augmentation, inspiring us to conduct relevance label augmentation on the augmented queries (i.e., cropped sentences and GenQ).
s(g, d) A CUCLsI CLS
(1)
where egisss and (crss the [CLS] vectors at the last layer of BERT (Devlin al., 2018).
Contrastive Learning is commonly used method for training DR models by contrasting pos-
2.3 Settings for Empirical Studies
itive pairs against negatives. Specifically, given a query q and its relevant document dt,, we minimize the InfoNCE loss:
We introduce some basic experimental settings f'acilitate the presentation of our empirical studies on data augmentation Section 3. More detailed settings can be found in Section 4. Following pre-
exp(s(q,
log
(2)
exp(s(q, )) exp(s(g.d))
vious work (Izacard et al., 2021; Xiao et al., 2022;
Yu et al., 2022; Formal et al., 2022; Santhanam et al., 2022b), we consider MS MARCO (Bajaj et al., 2016) supervised data and BEIR datasets for zero-shot evaluations. Thus, we use the 8.8 million MS MARCO passage corpus to conduct data augmentation and evaluate our trained models on MS MARCO Dev, consisting of 6980 queries from the development set with one relevant passage per query on average. We report MRR @ 10 (abbre- viated as RR @ 10) and Recall @ 1000 (R @ 1K) the evaluation metries. For zero-shot evaluations. BEIR (Thakur et al., 2021b), consisting of 18 IR datasets spanning diverse domains and tasks including retrieval, question answering, fact check- ing, question paraphrasing, and citation prediction. We report the averaged nDCG@ 10 over 13 public BEIR datasets, named BEIR-13, making the num- bers comparable to most existing approaches (For- mal et al., 2021;
2.2 A Unified Framework of Improved Dense Retrieval Training: Data Augmentation
Data augmentation (DA) for contrastive learning has been widely used in many machine learning tasks (Chen et al., 2020; Thakur al., 2021a). In fact, many recent approaches to train better DR, such as knowledge distillation, contrastive pre- training and pseudo query generation (GenQ), can be considered DA with different recipes respec- tively listed in the first three main rows of Table 1. We compare the DA recipes from the perspectives of query and relevance label augmentation. We re- fer readers to Appendix A.9 for more related work of advanced DR training strategies.
Query Augmentation. There are two common au- tomatic approaches to increase the size of training queries from a given corpus, sentence cropping and pseudo query generation. The former can easily scale up query size without any expensive compu- tation, which is used by the models for contrastive pre-training (the second section of Table 1; Gao and Callan, 2022; Izacard al., 2021; Wu al., 2022). The latter generates quality but more ex- pensive human-like queries using large language models for DR pre-training (Oguz et al., 2022) or domain adaptation (the third section of Table I; Wang et al., 2022; Dai et al., 2022). Concurrently to our work, Meng et al. (2023) explore various approaches to query augmentation, such as span selection and document summarization.
Santhanam et al., 2022b).
Pilot Studies on Data Augmentation
In this section, we first discuss the exploration space of data augmentation (DA) based on the framework in Section 2.2 and then conduct empiri- cal studies on how better train dense retriever. Based on the empirical studies, we propose our DA recipe to train DRAGON, a Dense Retriever with diverse AuGmentatiON.
3.1 An Exploration of Data Augmentation
Query Augmentation. Following the discus- sion in Section 2.2, we consider the two com- approaches to automatic query augmenta- tion. Specifically,, for sentence cropping, following Chen et al. (2022), we use the collection of 28 million sentences from the MS MARCO corpus
Relevance Label Augmentation. The aforemen- tioned approaches to query augmentation often as- sume that the (or part of the) original document is relevant to the augmented queries, which may not be true and only provides a single view of relevance labeling. The recent work (the first section of Ta- ble 1; Ren et al., 2021; Zeng et al., 2022) improve
'CQADupStack, Rohust04, Signal-IM, TREC-NEWS, BioASQ are excluded

---

consisting of 8.8 million passages. As for pseudo query generation, we use the 28 million synthetic queries sampled from the query pool generated by doctSquery (Nogueira and Lin, 2019). In addition, we also consider augmenting the type of queries by mixing cropped sentences and synthetic queries.
Labels by remeverr
Labels by renever
Label retnever3
Itraaionn
Itrration2
Iteration
b
Haad
2
2
Label Augmentation with Diverse Supervisions. Although cross encoder (CE) is known to create rel- evance labels with strong supervision, we hypothe- size that CE still cannot capture diverse matching signals between text pairs. A query is often relevant to many documents from different perspectives (e.g., semantic or lexical matching), which can- not capture by a single labeling scheme (a strong model or even human). In this work, we seek mul- tiple sources of supervisions from existing sparse, dense and multi-vector retrievers, which are more efficient than CE and suitable for labeling a large number of queries (see discussion in Section 5).
Sampled triplets (laitch sine B)
Sampleed tripicts (Batch size 3)
Sampled triplets (Banch size 3)
Figure 2: Illustration progressive label augmentation. For each iteration of training, additional relevance labels from a teacher are augmented in the training data. By contrast, uniform supervision directly exposes models to all the supervisions (as in iteration 3) in the beginning.
UI1 This approach naturally encourages the positive samples appearing in more ranked lists be sampled and vice versa. The advantage is that fusion weight tuning is not required. Furthermore, models can see diverse supervisions from different teachers in contrast to fused supervision, which may be dominated by a single strong teacher.
3.2 Training with Diverse Supervisions
We have introduced our searching space for query and label augmentation (with diverse supervisions); however, training a dense retriever on such aug- mented data is not trivial. First, how can we create training data using a teacher from any augmented queries (i.c., cropped sentences or pscudo genera- tive queries)? Second, with the training data sets created from multiple teachers, how can we train a dense retriever to digest the multiple supervisions? Formally speaking, given N teachers, for each augmented query g, we retrieve N ranked lists (i.e.. P,] Di...... .... ,Py with each list has K passages) from the corpus with the respective teachers. We consider the ranked list Pr from the n-th teacher a source of supervision since the top-k and last-k' passages in Ph contain the teacher's view on what is relevant and less relevant for the given query. We then discuss possible strategies to train dense retriever with diverse supervisions.
Progressive Supervision. The previous two ap- proaches directly give models supervisions from multiple teachers at once; however, learning di- rectly from the mixture of supervision is challeng- ing, especially for DR models which compute text matching with simple dot product. Inspired by the success of curriculum learning (Zeng et al., 2022), we propose an approach to progressive la- bel augmentation to guide DR training with pro- gressively more challenging supervision. Specif- ically, we train our models with uniform super- vision for N iterations and at each iteration, we augment relevance label using additional teacher, as illustrated in Figure 2; i.e., at iteration T S N, we uniformly sample a source of supervision, P where n ~ (11 (see Appendix A.6 for more study and explanation). A key factor of this ap- proach is how to arrange the order for easy-to-hard supervisions; namely, the trajectory of progressive supervision.
Fused Supervision. An intuitive strategy is to fuse the multiple sources into a single high-quality supervision, which dense retrievers can learn from. For the augmented query 4, we conduct linear score fusion (Ma et al., 2021) on the N ranked lists to form a new ranked list Fg fused supervision. Uniform Supervision. Another simple strategy is to provide a dense retriever with equal expo- sures to multiple sources of supervisions. Specifi- cally, given query, we uniformly sample a source of supervision; i.e., a ranked list Pr, where n ~
With any aforementioned strategy to obtain di- verse supervisions, we train our dense retrievers using contrastive loss in Eq. (2). Specifically, given query q, we first obtain a source of supervision either from sampling (PT)) or fusion (F₂); then, we randomly sample positive and hard negative from the top 10 passages and top 46-50 passages, re- spectively to form a triplet. The sampling scheme

---

Table 2: Strategies to obtain multiple supervisions using cropped sentences queries.
Table 3: Study on trajectory of progressive supervision using cropped sentences queries.
3
5+
Progressive supervision
MARCO dev
BEIR-13
WWCOOL
Conrreevr
CaIBERIV
trujectories
RR@10
nDCG@
three teachers
(0) uni - Contriever - CoIBERTv2
36.6
49.3
Teacher
(1) Contriver - uniCOIL- ColBERTv2
36.7
48.4
(2) ColBCRTV2-+ Contrever uniCOIL.
36.4
47.7
fused unif. prog-
(3) uniCOIL y Contriever CoiBBBRTy2
36,8
47.4
effectiveness of student
CoIBERTv2 only teacher last (3rd) iteration.
MARCO Dev
34,9
33.9
36.4
36.7
36.9
36.6
BEIR-13
46.7
47.0
46.3
46.6
47.7
49.3
effectiveness of teacher
cating that learning from the diverse relevance la- bels from multiple retrievers separately rather than single strong supervision (ColBERTv2 fused pervision) is key to gain generalization capability. Finally, we observe that progressive supervision can further guide dense retriever to gain gener- alization capability over uniform supervision (col- umn 4 vs 5). Thus, we use progressive supervision in the following experiments.
MARCO Dev
35.1
34.1
39.7
40.0
BEIR-13
47.5
49.9
The condition of column 5 corresponds to row 0 in Table 3. We do evaluate uniCOII. BEIR due its require- ment of expensive document expansion from corpus.
has been empirically proved to well preserve the su- pervised signal from a single teacher (Chen et al.. 2022) (also see our study in Appendix A.5). In this work, we further extend the sampling scheme to obtain diverse supervisions from multiple teachers.
Trajectory of Progressive Supervision. We then study how to better arrange the trajectories of pro- gressive supervision in Table 3. We observe that different trajectories have much impact on mod- els' zero-shot retrieval effectiveness while minor impact supervised evaluation can be seen. For example, switching the sampling order between uniCOIL and Contriever results in degrade of point on the averaged nDCG@ 10 over BEIR-13 (row vs 1) while reversing the whole trajectory Icads to a degrade with more than 1.5 points (row 0 vs 3). This observation reflects an intuition that the retrievers with better generalization capability may capture more complex matching signal between text pairs (ColBERTv2 shows better generaliza- tion capability than the other two teachers); thus, their relevance labels should be augmented at a later stage of model training. Finally, in row 3, we follow the trajectory in row 0 but only use Col- BERTv2 as the only source of supervision instead of obtaining uniform supervision from the three teachers at the last iteration. This change results in worse zero-shot retrieval effectiveness, indicating that learning from diverse supervisions is key training generalizable dense retriever.
3.3 Empirical Studies
Strategies to Obtain Diverse Supervisions. We first conduct empirical studies on how to better train a dense retriever in a simplified setting by using the MS MARCO cropped sentences as aug- mented queries and obtain supervised labels using three teachers with diverse relevance score compu- tation: uniCOIL (sparse), Contriever (dense) and CoIBERTv2 (multi-vector). To compare the dif- ferent strategies discussed in Section 3.2, We re- port the models trained with single (columns 0-2) and multiple (columns 3-5) sources of supervi- sions for 20 epochs and 60 epochs, respectively. For progressive supervision, we follow the super- vision trajectory: uniCOIL-> Contriever - Col- BERTv2 with 20 epochs for each of the three itera- tions (N 3). Note that for fused supervision, use MS MARCO Dev queries to tune and obtain the best hyperparameters to create fusion list.
The results are tabulated in Table 2. We ob- serve that when learning from a single supervision (columns 0-2), there is a tradeoff between super- vised and zero-shot retrieval effectiveness. Learn- ing from the fusion list only sees slight improve- ment over supervised evaluation while no improve- ment observes in zero-shot evaluations (columns 0-2 vs 3). By contrast, the model sees notable im- provements in zero-shot evaluations when trained with uniform supervision (columns 0-3 vs 4), indi-
Query Augmentation. We study the impacts of query augmentation models' effectiveness under the scenario where supervision is given, which has not been studied so far in the field of dense retrieval. With the best trajectory progressive supervision, we compare models trained using cropped sen- tences (rectangles), generative queries (GenQ; cir-

---

will discuss the impacts of initialization in Sec- tion 5. The implementation of DRAGONS and the fully augmented training data are detailed in Ap- pendix A.3 and A.4, respectively.
N45 MASMARRCO De: B0M0310
10
B Copppel dracnss
Umppel sorrrcee
o-Ges0
GenQ
32
S-Cipped serteace Ger2
a Enpped sereece G
4 Comparison with the State of the Art
Kiltmaan
haron
30
Query
500E
500E-05
500E-06
R00E-07
4.1 Datasets and Baseline Models
(a) MS MARCO Dev
(b) BEIR-13
In addition to MS MARCO development queries, we evaluate model supervised effectiveness on the TREC DL (Craswell et al.. 2019, 2020) queries, created by the organizers of the 2019 (2020) Deep Learning Tracks at the Text REtrieval Conferences (TRECs), where 43 (53) queries with on average 95 (68) graded relevance labels per query (in con- trast to 6980 queries with on average non-graded relevance label per query in MS MARCO Dev) are released. We report nDCG@ @ 10, used by the organizers as the main metric. For zero-shot evalua- tions, report models' effectiveness all the 18 datasets in BEIR (Thakur et al., 2021b). In addition, we use LOTTE (Santhanam al., 2022b) consist- ing of questions and answers posted on StackEx- change with five topics including writing, recre- ation, science, technology, and lifestyle. We eval- uate models' retrieval effectiveness the pooled setting, where the passages and queries from the five topics are aggregated. Following Santhanam et al. (2022b), the retrieval effectiveness of Suc- cess @5 5 search and forum queries are reported. The detailed evaluation on LoTTE is listed in Ap- pendix A.8.
Figure 3: Impacts of query augmentation.
cles), their mixture (triangles) and human queries (cross) in Figure 3. We observe that query size is the key to successful training. Although training with limited (0.8M) cropped sentences cannot per- form well in MS MARCO dataset, scaling up the size (28M) sees significant improvement over the model trained on 0.8M human queries. Similarly, in Figure 3 (b), model's generalization capability shows huge jump when scaling up query size. While surprisingly, cropped sentences can help dense retrievers to gain more generalization capa- bility than human-like GenQ, a mixture of cropped sentences and GenQ yields strong retrieval effec- tiveness in supervised and zero-shot evaluations, when the query size is not large enough (0.8-8M).
3.4 Training our DRAGONS
With the empirical studies on DR training, we then propose the final recipe to train our DRAGON. We train DRAGON for 20 epochs (around 130K steps) at each iteration, with the trajectory of progres- sive supervision: uniCOIL- Contriever + GTR- XXL ColBERTv2 + SPLADE++. We list all the teacher model checkpoints for label augmen- tation in Appendix A.2. This trajectory is based on models' retrieval effectiveness on BEIR with the intuition gained from Section 3.3 that a more generalizable model creates relevance labels with more complex matching signals. For query aug- mentation, we mix half of cropped sentences and synthetic queries as training queries. Note that we do not further fine-tune our models on the MS MARCO training queries. In addition, we train other three DRAGON variants. DRAGON-S and DRAGON-Q only use cropped sentences and syn- thetic queries, respectively. As for DRAGON+, we follow the same training procedure of DRAGON but switch the initialization from BERT to the masked auto-encoding pre-trained model, RetroMAE. We
We compare DRAGONS with dense retrievers us- ing the backbone of ber-bbase-nncsed trained with advanced techniques, such as knowledge distilla- tion (Ren et al., 2021; Zeng et al., 2022), contrastive pre-training (Gao and Callan, 2022; Izacard et al., 2021; Yu et al., 2022), masked auto-encoding pre- training (Wu al., 2022: Xiao al., 2022) and domain adaptation (Wang et al., 2022; Dai et al., 2022). We refer readers to more detailed baseline model descriptions in Appendix A.I.
4.2 Results
Supervised Evaluations. The first main row in Ta- ble 4 reports models' retrieval effectiveness on MS MARCO passage ranking dataset. We first observe that some baseline dense retrievers which perform well in MS MARCO Dev set are either pre-trained on MS MARCO corpus (coCondenser and COT- MAE) or well fine-tume tune-t on MS MARCO training queries with cross-encoder distillation (CL-DRD

---

Table 4: Comparison with existing state-of ootte-of-th-arrr dense retrievers. Bold (underline) denotes the best (sccond best)
effectiveness for each among baseline dense models.
Rep type
sparse mul-vec dense
baseline dense
dense
SPL ADEW
CalBERTV2
GTR-XXL
CL-D CL-DRD
RackeQQA113
COT-MAE
RetroM AE
Ca CoComdenner
Contrieve
COCO-DR
DRAGON-S
DRASON-O
DRAGIN
DRAGON+
GPL
PTR
Pre-training
X
X
/
K
X
Distillation
/
/
X
X
X
X
/
X
/
/
Target Corpus'
X
X
X
X
K
X
X
X
X
/
K
X
X
X
MS MARCO (Supervised)
Dev @
38.9
39.7
38.8
38.1
38.8"
39.9"
35,4
38.6"
34.1
35.8
38.1
39.1
39.3
39.0
Dev (@)
98.2
98.4
99.0
97.9
98.1"
98.5"
97.5
98.4"
97.9
97.9
98.3
98.8
98.5
98.6
DL2019 (nDCG@ 10)
74.3
74.6
72.5
70.0"
68.8
71,5*
67.8
74.1
73.6
74.0
74.1
74.4
212020 (nDCG@ 10)
71.8
75.2
68.3
67.8"
71.4
68.1"
66.1
69.7
70.0
72.6
72.9
72.3
nDCG@ 10
BEIR (Zero-shot)
TREC-COVID
71.1
73.8
50.1
58.4
67.5
56.1
17.2
71.2
59.6
78.9
70.0
72.7
73.9
73.2
74.0
75.9
NFCorpus
34.5
33.8
34.2
31.5
29.3
32.1
30.8
32.5
32.8
35,5
34.5
33,4
32.2
33.0
32.9
33,9
FiQA-2018
35.1
35.6
46.7
30.8
30.2
28.3
31.6
27.6
32,9
31.7
34.4
40.4
35.6
35.3
35.0
35.6
ArguAna
52.1
46.3
54.0
41.3
45.1
27.8
43.3
29.9
44.6
49.3
55.7
53.8
51.5
45.5
48.9
46.9
Touche-2020
24.4
26.3
25.6
20.3
24.7
21.9
23.7
19.1
23.0
23.8
25.5
26.6
26.5
26.0
24.9
26.3
Quora
81.4
85.2
89.2
82.6
74.9
75.6
84.7
85.6
86.5
86.7
83.6
86.4
87.1
86.9
87.5
SCIDOCS
15.9
15.4
16.1
14.6
13.1
13.2
15.0
13.7
16.5
16.0
16.9
16.3
15.9
15.0
15.4
15.9
SciFact
69.9
69.3
66.2
62.1
56.8
60.1
65.3
61.5
67.7
70,9
67.4
62.3
67.8
67.2
67.5
67.9
NQ
54.4
56.2
56.8
50.0
50,5
48.3
51.8
48.7
49.5
50.5
48.3
53.3
52.3
53.1
53.7
HorpotQA
68.6
66.7
59.9
58.9
53.3
53.6
63.5
56.3
63.8
61.6
58.2
60.4
65.6
62.7
64.8
66.2
DBPedia
44.2
44.6
40.8
38.1
35.6
35.7
39.0
36.3
41.3
39.1
38.4
36.4
40.6
41.4
41.4
41.7
FEVER
79.6
78.5
74.0
73.4
67.6
50.6
77.4
49.5
75.8
75.1
75.9
76.2
76.4
75.1
75.8
78.1
Climate-FEVER
22.8
17.6
26.7
20.4
18.0
14.0
23.2
14.4
23.7
21.1
23.5
21.4
21.8
20.3
22.2
22.7
CQADupStack
34,1
39.9
32.5
29.7
34.7
32.0
34.5
37.0
35.7
35.9
34.4
35.2
35.4
Robust04
45.8
50.6
37.7
30.8
44.7
35.4
47.6
44.3
43.7
46.3
45.3
47.2
47.9
Signal-IM
29.6
27.3
28.2
26.5
28.1
19.9
27.1
27.6
29.3
28.9
30.1
30.1
TREC-NEWS
39.4
34.6
38.0
26.1
42.8
33.7
42.8
40.3
42.1
42.1
40.0
41.6
44.4
BioASQ
50.4
32.4
37.4
26.2
42.1
25.7
38.3
42.9
44.2
41.3
41.9
41.7
43.3
Averaged IDCG@10
PIR subsets
47.1
46.2
44.9
40.9
40.1
35.7
44.5
37.4
43.8
45.7
45.5
45.5
46.2
45.0
45.7
46.5
BEIR-13
50.3
49.9
49.3
44.8
43.6
39.8
48.2
42.0
47.5
49.2
48.6
49.8
48.8
49.4
50.2
Avg. all 18 datasets
47.4
45.8
42.0
36.2
45.4
38.9
44.5
46.2
45.9
46.8
45.8
46.6
47.4
Success G
LOTTE (Zero-shot)
Search (pooled)
70.9
71.6
65.8
69.8
63.4
66.8
62.5
66.1
67.5
71.4
72.4
72.6
73.5
Forum (pooled)
62.3
63.4
55.0
57.7
$1.9
58.5
52.1
58.9
56.8
61.1
61.2
61.4
62.1
The appreach assumes the target corpus (e.g., BEIR) is available while training.
These numbers not comparable duc to the use of pon-standard MS MARCO corpus augmented with title (Lassance and Clinchant, 2023).
and RocketQAv2). However, their retrieval effec- tiveness on MS MARCO Dev set is not well corre- lated to TREC DL queries, which have fine- fine-graing grained human labels with different degrees of relevance. We hypothesize that these models able to re- trieve the most relevant passage from the corpus but cannot retrieve diverse passages with different degrees of relevance. By contrast, all the variants DRAGON trained with diverse augmented rele- vance labels show consistently strong effectiveness in MS MARCO Dev and TREC DL queries.
better zero-shot retrieval effectiveness (e.g., Con- triever, COCO-DR and RetroMAE). These mod- els are pre-trained (with data augmentation) on corpus other than MS MARCO to combat do- main shift issue in dense retrieval (Xin et al., 2022; Yu al., 2022). On the other hand, DRAGONS trained on augmented data from MS MARCO corpus only transfer well to BEIR datasets. Fur- thermore, DRAGON+ reaches state-of-the tftth-art re- trieval effectiveness on BEIR as the sparse retriever, SPLADE++.' In addition, all the DRAGON variants outperform other dense retrievers by a large mar- gin and compete SPLADE++ and ColBERTv2 in the LoTTE dataset. It is worth mentioning that the models trained with domain adaptation (columns
Zero-Shot Evaluations. The second main row in Table 4 reports models' zero-shot retrieval ef- fectiveness on the BEIR datasets. We observe a reverse trend that those dense retrievers performing relatively poorly in MS MARCO Dev queries have
*Scc our BEIR leaderboard submission here.

---

Table 5:
Label augmentation with cross encoder (CE) using cropped sentences queries.
Table 6: Ablation on initialized checkpoint using the
mixture of cropped sentences and GenQ queries.
0'
2
3
MARCO dev
BEIR-13
Initialized checkpoint
DRAGON-S initialization
V
RR@
nDCG@ 10
Source supervision
all't
only
only
MARCO Dev (RR 10)
38.1
38.1
37.5
36.8
(0) BERT base (DRAGON)
39.3
49.4
BEIR-13 (nDCG@ 10)
49.8
49.7
48.7
47.7
Contriever
38.7
49.3
2) RoBERTa base
39,4
49.9
Columa corresponds DRAGON-S.
RetroMAE (DRAGON+)
39.0
50.2
"all" denotes the five teachers used for training DRAGON-S.
5
Discussions
Is it necessary to augment relevance labels with a cross encoder? To answer this question, we further train DRAGON-S with the augmented rel- evance labels from a cross encoder (CE). Specif- ically, we create ranked list of CE by first trieving top 1000 passages with DRAGON-S and re-ranking them with the CE for each cropped sen- tence as a query. With the CE ranked list, we con- duct another iteration (20 epochs) of training for DRAGON-S however, we do not see retrieval ef- fectiveness improvement in Table (column O vs 1). In addition, the retrieval effectiveness becomes even worse when we further train DRAGON-S by only sampling CE ranked list instead of uniformly sampling all the six ranked lists (column 1 2). Finally, we initialize from pert-base-uncased and re-train the model for three iterations (60 epochs) only with the CE ranked list. We observe that its effectiveness (column 3) is even worse than the models trained with the ranked lists from three retrievers (see columns 4 and 5 in Table 2). This re- sult contradicts the general belief that CE provides the strongest supervision
9-B) perform better than the others (columns 3-8) but still underperform DRAGON in zero-shot eval- uations. Using DRAGON the base model for domain adaptation is possible gain DR zero-shot effectiveness, which we leave our future work.
A comparison between DRAGON-S and DRAGON-Q (columns C and D) shows that augmented query type has an impact on retrieval effectiveness in different datasets. DRAGON-S trained cropped sentences surprisingly sees the highest retrieval effectiveness on BEIR but only sacrifices bit on MS MARCO datasets. This means that cropped sentences, the cheap query type (compared to neural generative queries), are sufficiently helpful for models to learn domain-invariant retrieval capability. By contrast, we observe that DRAGON-Q trained with human-like queries performs poorly compared DRAGON-S the datasets where queries are far different from human-like queries, such as ArguAna (45.5 vs 51.5) and CQADupStuck (34.4 vs 35.9), while mixing different types of queries (DRAGON) can mitigate the issue. Finally, DRAGON+, combined with masking auto-encoding (MAE) pre-training and our approach, sees further improvement on zero-shot evaluations without sacrificing in-domain ones, indicating that MAE pre-training may be orthogonal to our approach based on contrastive learning.
dense retriever (Hof- stätter et al.. 2020). Moreover, it demonstrates the effectiveness of using diverse supervisions to train a generalizable dense retriever, rather than relying on a single strong supervision. Furthermore, leverag- ing all the retrievers to augment labels still more efficient than cross encoder (see Appendix A.7).
Does DRAGON benefit from unsupervised pre- training? Table 6 compares the models trained from different checkpoint initialization. Note that for Contriever and RetroMAE, we initialize from the checkpoint with only unsupervised pre-training (without fine-tuning on MS MARCO). We observe that our approach benefits from further masked auto-encoding rather than contrastive pre-training (rows 2,3 vs 1). The result is sensible since our
To sum up, DRAGONS advance state-of-the-art zero-shot effectiveness while keeping strong effec- tiveness in supervised evaluation. The experimen- tal results demonstrate that our data augmentation approaches enable dense retrievers to learn domain- invariant matching signal between text pairs as the other models with fine-grained late interaction (SPLADE++ and Co1BERTv2) or 40 times larger model size (GTR-XXL).
We do not notice effective improvement with more iterations of training both in supervised and zero-shot evaluations.

---

Passage:
The Manhaitan Project and lis stomic bomb helped bring an end to World War II. Its legacy of peaceful uses of atomic energy continues to have an impast on history and science.
Query Augmentation
Augmented rel (4)
Example of augmented rel
Manhattan Project and
Manhattan project wias secret research and development project U.S to develop the etomic bomb. Its success granted the U.S the bombs that ended the war with Japan well as ushering the country into the atomic era.
Q1
atomic bomb helped bring an end to World War II
30
Cropping
Its legacy of peaceful uses of atomic energy continues have en impact on history science.
An earty nucleer power piant that used atomic energy to generate electricity. The Atomic Age, also known as the Atomic Era, is the period history tollowing the delonation
Q2
30
first nuclear (alomic) bomb.
what major contributions manhattan effort
The Manhattan Projact an affort during World War in United States develop first nuclear weapon.
GenQ
Manhattan Project, which included some history's greatest scientific minds, lead to the end of the war against the Japanese. But was worth the environmental and financial costs? This massive site provides loads of
what impact did the manhattan project have on history
Q2
26
Figure 4: Examples of augmented queries and relevance labels from a passage. Augmented rel (4) denotes the number of unique relevant passages labeled by all our five teachers.
approach can be considered an improved version of contrastive pre-training, which appears to be orthogonal to masked auto-encoding pre-training. We leave the investigation of improving DR with generative and contrastive pre-training combined for future research.
generative human-like query. Thus, teachers show more disagreement between each other on cropped sentences. We hypothesize that dense retriever trained cropped sentences can capture more di- verse supervised signals from multiple teachers than generative queries. This explains the reason why DRAGON-S shows better generalization capa- bility than DRAGON-Q.
Can we use the soft labels from multiple teach- ers? In the literature, using the relevance scores from a teacher as soft labels is a standard of knowl- edge distillation (Lin et al., 2021b; Hofstätter et al., 2021). However, in our study, even when train- ing with uniform supervision from a sparse and dense retriever (i.e., uniCOIL and Contriever), it is challenging to normalize their relevance scores and create universal soft labels, yielding significant supervised and zero-shot effectiveness drops. We suspect that dense and sparse retrievers have many different views on relevance score computation; thus, it is even harder for a dense retriever to learn the score distributions from the different teachers.
6 Conclusion
We present DRAGON, Dense Retriever trained with diverse AuGmentatiON and unified frame- work of data augmentation (DA) understand the recent progress of training dense retrievers. Based on the framework, we extensively study how to improve dense retrieval training through query and relevance label augmentation. Our experiments un- cover some insights into training dense retriever, which contradicts common wisdom that cross en- coder is the most effective teacher and human-lite queries are the most suitable training data for dense retrieval. Then, we propose a diverse data augmen- tation recipe, query augmentation with the mixture of sentence cropping and generative queries, and progressive relevance label augmentation with mul- tiple teachers.
Why sentence cropping yields a generalizable dense retriever? Figure 4 showcases the aug- mented queries by sentence cropping and neural generation and their respectively augmented rele- vant passages other than the original passages. We observe two main differences between the cropped sentences and generative queries. Cropped sen- tences provide diverse queries from passage; i.e., the two cropped sentences in Figure 4 include slightly different topics (Manhattan Project and atomic energy). By contrast, all generative queries surround the same main topic, Manhattan Project, about the original passages. Second, the cropped sentences have more unique augmented relevant passages than generative queries. This is maybe because a cropped sentence, containing more in- formation (keywords), is more challenging than a
With our proposed recipe of DA, DRAGON is the first to demonstrate that a single BERT-base- sized dense retriever can achieve state-of-the -ftt--of-the-rt effectiveness in both supervised and zero-shot re- trieval tasks. We believe that DRAGON as a strong foundation retrieval model for domain adaptation retrieval tasks (Wang et al., 2022: Dai et al., 2022) or the existing retrieval augmented language models (Izacard et al., 2022; Shi et al., 2023; Mallen et al., 2023).
serve

---

Limitations
Zhuyun Dai, Vincent Y. Zhao, Ji Ma. Yi Luan, Jianmo Ni. Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-We Wei Chang. 2022. Promplaga- tor: Few-shot dense retrieval from examples. arXiv:2209.11755.
Despite of the easy usage of single-vector dense retrieval compared to the models with more fine- grained late interactions (c.g., SPLADE++ and Col- BERTv2), the limitations of DRAGONS mainly from the cost of training. First, conduct di- verse relevance label augmentation, well trained dense, sparse and multi-vector retrievers are re- quired. Second, to optimize DRAGONS' effective- ness, we scale up training queries to the size of 28 millions (compared to 0.8 millions in MS MARCO training queries) and leverage the progressive train- ing strategy, which costs five days of training time with 32 A100 (40 GB) GPUs. The training cost can be reduced by removing repetitive or meaningless queries, which we leave for future work.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language under- standing. arXiv:
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2022. From dis- tillation to hard negative sampling: Making sparse neural IR models more effective. In Proc. SIGIR, page 2353-2359
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant, 2021. SPLADE v2: Sparse lexical and expansion model for informa- tion retrieval. arXiv:2109. 10086.
Luyu Gao and Jamie Callan. 2021. Condenser: a pre- training architecture for dense retrieval. In Proc. EMNLP, pages 981-993.
References
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Scbastian Riedel, Hannanch Ha- jishirzi, and Wen-tau Yih. 2023. Task-aware retrieval with instructions. Proc. Findings of ACI, pages 3650-3675.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. In Proc. ACL, pages 2843-2853.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chem, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In Proc. ICML.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset, arXiv: 1002668.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in neural network. In Proc. NIPS.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009, Curriculum learning. In Proc. ICML, page 41-48.
Sebastian Hofstätter, Sheng-Chich Lin, Jheng-Hong Yang. Jimmy Lin, and Allan Hanbury. 2021. Ef- ficiently teaching an effective dense retriever with balanced topic aware sampling. Proc. SIGIR, page 113-122.
Wei-Cheng Chang. Felix X. Yu, Yin- Wen Chang, Yim- ing Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-basee large-scale retrieval. In Proc. ICLR.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking mod- els with cross-architecture knowledge distillation. arXiv:2010.02666.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for con- trastive learning of visual representations. In Proc. JCMI..
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv:2112.09118.
Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev. Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2022. Salient phrase aware dense retrieval: Can dense retriever imitate sparse one? In Proc. Findings of EMNLP, pages 250-262.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot Learning with Retrieval Aug- mented Language Models. arXiv:2208.03299
Nick Craswell, Bhaskar Mitra, and Daniel Campos. 2019. Overview of the TREC 2019 deep learning track. In Proc. TREC.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020. Overview the TREC 2020 deep learning track. In Proc. TREC.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with GPUs. IEEE Trunsactions on Big Data pages 535-547.

---

Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proc. EMNLP, pages 6769-6781.
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin. Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mchdad. 2022. Domain-matched pre-training tasks for dense retrieval. In Proc. NAACI, pages 1524-1534.
Carlos I Lassance and Stephane Clinchant, 2023. The tale of two MSMARCO- and their unfair comparisons. In Proc. SIGIR, page 243 1-245..
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized train- ing approach to dense passage retrieval for open- domain question answering. In Proc. NAACL, pages 5835-5847.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proc. ACL, pages 6086-6096.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-61
Jimmy Lin, Xueguang Ma, Sheng-Chie Chenn Lin, Jheng- Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. 2021a. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations, In Proc. SIGIR, page 2356-2362.
Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021b. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proc. RepLANLP, pages 163-173.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks, In Proc. EMNLP, pages 3982-3992.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao. QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proc. EMNLP, pages 2825-2835.
Zhenghao Lin, Yeyun Gong, Xiao Liu, Hang Zhang, Chen Lin, Anlei Dong, Jian Jiao, Jingwen Lu, Daxin Jiang, Rangan Majumder, and Nan Duan. 2022. Prod: Progressive distillation for dense retrieval. arXiv:2209. 230.33335.3
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022a. Plaid: An efficient engine for late interaction retrieval. arXiv:2205.0977 097007
Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021. Less is more: Pretrain strong Siamese encoder for dense text retrieval using a weak decoder. In Proc. EMNLP, pages 2780-2791.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022b. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proc. NAACL, pages 3715-3734.
Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021. A replication study of dense passage retriever. arXiv:2104.05740.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmennt black-box language models. arXiv:2301.12652.
Joel Mackenzie, Andrew Trotman, and Jimmy Lin. 2021. Wacky weights in learned sparse represen- tations and the revenge of score-at-an a-tma- query eval- uation. arXiv:21 15500
Nandan Thakur, Nils Reimers, Johannes Daxen- berger, and Iryna Gurevych. 2021a. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. Proc. NAACL, pages 296-310.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. Proc. ACL, pages 1802-9822..
Rui Meng, Ye Liu, Semih Yavuz, Divyansh Agarwal, Lifu Tu, Ning Yu, Jianguo Zhang, Meghama Bhat, and Yingbo Zhou. 2023. AugTriever: Unsuper- vised dense retrieval by scalable data augmentation. arXiv:2212.088.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021b. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proc. NIPS
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022, GPL.: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proc. NAACL, pages 2345 2360.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hemnandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Meng-Wei Chang, and Yinfei Yang. 2022, Large dual encoders are generalizable retrievers. In Proc. EMNLP, pages 9844 9855.
Xing WU, Guangyuan Ma. Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2022. Contex- tual masked auto-encoder for dense passage retrieval. arXiv:2208.07660.
Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to doeTTTTTquery.

---

Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. RetroMAE: Pre-training retrieval-oriented lan- guage models via masked auto-encoder. In Proc. EMNLP, pages 538-548.
Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. 2022. Zero- shot dense retrieval with momentum adversarial do- main invariant representations. In Proc. ACL, pages 4008 4020.
Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Amool Overwijk. 2022. COCO-DR: Combating dis- tribution shift in zero-shot dense retrieval with con- trastive and distributionally robust learning. In Proc. EMNLP, pages 1462-1479.
Hansi Zeng, Hamed Zamani, and Vishwa Vinay. 2022. Curriculum learning for dense retrieval distillation. In Proc. SIGIR, page 1979-1983.

---

A Appendices
github. com wwwmm/tatoordddiiiiiicoooooooooo futurrraaaaaat
1BERT; (5) SPLADE++; http://cowwloa.
A.1 Baseline Models
-de tellaaa_lan2//ggaaadedddsi.....,
We compare DRAGON with dense retrievers us- ing the backbone of bert-base-uncased trained with advanced techniques. (1) Knowledge Distillation: RocketQAv2 (Ren et al., 2021) distills knowledge from cross encoder while CL-DRD (Ren al., 2021) combines curriculum learning and cross- encoder distillation. They all use cross encoders' knowledge to augment positive relevance labels as our approach. (2) Contrastive Pre-training: coCon- denser (Gao and Callan, 2022), Contriever (Izacard et al., 2021) and COCO-DR (Yu et al., 2022) are first pre-trained on different corpus listed in Ta- ble 1, and then fine-tuned MS MARCO training queries. (3) Masked Auto-Encoding Pre-Training: COT-MAE (Wu et al., 2022) and RetroMAE (Xiao et al., 2022) are first pre-trained to recover polluted sentences and then fine-tuned on MS MARCO training queries. For RetrOMA,, we use the vari- ant with the best BEIR retrieval effectiveness for comparison. (4) Domain adaptation: We consider GPL (Wang et al., 2022) and Promptagator (PTR; Dai et al., 2022), which generative models to create pscudo relevance data for each corpus in BEIR and train one expert dense retriever for each corpus. This approach requires the target corpus while training. Note that COCO-DR can also be considered domain adaptation on BEIR although it uses one model for all tasks.
(
httess//aaggiiogfacc.co//ooosss/
odder/nsssmaarrooMMMM11111--1--1
Baseline Models: (1) CL-DRD: https:
thub. www.ccom//aaaeee Zeng Zeng/CI-DRD, etQAv2: we directly copy the numbers from San- thanam et al. (2022b); (3) COT-MAE: https:
CL-DRD; (2) Rock-
/ //wgggnggaace.e.ecooooa co/ wttpa//ccccccccccocccoeooooooooo __aae__osaacc_rreriieee. (4) Retro-
MAE: MAl.httssrr/augggngfccc.c///
( https:
nsse-maarco-reetrlever (6) Contriever:
https://aagg2ngffccccc//eeee
k/contrievee--nmarco (7) COCODR: COCODD
https:///ugginnffeeccc//////p//p
/cocodr mamarco; (8) Promptagator (PTR) and GPL: we we directly copy the numbers from their original papers (Dai et al., 2022; Wang et al., 2022).
A.3 Implementation Details
We train our dense retrievers initialized from bert- base-uncased 32 A100 GPUs (40GB) with a Per-GPU batch size of 64 and a learning rate of 3e 5. Each batch includes an augmented query with its positives and hard negatives. Following Karpukhin et al. (2020), we use asymmetric dual encoder with two distinctly parameterized encoders and leverage in-batch negative mining. Note that symmetric dual encoder shows poor generalization capability in our initial experiments. We set the maximum query and passage lengths 32 and 128 for MS MARCO training and evaluation. For BEIR evaluation, we set maximum input lengths to 512.
Note that coCondenser and COT-MAE are fine- tuned on the "non-standard" MS MARCO passage corpus that has been augmented with title. Thus, we also conduct inference on the corpus with ti- tle for them; otherwise, we use the official MS MARCO passage corpus. In addition, we also re- port the retrieval effectiveness of GTR-XXL. (Ni et al., 2022) and C (Santhanam et al., 2022b) from their original papers and conduct re- trieval for SPLADE+ using Pyserini (Lin al., 2021a) for reference. We list all the other model checkpoints used for evaluations in Appendix A.2,
Table 7: MS MARCO and our augmented training queries statistics.
number Avg. # takens Avg. # rel
A.2 Model Checkpoints
passages in corpus training queries
8,841,823
78.8
532,761
8.2
Teacher Models: (1) uniCOIL: https: huu
augmented training queries
cropped sentences generative queries
28.545,938
24.4
23.1
smarco-passage: (2) Contriever: https:
28,545,938
8.0
24.7
//muggingfacc.cc//accccooo/////
test queries
Dev
6,980
7.8
DL2019
43
7.6
95,4
DL2020
54
7.5
68.0
s/ s/gtr-t5-xxi; (4) CoIBERTv2: https://

---

A.4 MS MARCO Dataset Statisties
Table 9: Uniform and progressive supervision effective- ness comparison each training iteration. The models are trained using cropped sentences as queries.
Table lists the data statistics of MS MARCO dataset, including the original training queries and test queries (i.c., Dev, DL19 and DL20). In addi- tion, we also list the augmented queries used to train DRAGONS with full relevance label augmen- tation by five teachers.
uniform
progressive
Teacher / /iteration
2
2
1
2
uniCOIL
/
V
V
Contriever
X
ColBERTv2
X
v
A.5 Impacts of Top-k Positive Sampling
MARCO Dev
36.2
37.0
36.9
34,9
35,8
36,6
BEIR-13
46.6
47.4
47.6
46.7
48.6
49.3
effectiveness of teacher
Table 8: Ablation on progressive label augmentation from top-k passages using cropped sentences as queries.
MARCO Dev
39.1
39.1
39.1
35,1
36.5
39.1
diversity of teacher
Avg. rel
17.5
17.5
17.5 10.0
14.9
17.5
top-l positives
5
MARCO Dev (RR @ 10)
33.1
36.4
36.6
BEIR-13 13 (nDCG@10) 10)
42.4
48.0
49.3
each teacher labels the top 10 (k 10) retrieved passages positives in our labeling scheme. Note that, in the case where multiple positives have equal probability, we further rank them according to their sum of reciprocal rank, For instance, if the two pas- sages (c.g., P1 and p2) are retrieved by all the three teachers; then, we further rank them according to their scores 1 + 1 + FL and +
Trajectory: uniCOIt- Contriever ColBERTv2.
In Section 3.2, we mention that our sampling scheme treats top 10 passages from each teacher ranked list as positives and top 45-50 as negatives. We further conduct experiment to study the im- pact of the positive sampling scheme. Following the experiment setups in Section 3.3, we use the sentences cropped from MS MARCO corpus as augmented queries and we conduct progressive la- bel augmentation using top-k passages as positive. The results are tabulated in Table 8. We observe that treating top-10 passages from each teacher as positives yields the best supervised and zero-shor shot effectiveness. On the other hand, using only the top passage as positive results in significant effective- ness drop. This result indicates that the top passage labeled by teacher cannot transfer its knowledge well to student. This result is similar to the obser- vation from Chen et al. (2022).
ri ria where I'm denotes the rank of the passage pm by the n-th teacher. In addition, we also estimate the diversity of supervision by computing the number of positive passages in union sets from the sources of supervisions.
r22 r2g
Table reports the detailed effectiveness and supervision quality (accuracy and diversity) com- parison at each training iteration between uniform and progressive supervision discussed in our pilot study. We observe that uniform supervision provides accurate and diverse supervision in the beginning of training; however, the generalization improvement over iteration is less than progressive supervision.
A.6 An Intuition Behind Uniform and Progressive Supervisions
A.7 Latency Measurement for Relevance Label Augmentation
As shown in Section 3.2, uniform supervision pro-
vides good supervision without fusion weight tun- ing as fused supervision. Intuitively, positive re- trieved by more teachers has higher probability to be sampled and may be more relevant to a query. To provide a sense of why uniform supervision works, we estimate the accuracy of supervision by comput- ing the probability of each positive sampled under uniform supervision, and rank the positives accord- ing to the simulated probability. For example, the 3rd iteration of progressive training, given a query, a positive passage is labeled positive by all the three teachers, the probability of the passages being sam- pled is 1 5 + BE + 1. In our experiments, 3
We measure the latency of label augmentation us- ing batch retrieval on a single NVIDIA A100 40GB GPU for GPU search and 60 Intel(R) Xeon(R) Plat- inum 8275CL CPUs @ 3.00GHz for CPU search. For cross encoder, we conduct label augmentation by re-ranking text pairs with batch size of 100. For dense retrieval (Contriever and GTR-XXL) and sparse retrieval, we use Faiss-GPU index and Lucene index from Pyserini (Lin et al., 2021a) with 60 threads, respectively, and search with a batch size of 100. Note that we use a batch size of 25 to encode queries using GTR-XXL due to GPU mem-
E-

---

Table 10: The latency comparison relevance label augmentation with batch inference using different teach- ers on MS MARCO.
A.9 More Related Work
Knowledge Distillation. Our work is closely re- lated the previous work exploring knowledge distillation (KD; Hinton et al., 2015) from Col- BERT, cross encoder or their ensemble (Hofstätter et al., 2021; Hofstätter et al.. 2020) to improve the effectiveness of DR (Lin et al., 202 lb; Qu et al., 2021). However, they only take the advantage of soft labels from KD and use the relevant passages labeled by humans. The recent work (Ren et al., 2021; Zeng et al., 2022) mines more positive sam- ples using cross encoder to further augment the limited relevance labels by humans. Nevertheless, is challenging for cross encoders to augment rel- evance labels for queries in scale due to its low efficiency. Chen et al. (2022) first explore label augmentation using singe sparse retrieval model on large-scale queries and demonstrate that dense retriever can mimic teacher of a sparse retriever (e.g.. BM25). Different from the previous work, we explore label augmentation using multiple su- pervisions on large-scale augmented queries.
candidates
latency (ms/q)
Type
Model
(#)
GPU
CPU
cross-encoder
Mi6iL
1K
600
dense
Contriever
8.8M
<1
dense
GTR-XXL
8.8M
10
spurse
uniCOIL
8.8M
84
spurse
SPLADE++
8.8M
144
multi-vec
CoIBERTv2
8.8M
Table 11: Detailed effectiveness (Success5) LoTTE
sparse multi-voe
dense
SPLA SPLLADE LADDA
CoIBERTV2
DRAGON
DRAGON-
DRAGON
DRAGOW!
writing
78.7
80.1
78.8
78.2
79.2
81.5
Search
recreating
71.9
72.3
73.4
74.6
76.0
73.9
science
56.6
56.7
55,3
56.9
56.9
$7.9
technology
65.9
66.1
64.9
68.8
65.4
67.6
lifestyle
83.7
84.7
84.9
84.7
85.6
85.9
pooled
70.9
71.6
71.4
72.4
72.6
73.5
writing
75.2
76.3
76.2
75.2
75.6
77.5
Curriculum Learning. Easy-to-hard training
Porum
recreating
69.2
70.8
69.9
69.3
70.3
69.1
strategies (Bengio et al., 2009) have been applied to improve many machine learning tasks, includ-
science
44.9
46.1
40.1
40.1
40.7
41.4
technology
53.1
53.6
50.5
51.2
50.5
51.4
lifestyle
76.9
76.9
77.0
77.4
76,9
77.7
ing dense retrieval (Zeng et al., 2022; Lin et al., 2022). The previous work focuses distilling complex knowledge from cross encoders dense retriever with a curriculum training strategy and demonstrates improved effectiveness in supervised retrieval tasks. In our work, we explore to pro- gressively train a dense retriever with the diverse supervisions from dense, sparse and multi-vector retrievers to improve both supervised and zero-shot effectiveness.
pooled 62.3
63.4
61.1
61.2
61.4
62.1
ory constraint, which is also the main bottleneck for GTR-XXL batch retrieval. For ColBERTv2, v2, use the improved version of multi vector retrieval, PLAID (Santhanam al., 2022a), and search with a batch size of 1, which is the only option.
Table 10 compares the latency cost per query for relevance label augmentation with different neural rankers and demonstrates that leveraging all the retrievers to augment relevance labels are still more efficient than a cross encoder.
Pre- Pre-Training, There are two popular approaches to pre-training a dense retriever. The first one is contrastive pre-training, aiming increase the size of training data by creating artificial text pairs (Lee et al., 2019; Chang al., 2020; Izacard al., 2021) from a corpus or collecting question-answer pairs (Oguz et al., 2022: Ni et al., 2022) from web- sites. The second one is masked auto encoding pre-training, where models are trained to recover the corrupted texts (Gao and Callan, 2021; Lu et al., 2021; Xiao et al., 2022; Wu et al., 2022). Our work is similar to contrastive pre-training but instead of creating large-scale training data in an unsuper- vised or weakly supervised manner, we investigate how to conduct supervised contrastive learning on artificially created text pairs. We demonstrate that
A.8 Detailed evaluation on LoTTE
Table 11 lists DRAGON's effectiveness on five top- ics without aggregation. Although all the variants of DRAGON show strong effectiveness LoTTE, we find that DRAGONS perform poorly on the Fo- rum queries about topics of science and technology compared SPLADE++ and ColBERTv2. Com- bining science corpus pre-training with DRAGON training strategy a possible solution.

---

combining masked auto encoding pre-training and our supervised contrastive learning can further im- prove models' generalization capability.


Reviews:
[{"review_id": "kHays1T2jI", "reviewer": "Reviewer_VbrU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s insightful discussion. \n\n1. Statistical significance for improvement is not reported: \nWhile we agree that conducting statistical significance tests can provide a more solid conclusion, we believe that our experiments sufficiently demonstrates DRAGON’s clear win over other baseline models. For example, in Table 11 (in Appendix), we report the detailed numbers of 18 BEIR datasets and DRAGON+ shows the best or second best result over 12 BEIR datasets. More importantly, DRAGON+ is the only DR model reaching state-of-the-art performance in both supervised (MS MARCO0) and zero-shot (BEIR, LoTTE) evaluations. Finally, as we mentioned in our paper, some compared models using ``non-standard’’ MS MARCO corpus augmented with titles or assuming BEIR corpus are available while training or training one expert model for each dataset. Those approaches have unfair advantages in either MS MARCO or BEIR dataset. Thus, it may not be easy to claim a significant win over those models. \n\n2. Comparison to other larger models: \nWe agree with this point, and we’ve reported GTR-XXL’s (4.8B) performance in the main comparison. Other larger bi-encoder models worth mentioning are CPT_XL (175B), E5_large (335M) and COCO-DR large (335M). As they adopt different subsets of BEIR, we provide the averaged numbers for their common BEIR datasets (see CPT Sub in [1]) as reference. The corresponding average numbers for CPT_XL, E5_large and COCO-DR large are 52.7, 53.0 and 54.1 compared to 53.0 for DRAGON+. However, as far as we know, most of the larger generalizable dual encoders cannot be fairly compared with DRAGON except for GTR. For example, COCO-DR large [1], as we highlighted, is pre-trained on all the corpus in BEIR datasets. In addition, E5 large [2] is fine-tuned on both MS MARCO and NQ (NQ is included in BEIR evaluation and multiple datasets in BEIR use wikipedia as corpus). We will explicitly mention this in the paper if accepted. \n\n\n[1] Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. Coco-dr: Combating distribution shifts in zero-shot dense retrieval with contrastive and distributionally robust learning. In Proc. EMNLP.\n[2] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022b. Text embeddings by weakly-supervised contrastive pre-training. ArXiv, abs/2212.03533.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and well-structured. It investigates the important problem of the capacity of BERT-size models for dense retrieval in the presence of multiple approaches for data augmentation. The paper explores various approaches for training using multiple sources of label relevancy. The experimentation shows the effect of each query augmentation approach with each of the training supervision approaches.", "reasons_to_reject": "Statistical significance is not reported for improvement which makes it challenging to interpret the results. \nThe authors mentioned they compare Dragons with the dense retrieval models with the backbone of bert-base-uncased trained with advanced techniques (see line 474), however, they claim that Dragons achieve state-of-the-art results without increasing the model capacity. This statement prompts a comparison of Dragons to other methods that use larger models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a Data Augmentation framework for Dense Retrieval models (DRAGON), allowing BERT-size dense retrieval models to achieve state-of-the-art results. Dragon consists of both query augmentation (sentence cropping and psuedo-query generation) and relevance label augmentation(multiple retrievers for different relevancies). Authors follow standard settings for training ad evaluation of their model using MSMarco training corpus as training and DA source and MSMarco Dev set for supervised evaluation and BEIR benchmark for zero-shot evaluation. The paper explores three different training supervision (Fused, Uniform, Progressive) to utilize different rank lists obtained from diverse relevancy labels.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6HSAfZkNlW", "reviewer": "Reviewer_CLak", "summary": "", "questions": "Is there a possibility that DRGAON experiences overfitting or even model collapse due to augmented data? Could you delve deeper into this aspect?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s insightful discussion. See below responses to the concerns and questions.\n\n1. Potential  negative impact of augmented data / Possibility of DRAGON overfitting due to augmented data:\nIt is indeed the central focus of our paper to train a more generalizable retriever and avoid overfitting. Through a systematic study of various data augmentation approaches, we believe that our “diverse” augmentation strategy yields a more generalizable retriever compared to existing works. This is evidenced by the extensive zero-shot evaluations we conducted on benchmarks like BEIR and LoTTE. This experimental setting makes sure our model does not see the evaluation data (even corpus) during training. The fact that DRAGON achieves a state-of-the-art performance on these datasets shows that the diversity of our query and relevance label augmentation ensures the effectiveness of DRAGON across a wide range of domains and datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation is well-founded. The explored content in the article demonstrates an interesting phenomenon: that we can further enhance performance while keeping or even reducing the model's size. This may inspire in the era of large models.", "reasons_to_reject": "The discussion about the negative impact of augmented data in the article is not sufficient. The presence of a significant amount of augmented data might impose limitations on the model's performance.", "questions_to_authors": "Is there a possibility that DRGAON experiences overfitting or even model collapse due to augmented data? Could you delve deeper into this aspect?", "paper_topic_and_main_contributions": "The paper presents a data augmentation-based pipeline, which allows the model to achieve Retrieval State-of-the-Art (SOTA) performance while maintaining the parameter count of BERT-base.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fyuC0MLheP", "reviewer": "Reviewer_1UUD", "summary": "", "questions": "1. In section 5, for cross encoder experiment, what is the performance of CE model in both supervised and zero-shot setting? It seems that CE is added in the last iteration, does this follow the rule of ordering different supervision mentioned in section 3.3 (i.e. augment a supervision at a later stage if it has better generalization capability?)  Why does CE model not provide additional benefit?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s insightful suggestions and discussion. See below responses to the concerns and questions.\n\n1. The proposed approach increases the complexity from a different perspective: \nWe agree that conducting auto labeling with multiple retrievers may increase the complexity (e.g., training data creation), although Table 10 shows that using all the retrievers to create training data is still more efficient than using one cross encoder.   In this paper, we want to highlight the importance of diverse supervision signals as opposed to the widely accepted wisdom that single and strong cross-encoder can provide sufficient supervision for DR training. We agree that creating diverse supervision signals with a less complex method is also important and will add this point to our future work.\n\n2. It is not clear what is the optimal number of supervisions: \nInstead of claiming how many teachers could give the optimal results, our intuition is that as long as we can find a teacher that creates a more diverse signal in our augmentation data, we can still improve our model’s retrieval effectiveness. See Table 9 in Appendix A.6 on how we measure the diversity of our progressively augmented data.\n\n3. Performance of CE model and do you follow the rule of supervision ordering: \nWe use the cross-encoder ``cross-encoder/ms-marco-MiniLM-L-12-v2`` from Huggingface mentioned in Appendix A.2; its ranking effectiveness (using DRAGON as the first-stage retriever) is above 0.4 in MRR@10 on MS MARCO development queries. Although we did not measure its zero-shot effectiveness on BEIR, the results from [1] shows that ``cross-encoder/ms-marco-MiniLM-L-6-v2``, the smaller version of our cross-encoder renaker, reaches over 0.49 on BEIR 18 datasets. Thus, we assume that our CE model has better zero-shot effectiveness than the other teachers and the experiments in section 5 do follow the rule of supervision ordering we proposed.\n\n4. Why does the CE model not provide additional benefit: \nWe hypothesize that the CE supervision cannot create a more diverse signal since the training data is created by re-ranking DRAGON top-1000 results, which is still very similar to the original DRAGON ranked list.\n\n[1] Guilherme Moraes Rosa, Luiz Bonifacio, Vitor Jeronymo, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. No parameter left behind: How distillation and model size affect zero-shot retrieval. arXiv preprint arXiv:2206.02873, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method shows state-of-the-art performance on supervised and zero-shot evaluation settings.\n2. The proposed data augmentation approaches are less expensive compare with prior work. For example, using sentence cropping in fine-tuning and using retrievers instead of expensive cross-encoder model for labeling.\n3.  The paper is mostly well written and easy to follow.", "reasons_to_reject": "1. Even though using retrievers for labelling are more efficient than cross-encoder models, the proposed approach requires multiple teacher retrievers, which increases the complexity from a different aspect. \n2. It is not clear what is the optimal number of supervisions needed for relevance label argumentation. ", "questions_to_authors": "1. In section 5, for cross encoder experiment, what is the performance of CE model in both supervised and zero-shot setting? It seems that CE is added in the last iteration, does this follow the rule of ordering different supervision mentioned in section 3.3 (i.e. augment a supervision at a later stage if it has better generalization capability?)  Why does CE model not provide additional benefit?\n", "paper_topic_and_main_contributions": "This paper discusses two data augmentation approaches that are commonly used in dense retrieval models and proposes a unified framework to leverage the benefits of two approaches to improve dense retrieval. For query argumentation, the proposed method mixes two types of data, namely sentence cropping and synthetic queries. And for relevance label argumentation, author proposed a progressive label method to distill knowledge from multiple and diverse supervisions. The proposed model has smaller size compared with other state-of-the-art models but has comparable performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Table 4, Contriever (8) is listed as a baseline dense retriever, but it is used as one of the supervision. would it better to move it next to GTR-XXL and to not use for comparison with proposed models. "}}, {"review_id": "SKKS7ZJb7K", "reviewer": "Reviewer_a2hW", "summary": "", "questions": "1. The author should provide more technical details, such as code disclosure, to facilitate easy reproduction and evaluation of the proposed method. \n\n2. Additionally, conducting a theoretical analysis of the proposed technology, particularly on how to prevent the propagation of errors generated during data augmentation, would enhance the paper's scientific rigor and reliability.\n\n\n\n\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s helpful suggestions. See below responses to the concerns and questions.\n\n1. Issue of reproduction: \nWe have released all our training and evaluation code, augmented training data and model checkpoints but cannot provide the link for anonymity.\n\n2. Theoretical analysis of the proposed approach on the prevention of error propagation generated during data augmentation:\nThe intuition behind our training strategy is to prevent errors created during data augmentation (DA). As discussed in lines 194--199, we challenge the previous approaches (e.g, contrastive pre-training and GenQ) which assume that the generated queries or segmented sentences are relevant to their original documents. Thus, we propose relevance label augmentation to mine relevant documents for each augmented query. Furthermore, we improve the quality of relevance labeling using multiple teachers. We believe our extensive evaluations on diverse datasets can justify our approach. \n\nSecond, while it is challenging to provide a theoretical analysis of a deep neural model without making over-simplifying assumptions, we attempt to provide insights and explanations to our modeling choices and observations whenever possible. For example, we draw inspiration from curriculum learning to explain the result that uniform sampling provides better supervision accuracy than progressive supervision but leads to suboptimal performance (see the detailed analysis in Appendix A.6).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results demonstrate the effectiveness of the proposed data augmentation strategy.\n\n\n\n\n", "reasons_to_reject": "The technical details provided in this paper are not sufficient for easy reproduction and lack theoretical analysis, particularly regarding error propagation prevention during data augmentation and discussion on error rates in formulas. Further clarification and additional theoretical analysis are needed to strengthen the paper's scientific rigor and reproducibility.\n\n\n\n\n", "questions_to_authors": "1. The author should provide more technical details, such as code disclosure, to facilitate easy reproduction and evaluation of the proposed method. \n\n2. Additionally, conducting a theoretical analysis of the proposed technology, particularly on how to prevent the propagation of errors generated during data augmentation, would enhance the paper's scientific rigor and reliability.\n\n\n\n\n", "paper_topic_and_main_contributions": "This paper presents a progressive method for data augmentation to enhance query retrieval and obtain a more comprehensive set of retrieval content.\n\n\n\n\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new DA approach with diverse queries and sources of supervision to progressively train a generalizable dense retrieval. Although there are varying opinions among the reviewers, the majority have given positive scores. Reviewers have raised concerns on comparison with larger model or potential model collapse when enhancing  with negative samples, or model complexity. The rebuttal has addressed these concerns. These suggestions by reviewers have contributed to strengthening this paper. Reviewers have high excitement score to this paper.
Decision: Accept-Findings