CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion
Authors: Xingwei He
Yeyun Gong
A-Long Jin
Hang Zhang
Anlei Dong
Jian Jiao
Siu Ming Yiu
Nan Duan
Venue: EMNLP 2023 Main (2023)
Keywords: Dense Retrieval
Document Expansion

Abstract: The dual-encoder has become the de facto architecture for dense  retrieval. Typically, it computes the latent representations of the query and document independently, thus failing to fully capture the interactions between the query and document. To alleviate this, recent research has focused on obtaining query-informed document representations. During training, it expands the document with a real query, but during inference, it replaces the real query with a generated one. This inconsistency between training and inference causes the dense retrieval model to prioritize query information while disregarding the document when computing the document representation. Consequently, it performs even worse than the vanilla dense retrieval model because its performance heavily relies on the relevance between the generated queries and the real query. In this paper, we propose a curriculum sampling strategy that utilizes pseudo queries during training and progressively enhances the relevance between the generated query and the real query. By doing so, the retrieval model learns to extend its attention from the document alone to both the document and query, resulting in high-quality query-informed document representations. Experimental results on both in-domain and out-of-domain datasets demonstrate that our approach outperforms previous dense retrieval models.

CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion
Xingwei He Yeyun Gonga A-Long Jin', Hang Zhang,, Anlei Dong', Jian Jiao', Siu Ming Yiu"t, Nan Duan 'The University of Hong Kong, 'Microsoft Research Asia, 'Microsoft hex hexingweil 5@gmail. com, ajin@eee. hku. hk, smyiuêcs. hku. hk, {yegong, v-zhhang, anlei. anlei.dong, jian.jiao, nanduan)emicroseen
Abstract
to compute relevance between the query and docu- ment. Recently, there has been a surge of research interest in neural network- dense retricval (Karpukhin et al., 2020; Xiong et al., 2021). Dif- ferent from sparse retrieval, dense retrieval resorts to neural encoders to compute the dense represen- tations of the query and document. This enables dense retrieval to infer the relevance between them at the semantic level rather than the surface level, thus circumventing the term mismatch problem suffered by the sparse retrieval models.
The dual-encoder has become the de facto ar- chitecture dense retrieval. Typically, it com- putes the latent representations of the query and document independently, thus failing to fully capture the interactions between the query and document. To alleviate this, recent research has focused on obtaining query-informed doc- ument representations. During training, it pands the document with a real query, but dur- ing inference, replaces the real query with a generated one. This inconsistency between training and inference causes the dense retrieval model prioritize query information while dis- regarding the document when computing the document representation. Consequently, it per- forms even worse than the vanilla dense re- trieval model because its performance heavily relies on the relevance between the generated queries and the real query. In this paper, propose a curriculum sampling strategy that utilizes pseudo queries during training and pro- gressively enhances the relevance between the generated query and the real query. By doing so, the retrieval model learns to extend its atten- tion from the document alone to both the docu- ment and query, resulting in high-quality query- informed document representations. Experi- mental results on both in in-domain and out-of- domain datasets demonstrate that our approach outperforms previous dense retrieval models.
In recent years, the dual- architecture has been a standard workhorse for dense retrioval. One major disadvantage of this architecture that it can only partially extract the interactions between the query and document, since it encodes them sep- arately. By comparison, the cross-encoder archi- tecture effectively capture the deep correlation between them by taking the concatenation of the query and document as input. By directly concate- nating the query and document, the cross-encoder gains an advantage in capturing interactions, but also loses the advantage of pre-computing docu- ment representations during inference. Therefore, cross-encoder cannot wholly replace dual-encoder.
To enhance the retrieval models' ability to cap- ture interactions between queries and documents while maintaining retrieval efficiency, previous work mainly focuses on generating query-informed document representations. One approach, known as late interaction (Khattab and Zaharia, 2020), in- volves encoding the query and document indepen- dently in the early layers, while the later layers model their interactions. Late interaction combines dual-encoder and cross-encoder, making trf between retricval efficiency and performance. On the other hand, Li et al. (2022) proposed promis- ing retrieval architecture, (ual-cross-encoder As shown Figure (c), this architecture computes the query-related document representation by ex- panding the document with a real or pseudo query.
1 Introduction
Text retrieval aims to find the relevant documents for a given query from a large collection of doc- uments, playing an indispensable role in open- domain question answering (Chen et al., 2017). fact verification (Thorne et al., 2018) and retrieval- augmented generation (Lewis et al., 2020; He et al., 2022). At the early stage, sparse retrieval methods such TF-IDF or BM25 dominated passage re- trieval by relying mainly on lexical term matching
"Work done during internship at Microsoft Research Asia. +Corresponding authors.

---

Compared with late interaction, dual-cross- encoder (i.e., dense retrieval with document ex- pansion) gets the query-related document represen- tation without sacrificing the retrieval efficiency at inference. However, there exists a discrepancy between training and inference in the current dual- cross-encoder retriever. Specifically, during train- ing, the document is expanded using a real query, whereas during inference, the document enriched with a generated query. This diserepancy causes the learned retriever overly focus on the query, yet ne- glect the document, when computing the document representation. During inference, the generated query of significantly differs from the user-input query the query-related document representation will be misled by qr, thus degrading the perfor- mance. That is why the fuul-cross-encoder even underperforms the vanilla dual-encoder. To ad- dress this issue, Li al. (2022) proposed solu- tion by computing multiview document represen- tations using different generated queries for each document. While multiview document representa- tions improve retrieval performance, they come at the cost of significantly increased retrieval latency, which scales lincarly with the number of views.
ing and inference for dense retrieval with document expansion, further improving the query-informed document representation'. (2) We propose to com- pute the typical document representation rather than using multiview document representations at inference, which balances the retrieval efficiency and performance. (3) To verify the effectiveness. apply our proposed approach to two different dense retrieval models, DPR (Karpukhin al., 2020) and coCondenser, and conduct extensive experiments on three in-domain retrieval datasets, and the zero-shot BEIR benchmark. Experimental results show our proposed approach brings substan- tial gains over competitive baselines. (4) To the best of our knowledge, this is the first time that doc- ument expansion is successfully applied to dense retrieval without incurring extra retrieval latency.
2 Preliminary
In this section, we introduce the definition of text retrieval and three architectures for dense retrieval.
Task Description. Text retrieval meant to find the most relevant M documents D+ {d; .d.....dd ddir for the given query from a large corpus D = {di, dN, with N docu- ments (M < N).
In this paper, we propose CAPSTONE, curriculum sampling for dense retrieval with document expansion, to bridge the gap between training and inference for dual-cross-encooder Our motivation to expect the dual-cross-encoder re- trieval model can utilize both the document d and pseudo query of to compute the query-informed document representation. To achieve this, we train the mual-cross-encodee retriever by gradually in- creasing the relevance of the pseudo query qf to the gold query g. Specifically, at the carly training stage, pseudo query of irrelevant to q is selected, causing the retriever to solely rely on the docu- ment. As we progress to the late training stage, a highly related pseudo query qf is chosen, allow- ing the retriever to learn to augment the document representation with the pseudo query. By doing so, we alleviate the discrepancy between training and inference. During inference, if the user-input query q is similar to the pseudo query qr, then qf will contribute more to making the target document d be retrieved. Otherwise, the retrieval model will mainly rely on the relevance between q and d.
Enco-er Architecture. Dual-encoder (DE) is the typical dense retrieval architecture. As shown in Figure 1 (a), it consists of query encoder Eg and document encoder Eg, which encode the query q and document into dense vectors (q) and Fa(d), respectively. Previous works usually initialize the encoders with BERT and use the [CLS] vector at the last layer as the dense representation. The similarity score between and is measured with the inner product of their vectors:
sim d) (
(1)
Cross-Encoder Architecture. Since DE models and d separately, it is not good at capturing the rel- evance between them. To capture the interactions between them, cross-encoder (CE) directly takes the concatenation of q and d as input, shown in Figure 1 (b). It first computes the dense represen- tation E(q + d) for them and then uses the fully
To summarize, the main contributions of this pa- per are as follows: (I) We propose a curriculum learning approach to bridge the gap between train-
'Our code is available at: https://github.ccm mi nicrosoft/Siins//cesTtot

---

score
score
score
CLS
CLS
CLS
CL.S
CLS
Query Encoder
Document Encoder
Document Encoder
Query Encoder
Document Encoder
Query q
Document
Query q
Document
Query
Document d + Query q'
(a) Dual-encoder
(b) Cross-encoder
(c) Dual-cross-encoder
Figure 1: Illustration of the dual-encoder, cross-encoder and call--roos-eccoeee architectures, and denote the gold query and generated query for the document d. '+' is the concatenation operation.
connected layers (FCL) to compute the similarity score:
d +
Positive
Positive
+
Hard Negatives
Hard Negatives
d +
dj +
FCL(((
(2)
d a
4
dg 9
where '+' is the concatenation operation.
In-batch Negatives
In-batch Negatives
Dual-Cros--Enoder Architecture. Although CE can extract more fine-grained relevance be- tween q and d, it is not suitable for retrieval. To combine the advantages of DE and CE, Li et al. (2022) proposed mual-cross-encodee (DCE). As shown in Figure (c), DCE also consists of query encoder and a document encoder. The only differ- ence between DE and DCE is that the document encoder of DCE takes the document and the gener- ated query gf as input during inference. Therefore, DCE can be regarded as DE with document cxpan- sion. Similar to DE, DCE computes the similarity score between and d with the inner product:
d++ + q:
d++1 gét1
d, + 9,
d, + qn
Figure 2: Comparison of different document expansion methods when training DCE, where the left is the vanilla method and right is our proposed method. d is the positive document for q. dg....,d .d the hard negatives. di......, dn are the in-batch negatives. g' and of are the generated query for d and d..
where D is the set of negative documents, con- taining hard negatives and in-batch negatives.
During training, DCE expands the positive doc- ument d and hard negatives with the gold query (e.g., replacing d with d + q). For ease of un- derstanding, we show how to construct positive, hard negatives and in in-batch negatives in the left of Figure 2. As shown in this figure, DCE can filter out all in in-batch negatives by solely relying the query information from the expanded document, since only needs observe whether the input query appears the expanded document. The uti- lization of document information is necessary only for distinguishing the positive from the hard nega- tives. Consequently, the learned representation of the expanded document has a strong bias toward the query, while almost neglecting the document. At inference, DCE enriches the document with the generated query o' rather than the user-input query q. When q and of are different types of queries for the target document d, or Eq(4) will be far from g' or Ea(g). As the retrieval model
sim(g, = =g) Ea(d + qf).
3 Methodology
Section 3.1, we first identify the discrepancy between training and inference DCE. Then, we introduce how to bridge the gap with curriculum learning in Section 3.2. Finally, we will show our proposed inference method in Section 3.3.
3.1 Diserepancy in Training and Inference
The training objective of DE is to learn dense rep- resentations of queries and documents maximize the similarity score between the query and positive document. The training loss is defined as follows:
L(g, (
sim(g.d+)
=-lo esimlged
Σ
)
d d-

---

Generated queries for the document
B(d))
Edd; 941)
Sort them based on Mlq.97 9'i) Divide them into K groups.
Retrieve
Average
Edid; + ,/")
E,(9)
E(d)
Relevance to the query
Group
Group
Group K
E,dd, + q.¹)
Sample
Sample
Sample
Eldy)
Iteration
Figure The overview of the retrieval stage, d, is the i- 1/ document, and N refers to the size of the external corpus. of means the j-th generated query for d,.
Iteration
The query selection process based on cur- riculum learning in each training iteration. We first compute the relevance score M (4, y between the gen- erated query of and the gold query q, and then sort the generated queries in ascending order according to their relevance scores to Next, we divide them into K groups. At the i-th training iteration, we randomly sam- ple one generated query from the i-th group, and then use the sampled query to update the dual-cross encoder.
Figure 3:
both the document and the generated query during training, that the learned document representa- tion contains the information of both parts.
To fulfill this goal, we further propose the cur- riculum sampling. To be concrete, at the early training stage, the selected of has low correlations with q, forcing the retriever to use the document. As the training goes on, we select of with gradually increased relevance q. encouraging the retriever to use the query. Driven by this motivation, we first generate some queries for each document d. Then, we compute the relevance score between the generated query of and the gold query q with an automatic evaluation metric M (i.e., ROUGE-L. ). After that, we sort the generated queries in ascend- ing order according to their relevance scores to g and divide them into K groups. We summarize the curriculum sampling in Figure 3.
overly relies on the query part when computing the document representation, Ed + g') will be close easily infer that Ea(d+ q is far away from (q), making it more difficult to retrieve the target document d. As a re- sult, the dense retriever with document expansion even underperforms its counterpart without using document expansion. We attribute the performance degradation to the discrepancy between training and inference.
to Ea(4'). Based these,
3.3 Inference
3.2 Bridging the Gap with Curriculum Sampling
Before retrieving, we first generate S queries for each document with the generator. Next, we con- catenate a document with one generated query and compute its latent representation with the document encoder. Then, will get S different representa- tions for each document.
To bridge the gap between training and inference, we propose to expand the document with its gener- ated query during training. We show the training process of our proposed method in the right of Fig- ure 2. Although this idea is simple and intuitive, it brings a significant advantage: the retriever cannot take a shortcut during training, where the retriever cannot exclude in- in-batch negatives by simply check- ing whether the user-input query appears in the expanded document.
Corpus Expansion. Following Li et al. (2022), we keep S representations for each document, so the original corpus will be expanded S times. When a query comes, we need retrieve the rele- vant documents from the expanded corpus, which increases the retrieval latency by S times.
However, this is not the end of the story. If the generated query g' used to expand d is irrelevant to the gold query q, the document retriever of DCE will ignore q', which means DCE will degenerate into DE. On the other hand, if of and are similar, we will encounter the abovementioned problem (DCE will overly depend on g¹). We expect the dense retriever with document expansion can use
Computing the Typical Representation of Dif- ferent Views. To avoid increasing the retrieval latency, we can retain a single representation for each document. The easiest way to set S 1. However, we found that the retrieval performance
"The evaluation code for ROUGE-I. is available at: https: //huggingf aaaggnngga.e..cmmm wttspaccs//cpccccccccccoccc.. htt/spaaee/eaallsseeeeeeeeeeeeee ewwllate--eeeeeeeeee mettteerottiiirueeeeee

---

Datasets
Train
Dev
Test
#Passage
tracks and BEIR benchmark?
MS- MARCO
502,939
6,980
1,24,,23
TREC-19
43
8,841,823
Implementation Details. We set the maximum document length be 144 tokens, and the maxi- mum query length to be 32 tokens. All models are optimized using the AdamW optimizer (Loshchilov and Hutter, 2019) with the learning rate of 5 x 10 6 and a linear learning rate schedule with 10% warm- up steps for around three epochs. Every training batch contains 64 queries, and each query is ac- companied by positive and 31 hard negative documents. Following Gao and Callan (2022), we train our proposed model for two stages and initial- ize the retriever with coCondenser at each stage. At the first training stage, the hard negatives are sampled from the official BM25 hard negatives, but at the second training stage, the hard negatives are sampled from the mined hard negatives. For each training stage, we evaluate the last model training checkpoint the retrieval datasets.
TREC-20
54
8,841,823
Table 1: Basic statistics of retrieval datasets. #Passage is the number of unique passages in the retrieval corpus.
is positively correlated to the number of views, S. This is because a larger value of S increases the probability of the user-input query being relevant to one of the generated queries. Therefore, set- ting S to will degrade the retrieval performance compared with expanding the corpus.
Given this, we propose to compute the typical representation for document by averaging all S different views. This approach allows us to main- tain only one representation for each document, As shown in Figure 4, our proposed method does bring additional retrieval latency', yet performs much better than simply setting S to (please refer to Section 4.3 for more details).
The query generator is built upon the seq2seq architecture (Sutskever et al., 2014; He and Yiu, 2022), designed to take a passage as its input and has been specifically trained to create synthetic queries that correspond to the provided passage. Nogueira and Lin (2019) initialized the query gen- erator with the T5-base model. They trained it the MS-MARCO dataset and generated 80 queries for each passage within MS-MARCO. Following Li et al. (2022), we use the queries released by Nogucira and Lin (2019) to expand the passages in the MS-MARCO dataset. In parallel, for BEIR, we use the query generator" fine-tuned on MS- MARCO to generate 5 queries for each document within the BEIR benchmark. During the query generation process, we employ the top-k sampling approach with k set 10, and limit the maxi- mum length of the synthetic queries to 64 tokens. During training, we implement our proposed cur- riculum sampling with all 80 queries and divide them into K groups, but at inference, we resort to the first 10 (i.c., S = 10) and first 5 (i.c., S = 5) queries compute the typical document represen- tation for MS-MARCO and BER benchmark, re- spectively. At the first training stage, we set K 3, and at the second training stage, we set K to 4.
4 Experiments
4.1 Experimental Setups
Datasets. We conduct experiments on three pas- sage retrieval datasets: MS-MARCO passage rank- ing (Bajaj et al., 2016), TREC Deep Learning (DL) Track 2019 (Craswell et al., 2020) and 2020 (Craswell et al., 2021). MS-MARCO collects real user queries from Bing search and passages from web collection. TREC DL tracks share the same retrieval corpus with MS-MARCO and provide an- other two annotated test sets. We train our models on the MS-MARCO train set, and then evaluate them on the MS-MARCO development set", TREC DL 19 and 20 test sets. Table shows the statis- ties of these datasets. We evaluate the zero-shot retrieval performance BEIR benchmark (Thakur al., 2021), which contains 18 datasets across different domains.
Evaluation Metrics. Following previous work, we use MRR @ 10, Recall@500, and Recall @ 1000 evaluate the retrieval performance on MS- MARCO, where MRR@ 10 is the most important metric. We resort to nDCG@ 10 for TREC DL
'The official evaluation toolkit is available https:// gi thub. wow/beirr-elllr///aeee /beir. /https://git. wttp:/tttttaeeoo.oooooooioiaao cawmlii//in////////mmm..ma.mimma queries_tapk_sampllee "https: htttns://hugginggaae.. co/ http:/aaaa..cccctaannnniaa castorini/
WW can pre-generate queries and pre-compute document representations. Therefore, the retrieval latency mentioned this paper does not encompass these pre-processing steps. 'The MS-MARCO test set is not publicly available.
doc t5- base-msmarco

---

MS-MARCO MRR@IN R@
TREC DL nDCG
TREC
Models
nDCG@10
Sparse retrieval
BM25 (Yang et al., 2017)
18.5
58.5
85.7
51.2
47.7
DeepCT (Dai and Callan, 2019)
24.3
69.0
91.0
57.2
DocTSQuery (Nogueira and Lin, 2019)
27.7
75.6
94.7
64.2
Dense retrieval
DPR (Karpukhin et al., 2020)
31,4
95.3
59.0
62.1
ANCE (Xiong et al., 2021)
33,0
95.9
64.5
64.6
SEED (Lu et al., 2021)
33,9
96.1
STAR (Zhan et al., 2021)
34.7
68.3
TAS-B (Hofstätter al., 2021)
34,0
97.5
71.2
69.3
RocketQA (Qu et al., 2021)
37,0
85.5
97.9
COIL (Gao et al., 2021)
35.5
96.3
70.4
ColBERT (Khattab and 7aharia, 2020)
36,0
82.9
96.8
DCE (Li et al., 2022)
36,0
96.4
68.3
68.9
RetroMAE (Xiao al., 2022)
35.0
97.6
Condenser (Gao and Callan, 2021)
36,6
97.4
69.8
coCondenser (Gao and Callan, 2022y
37,9
86.3
98.4
70.7
69.8
CAPSTONE
38.6
86.6
98.6
71.1
70.3
Table Passage retrieval results on MS-Marco Dev, and TREC datasets. Results with are from our reproduction.
During inference, we pre-compute representations for all documents in the retrieval corpus, and build IndexFlatIP indexes for document representations with the FAISS library (Johnson et al., 2019) to accelerate retrieval.
model, CAPSTONE, fine-tuned with MS-MARCO the BEIR benchmark. During evaluation, compute the typical representation for each doc- ument with 5. As shown in Table 3, CAP- STONE performs much better than all baselines the average performance. Compared with coun- terpart without document expansion (i.e., coCon- denser), our model achieves improvements on 11 out of 18 datasets and increases nDCG@ 10 by 0.6 points in the average performance. These improve- ments demonstrate that approach generalizes well diverse domains under zero-shot settings.
We implement all models with the HuggingFace Transformers library (Wolf et al., 2019) and con- duct all experiments on 8 NVIDIA Tesla V100 GPUs with 32 GB memory. We also use the gradi- ent checkpointing technology (Chen al., 2016) reduce the memory of deep neural models, and mixed precision training to speed up training.
4.2 Experimental Results
4.3 Ablation Study and Analysis
To analyze our method more directly, experiments in this section are based the vanilla DPR initial- ized with the PRNII-2.0-bbai model (Sun et al., 2020) without using dense retrieval pre-training strategies, if there is no particular statement.
In-domain Performance. Table shows the per- formance of our proposed model and baselines on MS-MARCO, TREC-2019 and TREC-2020. Since we initialize our model, CAPSTONE, with coCon- denser, coCondenser is the main comparison model. our proposed model im- proves coCondenser by large margin, increasing by 0.7 points in MRR @ 10 on MS-MAN MS-MARCO, 0.4 points on TREC DL 19 and 0.5 points on TREC DL 20 datasets. These improvements verify the effectiveness of our proposed curriculum learning and typical representation strategy.
From Table
Corpus Expansion vs. Document Expansion. In this paper, we append only one query to doc- ument both training and inference. As stated in 3.3, corpus expansion will enlarge the original corpus to S times during inference. In contrast, document expansion for sparse retrieval (Nogueira and Lin, 2019) appends multiple generated queries to a document at inference, thus without bringing extra retrieval latency. Since document expansion" has shown effectiveness in sparse retrieval, it is very intuitive to consider whether it is effective for dense retrieval. To answer this question,
Zero-shot Performance. To test the out-of- domain generalization capabilities, we first use the T5-base model (Nogueira and Lin, 2019) trained MS-MARCO to generate queries for each doc- ument of BEIR benchmark, and then evaluate our
'I this section, document expansion means expanding document with multiple queries.
Phttss//gtthub.ccm////////aa

---

Dataset
BERT
LaPraDoR
SimCSE
DiffCSE
SEED
Condenser
coCondenser" CAPSTONE
TREC-COVID
64.9
49.5
52.4
49.2
61.2
75.4
74.0
77.9
BioASQ
26.2
23.9
26.4
29.7
31.7
34.1
34.3
NFCorpus
25.7
28.3
25.0
25.9
25.6
27.8
32.4
33.0
NQ
43.8
41.5
41.2
41.2
42.5
45.9
50.5
50.5
HatporQA
47.8
48.8
50.2
49.9
52.8
53.7
56.4
56.7
FIQA-2018 2018
23.7
26.6
24.0
22.9
24.4
26.1
30.0
30.4
Signal- IM (RT)
21.6
24.5
26.4
26.0
24.6
25.8
24.7
23.1
TREC-NEWS
36.2
20.6
36.8
36.3
33.5
35.3
39.1
40.3
Rohust04
36.4
31.0
35.3
34.3
34.8
35.2
40.3
40.7
ArguAna
50.3
43.6
46.8
34.7
37.5
40,9
39.2
Touche-2020
17.8
17.8
22.3
27.0
CQADupStack
28.4
32.6
29.5
30.5
28.5
31.6
30.0
30.0
Quora
78.2
84.3
84.8
85.0
84.9
85.5
84.3
83.8
DBPedia
29.8
32.8
30.4
30.3
32.4
33.1
37.2
38.0
SCIDOCS
11.5
14.5
12.5
12.5
11.7
13.6
14.3
14.3
FEVER
68.4
51.8
65.1
64.1
65.3
68.2
72.4
72.7
Climate-FEV FFVER
20.5
17.2
22.2
20.0
17.6
19.9
19.4
19.3
SciFact
50.4
48.3
54.5
52.3
55.6
57.0
58.3
60.5
Avg. Performance
37.6
35.8
37.7
37.2
37.7
40.3
42.7
43.3
Table Zero-shot dense retrieval performances on BEIR benchmark (measured with nDCG@ 10). are from reproduction.
Results with
16.0
Document Expansion
Retrieve
d
25.5
25.0
d
4
34.5
Corpus Expansion
Retrieve
d
14.0
DPR
4
corpus expansion
33.5-
DPR w/ decument expansion
d
asymmetric
33.0
Number of queries used during inference (5).
query 4
Figure 5: Results DPR with corpus, document, and asymmetric expansion the dev set of MS MarMo.
Figure 6: Illustration of document expansion and corpus expansion at inference. For simplicity, only the positive document d; of the query q is shown.
conduct experiments DPR with three different settings: (I) corpus expansion; (2) document ex- pansion; (3) asymmetric expansion, which appends one query to a document at training, but appends S queries to document at inference. As shown in Figure 5, unlike corpus expansion, using more queries does not bring clear improvements for doc- ument and asymmetric expansion, but sometimes damages their performances. To explain this, illustrate document expansion and corpus expan- sion in Figure Document expansion models the target document and all queries together, where irrelevant queries acting like noise will corrupt the document representation. By comparison, corpus expansion nicely bypasses this problem by model- ing different queries independently.
uments with different query selection strategies: (1) gold, meaning using the gold query (i.e., strategy used by the vanilla DCE (Li et al., 2022)); (2) ran- dom, meaning randomly sampling query from the generated queries; (3) top-k and (4) bottom- meaning sampling query from the top-k and bottom-k: sorted queries, respectively.
From Figure 7, we observe that: (1) DPR trained with the gold query performs worst among all strategies, even worse than the vanilla DPR (DPR w/o query) when 5 < 8. As we have stated above, expanding the document with the gold query during training will make the retriever pay more attention the query while ignoring the document. At infer- ence, when the provided queries are limited (i.e., is small), is unlikely to find a query similar to the user-input query. However, as S increases, the
Efffec Query Selection Strategies. To demon- strate the advantage of our proposed curriculum sampling, we expand the positive and negative doc-

---

36.5
Variants
MRR@10
R@ 1000
DPR
34,26
97.02
36.0
CAPSTONE+DER w/
35.15
97.19
CAPSTONEADEE w/ averuge
35.66
97.28
25.5
CAPSTONE+DER w/
35.45
97.25
CAPSTONE=DPR
97.20
MRR@10
Table Results of CAPSTONE initialized with DPR using different pooling methods to compute the typical representation on the dev set of MS-Marco.
DRR wio querry
34.0
DPR w/ gold
top-1
33.5
DPR bottom-1
Models
BM2S Negatives
Mined Negatives
DPR random
MRRØ1O R@1000
MRRG
RØ1000
DRR curriculum
DPR CAPSTONE-DFR
341.26
97.02
36.44
97.65
33.0
Number of queries used during inference (5).
CAPSTONE=ooCCndeeee
coCondenser
35.91
98.21
37.94
98.41
08.60
Figure 7: Comparison of different query selection strate- gies. During training, we first expand the document with query selected from the generated queries us- ing a specific strategy, and then train DPR with the expanded document. During inference, we evaluate the well-trained DPR retrievers with corpus expansion (solid lines) and the typical representation (dotted lines) the dev set MS-Marco, where the solid and dotted lines with the same color are results for the same DPR.
Table 5: Performance of CAPSTONE initialized with DPR and coCondenser at the two training stages on the dev set of MS-Marco.
Figure 7). our proposed curriculum sampling per- forms best, which again verifies the effectiveness of curriculum sampling. In addition, the retrieval performance of typical representations outperforms its counterpart using corpus expansion with S for all query selection strategies. More importantly, for the curriculum learning strategy, the typical rep- resentation enhances the vanilla DPR by 1.2 on MRR 10, and is even on par with its counterpart using corpus expansion with 3, but there still exists clear performance gap between the typical representation and corpus expansion with S = 10. Therefore, the typical representation balances the retrieval efficiency and performance.
probability of finding a query similar to the user- input query also increases. (2) DPR trained with the top-1 query performs much better than DPR trained with the gold query, since slightly shifts its attention from the query to the document. How- ever, still cannot exceed DPR w/o query when 5= indicating that it fails to fully utilize the doc- ument like DPR. (3) the document expanded with weakly related (bottom-1) or random query at training, DPR with both settings will outperform DPR w/o query when S = 1. These two strate- gies enable DPR to mine query information with- out sacrifieing document information. However, their performances are not strongly positively cor- related with S like the top-1 strategies, indicating does not fully use the query information. (4) DPR with the curriculum strategy outperforms DPR w/o query when S = 1, and its performance linearly in- creases with S, verifying this strategy enables DPR to utilize both the document and query to model the query- informed document representation.
Comparison of Methods for Computing the Typ- ical Representation. We consider three different pooling methods to compute the typical document representation: taking the averageemmxx/median pooling of different document views. From Table 4, we observe that (1) DPR with S improves DPR by 0.9 on MRR @ 10: (2) DPR with the typical representation brings an extra 0.5 improvements on MRR @ 00; (3) There is no significant difference the results of the three pooling methods. Given the simplicity, we use the average pooling to compute the typical representation.
Effect of the Typical Representation. Although corpus expansion makes the retrieval performance of our proposed curriculum sampling improve with the increase of S (see the solid red line in Figure 7), the retrieval latency also increases linearly. To alleviate this, we propose use the average of dif- ferent document views as the typical representation. Among all strategies (see the dotted lines in the
Multi-stage Retrieval Performance. To further verify the effectiveness of our proposed approach, apply our proposed approach DPR and co- Condenser. From Table 5, we observe that our ap- proach brings clear improvements to DPR with 1.4 and 0.8 increases on MRR@ @ at the two training stages, respectively. In addition, we witness similar

---

improvements on coCondenser, proving that our approach does not depend on retrieval models and is applicable to different retrieval models.
by adding relevant contexts, including the answer. the sentence containing the answer, and the title of a passage where the answer belongs. However, query expansion will increase the retrieval latency during inference, since the generation model is re- quired to generate these relevant contexts for query expansion. Therefore, we mainly focus on docu- ment expansion in this work.
5 Related Work
5.1 Dense Retrieval
In recent years, with the development of large- scale pre-trained language models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), we witness the research interest in infor- mation retricval shifting from the traditional sparse retrieval to neural dense retrieval. Karpukhin et al. (2020); Xiong et al. (2021) proposed dense pas- sage retriever (DPR), which uses two neural en- coders initialized with BERT to model the query and document, independently. The subsequent works follow this dual-encoder framework. One line of work improves dense retrieval by mining hard negatives, where they select the top-ranked documents retrieved by the recent retriever as hard negatives and then re-train the retriever with the newly mined hard negatives (Xiong et al., 2021; Qu et al., 2021), However, the mined hard nega- tives highly likely contain false negatives, harming the performance. To mitigate this, follow- ing studies denoise the mined hard negatives with re-rankers (Qu et al., 2021; Ren et al., 2021; Zhang et al., 2022). Another kind of work focuses on pre-training to make the pre-trained models more suitable for dense retrieval, such as Condenser (Gao and Callan, 2021), coCondenser (Gao and Callan, 2022) and SIMLM (Wang et al., 2022).
5.3 Document Expansion
Document expansion augments the document with generated queries, which the document might an- swer. Compared with query expansion, document expansion can be conducted prior indexing with- out incurring extra retrieval latency. Document expansion has shown its effectiveness on sparse re- trieval models (Nogueira et al., 2019; Nogueira and Lin, 2019), yet how to apply it to dense retrieval models is still under-explored. Li et al. (2022) first applied document expansion to dense retrieval models. However, their models suffer from the dis- crepancy between training and inference, thus even degrading the vanilla dense retrieval models under the single-view document representation settings. To mitigate this discrepancy, we propose a curricu- lum learning approach, which successfully applies document expansion to dense retrieval models.
6 Conclusion
This work proposes the curriculum sampling for dense retrieval with document expansion, which enables dense retrieval models to learn much better query-related document representations. In addi- tion, we propose to compute the typical representa- tion of different document views, which balances inference efficiency and effectiveness. Our experi- mental results on the in- and out-of-domain datasets verify the effectiveness of the curriculum sampling and typical representation.
Dense retrieval models typically depend on ex- tensive supervised data, comprising pairs of queries and positive documents. To address the challenge limited training data, Ma et al. (2021); Sun et al. (2021) proposed to train a query generator on high- resource information retrieval data, and then used the query generator to generate synthetic queries for low-source target domains. Additionally, Dai al. (2023) harnessed large language models in zero/few settings to produce synthetic queries for documents target domains, eliminating the need for training a general query generator. contrast to these approaches, our work leverages synthetic queries for document expansion.
Limitations
There are two possible limitations of this work. The first limitation is that we need to generate synthetic queries for each document in the retrieval corpus, which is very time-consuming. Luckily, this pro- cess does not bring extra delay retrieval. In addi- tion, limited by sufficient computational resources. we only verify the effectiveness of our method on vanilla DPR and coCondenser. In the future, we plan to apply our approach to other dense retrieval
5.2 Query Expansion
Query expansion enriches the query with various heuristically discovered relevant contexts. For ex- ample, GAR (Mao et al., 2021) expands the query

---

models and verify the effectiveness of our method on these models.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pee-t pre-training for dense pas- sage retrieval. In Proceedings of ACI., pages 2843- 2853, Dublin, Ireland. Association for Computational Linguistics.
Acknowledgements
This project is supported by HKU-SCF FinTech Academy, Shenzhen-Hong Kong-Macar Science and Technology Plan Project (Category C Project: SGDX200108831135537700
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of NAACL-HLT pages 3030-3042, Online. Association for Computational Linguistics.
and Theme- based Research Scheme of RGC, Hong Kong (T35- 710/20-R). We would like thank the anonymous reviewers for their constructive and informative feedback on this work.
Xingwei He, Yeyun Gong. A-Long Jin, Weizhen Qi Hang Zhang, Jian Jiao, Bartuer Zhou, Biao Cheng, Sm Yiu, and Nan Duan. 2022. Metric-guided dis- tillation: Distilling knowledge from the metric to ranker and retriever for generative commonsense rea- soning. In Proceedings of EMNLP, pages 839-852, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
References
Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng. Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated ma- chine reading comprehension dataset. arXiv preprint 16Xiv:1611.099 1.09268.
Xingwei He and Siu Ming Yiu. 2022. Controllable dictionary example generation: Generating example sentences for specific targeted audiences. In Pro- ceedings of ACL, pages 610-627, Dublin, Ireland. Association for Computational Linguistics.
Danqi Chen, Adam Fisch, Jason Weston, and An- toine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of ACL, pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Sebastian Hofstätter, Sheng-Chich Lin, Jheng-Hong Yang. Jimmy Lin, and Allan Hanbury. 2021. Ef- ficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of SIGIR, page 113- 122, New York, NY, USA. Associ- ation for Computing Machinery.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint 1rXiv:1604.000
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-555 547.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview
trec 2020 deep learning track. arXiv preprint 102iv:2102.07662.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP, pages for Computational Linguistics.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820
Online. Association
Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv: 1910. 10687.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of SIGIR, page 39-48, New York, NY, USA. Association for Computing Machinery.
Zhuyun Dai, Vincent Y Zhao, Ma. Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Wei Chang. 2023. Promptagator: Few- shot dense retrieval from 8 examples. In ICLR.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin. Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. generation for knowledge- intensive nlp tasks. Proceedings of NIPS, Red Hook, NY, USA. Curran Associates Inc.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of NAACL-HLT, pages 4171-4186.
Luyu Gao and Jamie Callan. 2021. Condenser: pre- training architecture for dense retrieval. In Proceed- ings of EMNLP, pages 981-993, Online and Punta Cana, Dominican Republic, Association for Compu- tational Linguistics.
Zehan Li, Nan Yang, Liang Wang, and Furu Wei. 2022. Learning diverse document representations with deep query interactions for dense retrieval. arXiv preprint 2xXiv:2208.04232

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zet,lemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 17Xi::1907.11622
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. In NIPS, volume 27. Curran Associates, Inc.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Shuqi Lu, Di Hc, Chenyan Xiong, Guolin Kc, Walced Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021. Less is more: Pretrain strong Siamese encoder for dense text retrieval using a weak decoder. In Proceedings of EMNLP. pages 2780-2791, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
James Thome, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The fact extraction and VERification (FEVER) shared task. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 1- 9, Brussels, Belgium. Association for Computational Linguistics.
Ji Ma. Ivan Korotkov, Yinfci Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot neural passage retrieval via domain-targe targettd synthetic question generation. In Proceedings of EACI., pages 1075-1088, Online. Association for Computational Linguistics.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. SimLM: Pre- treirraiiig with represen- tation bottleneck for dense passage retrieval. arXiv preprint 2rXtv:200..02578
Yuning Mao, Pengcheng He. Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Ham, and Weizhu Chen. retrieval for open-
2021. Generation-augmeett domain question answering. Proceedings of ACL, pages 4089-4100, Online. Association for Computa- tional Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' e' Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of-the art natural language processing. arXiv preprint
Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docttutqquey.
Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. RetroMAE: Pre-training retrieval-oriented lan- guage models via masked auto-encoder. Proceed- ings of EMNLP, pages 538-544 138-548, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. arXiv preprint arXiv: 190.003904
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An opti- mized training approach to dense passage retrieval for open-domain question answering. In Proceedings of NAACL-HLT, pages 1835-5847, Online. Association for Computational Linguistics.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amoll Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In ICLR.
Peilin Yang. Hui Fang, and Jimmy Lin. 2017. Anserini: Fnnallig the of lucene for information retrieval research. In Proceedings of SIGIR, page 1253-1255 New York. NY. USA. Association for Computing Machinery.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. Proceedings of EMNLP, pages 2825-2835, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. In Pro- ceedings SIGIR, page 1503-1512, New York, NY, USA. Association for Computing Machinery.
Sun, Yingzhuo Qian, Zhenghao Liu, Chenyan Xiong, Kaitao Zhang, Jie Bao, Zhiyuan Liu, and Paul Ben- nett. 2021. Few-shot text ranking with meta adapted synthetic weak supervision. Proceedings of ACL, pages 9030-5043, Online. Association for Computa- tional Linguistics.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2022. Adversarial retriever-ranker for dense text retrieval. In ICLR.
Yu Sun, Shuohuan Wang, Yukun Li. Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language un- derstanding. In Proceedings of AAAI, volume 34, pages 9965-8975.


Reviews:
[{"review_id": "TlRvoprgev", "reviewer": "Reviewer_1aty", "summary": "", "questions": "I would like to thank the authors for this work. I appreciate how the authors carefully highlight the limitation of the dual cross-encoders used for dense retrieval and then provide a simple but well-thought approach to counter them. The results are encouraging! I have the following comments and questions about the work.\n\nA. The process of generating synthetic queries is unclear to me. From the appendix, I understand that the authors used the queries provided by Nogueira and Lin (2019) for MS-MARCO. It'd be beneficial to include an overview of the approach they adopt. It was unclear how the authors fine-tuned a T5-based model to obtain the queries for documents in the BEIR benchmark. Furthermore, could the authors provide qualitative examples for the synthetic queries from each of the groups in curriculum learning – what do the queries that have low relevance with gold query (q) look like, and how do they compare to those with high relevance with q? \n\nB. The argument about no delay in retrieval time is unclear (mentioned in lines 554-557 and other places earlier). The authors propose an approach that generates multiple queries for each document, which are aggregated before final retrieval. Does generating multiple synthetic queries not add latency to the retrieval process? Why do the authors claim no additional latency?\n\nC. I would request authors to note which ROUGE implementation they used as a recent ACL'23 work demonstrates the discrepancy and bugs in ROUGE-based evaluation for an overwhelming number of published papers. Please see Grusky et al. (ACL 2023: https://aclanthology.org/2023.acl-long.107.pdf) for more details and pointers to ensure the ROUGE implementation they use is correct. \n\nD. What do 144 and 32 mean in lines 746-747? Are the maximum lengths for documents and queries set based on tokens, words, or character counts?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful and valuable comments. \n\n> *Q1*: About query generation. \n\nWe apologize for not describing this part clearly.\nThe query generation model is based on the sequence-to-sequence pipeline, which is initiliazed with the T5-base model. Following Nogueira and Lin (https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf), we feed the passage into the generation model and train the model to generate the query. For fair comparisons to DCE (https://arxiv.org/abs/2208.04232), we use the queries released by Nogueira and Lin (https://git.uwaterloo.ca/jimmylin/doc2query-data/raw/master/T5-passage/predicted_queries_topk_sampling.zip) to expand the passages of MS-MARCO. \nIn parallel, for BEIR, we use the T5-base model fine-tuned on MS-MARCO to generate 5 queries for each document within the BEIR benchmark. During inference, we use the top-k (k=10) sampling and set the max output length to 64 to generate queries. We will provide a clearer description about query generation in the final version. \n\nFor ease of understanding, we provide an example of the synthetic queries from each of the groups in curriculum learning. \nDuring training, suppose the gold query is \"who proposed the geocentric theory\". The synthetic queries in different groups are shown as follows (for simplicity, we only show one query for each group): \\\nGroup 1: who proposed the geocentric model \\\nGroup 2: who discovered the geocentric system of plato\\\nGroup 3: what was plato's geocentric theory of the planets\\\nGroup 4: what model did plato use to determine retrograde motion\n\nThe queries from group 1 are most relevant to the gold query, while the queries from group 4 are the least relevant to the gold query. \n\n\n> *Q2*:  About no delay in retrieval.\n\nWe apologize for not stating this clearly. We will provide a clearer description of this issue in the future version. What we are referring to is that before using the capstone model (our model), we need to generate queries for each document using the generation model, and this step is quite time-consuming. However, since we can pre-generate queries, the capstone model does not introduce any additional retrieval latency compared to the vanilla dense retrieval models at the retrieval stage. \n\n\n> *Q3*:  About ROUGE-L\n\nThank you for your thoughtful reminder. We utilize the ROUGE package from the HuggingFace Hub to compute ROUGE-L (https://huggingface.co/spaces/evaluate-metric/rouge). This package is a wrapper around Google Research's reimplementation of ROUGE (https://github.com/google-research/google-research/tree/master/rouge). We intend to incorporate this information into the implementation details for the future version. It is worth noting that we only utilize ROUGE-L to rank all candidate queries, rather than employing it as an evaluation metric. Therefore, the reproducibility issues highlighted in the ACL paper due to variations in the calculation of ROUGE are not applicable here.\n\n\n> *Q4*: About the maximum lengths.\n\nIn lines 746-747, the maximum lengths for documents and queries are set based on tokens, where the inputs are tokenized with the huggingface tokenizers. We will provide a clearer description in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach is simple and intuitive, and the authors clearly articulate the motivation behind the approach. Overall, I found the contributions of this work to be well-motivated\n- The results demonstrate the effectiveness of the proposed approach over several baselines on multiple datasets", "reasons_to_reject": "- The approach used to construct the synthetic queries is unclear to me. The authors defer the reader to information in the appendix, which doesn't provide sufficient information. The construction of synthetic queries is one of the key steps of the proposed approach\n- The authors argue that the existing dual cross-encoding approaches tend to disregard in-batch negatives while learning. This would have been a more compelling argument if the authors could provide empirical evidence. For instance, do the visualization of weights provide evidence for heterogeneity between training- and inference times?", "questions_to_authors": "I would like to thank the authors for this work. I appreciate how the authors carefully highlight the limitation of the dual cross-encoders used for dense retrieval and then provide a simple but well-thought approach to counter them. The results are encouraging! I have the following comments and questions about the work.\n\nA. The process of generating synthetic queries is unclear to me. From the appendix, I understand that the authors used the queries provided by Nogueira and Lin (2019) for MS-MARCO. It'd be beneficial to include an overview of the approach they adopt. It was unclear how the authors fine-tuned a T5-based model to obtain the queries for documents in the BEIR benchmark. Furthermore, could the authors provide qualitative examples for the synthetic queries from each of the groups in curriculum learning – what do the queries that have low relevance with gold query (q) look like, and how do they compare to those with high relevance with q? \n\nB. The argument about no delay in retrieval time is unclear (mentioned in lines 554-557 and other places earlier). The authors propose an approach that generates multiple queries for each document, which are aggregated before final retrieval. Does generating multiple synthetic queries not add latency to the retrieval process? Why do the authors claim no additional latency?\n\nC. I would request authors to note which ROUGE implementation they used as a recent ACL'23 work demonstrates the discrepancy and bugs in ROUGE-based evaluation for an overwhelming number of published papers. Please see Grusky et al. (ACL 2023: https://aclanthology.org/2023.acl-long.107.pdf) for more details and pointers to ensure the ROUGE implementation they use is correct. \n\nD. What do 144 and 32 mean in lines 746-747? Are the maximum lengths for documents and queries set based on tokens, words, or character counts?", "paper_topic_and_main_contributions": "The paper proposes a new approach for representing documents and queries to enable dense retrieval. The approach aims to generate synthetic queries for retrieving documents so that the model learns better representations of documents. This is done by discouraging the modeling of trivial associations between queries and documents within a batch, which the authors argue that current dual cross-encoders tend to do. Overall, the approach is simple and intuitive, and results on two existing datasets demonstrate the approach's effectiveness over several competitive baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TDOVsJ3ktM", "reviewer": "Reviewer_4QHG", "summary": "", "questions": "1. Are the results statistically significant?\n2. Do you have any idea why your reproduced coCondenser results are different then the results reported in their paper?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and invaluable feedback. \n\n> *Q1*: About the evaluation results.\n\nWe conducted t-test comparisons between our model and baselines with p-values<0.05, which indicates our results are statistically significant.  \n\nWe conducted experiments adhering closely to the configuration in coCondenser’s paper, including the utilization of AdamW optimizer with a learning rate of 5e-6, a linear learning rate schedule, and a batch size of 64 for 3 epochs. However, our replication yielded a result of 37.9, whereas the original paper reported 38.2. This discrepancy might stem from variances in experiment random seeds. We set our random seed to 42, whereas they did not mention the selected random seed in their paper. \n\nIn the ablation study, we did not use the pre-trained retrieval model, coCondenser, with the aim of directly assessing the effectiveness of our approach. As depicted in Table 5, our method demonstrates its effectiveness even when applied to coCondenser.\n\n\n> *Q2*:  About query generation.\n\nWe apologize for not describing this part clearly.\nThe query generation model is based on the sequence-to-sequence pipeline, which is initiliazed with the T5-base model. Following Nogueira and Lin (https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf), we feed the passage into the generation model and train the model to generate the query. For fair comparisons to DCE (https://arxiv.org/abs/2208.04232), we use the queries released by Nogueira and Lin (https://git.uwaterloo.ca/jimmylin/doc2query-data/raw/master/T5-passage/predicted_queries_topk_sampling.zip) to expand the passages of MS-MARCO. \nIn parallel, for BEIR, we use the T5-base model fine-tuned on MS-MARCO to generate 5 queries for each document within the BEIR benchmark. During inference, we use the top-k (k=10) sampling and set the max output length to 64 to generate queries. We will provide a clearer description about query generation in the final version. \n\nUsing the query generation model to generate queries can be a time-consuming process. For instance, generating queries for the BEIR dataset took approximately 15 days using 8 V100 GPUs. In light of restricted computational resources, we didn't explore impacts of different generation strategies. We leave them as future work.\n\n> *Q3*:  About typos \n\nWe will fix typos and grammar issues in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is well written, figures are informative\n- Novel method for training dual-cross-encoders\n- Improvement upon baseline training procedure", "reasons_to_reject": "- I have some concerns regarding the evaluation process: first, are the results statistically significant? specifically, the interesting baseline is of course coCondenser- is CAPSTONE statistically significant better than it? Further, coCondenser’s paper reported MRR@10 of 38.2 on MS MARCO and in this paper the reported score is 37.9, what is the reason for this difference? Finally, it is unclear to me why the main results are based on coCondenser and the ablation study is performed on a vanilla DPR model.\n\n- Authors do not discuss the queries generation process, which in my opinion is a crucial part of the proposed training procedure. I think it can be interesting to study and get some insights into how generating queries differently effects the training process. For most use cases, pre-generated queries will not be available and in order to use the proposed method these queries will need to be generated.", "questions_to_authors": "1. Are the results statistically significant?\n2. Do you have any idea why your reproduced coCondenser results are different then the results reported in their paper?", "paper_topic_and_main_contributions": "The paper deals with the training procedure of dual-cross-encoders for dense retrieval. It builds on the architecture proposed in Li et al (2022) where the dual encoder is composed of a query encoder and a document+query (aka expanded document) encoder. The idea is to enable the document encoder to learn complex relations between query and document (like a cross encoder) while keeping the dual encoder’s reasonable inference time. The novelty over Li et al (2022) includes the following. First, the authors suggest generating queries for the expanded documents during training (and not only during inference), instead of just using the train-set positive queries during training and generating a query only in inference time. Second, In early stages of training authors suggest using queries that are different from the actual positive query and as training proceeds they use queries that are more and more similar to the positive train-set query and 3. To limit the degradation in inference time (caused by the fact that multiple queries are generated for each document in the corpus), they average the different embeddings generated for each document and use only a single embedding per document.\nExperiments on several datasets (MS MARCO, TREC 19+20, BEIR) show the method proposed improves upon the baseline.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 358: “As shown in Table 3, CAPSTONE performs much than all baselines in the average performance” - much what? :)\n\nline 407: you say you show results on “top-k” and “bottom-k” but the table only shows results for and top-1 and bottom-1"}}, {"review_id": "zaZAs5t5mA", "reviewer": "Reviewer_p4HK", "summary": "", "questions": "How do you generate document-related queries? And how do you guarantee the diversity of generated queries?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful and valuable comments. \n\n> *Q1*: Related work about query generation.\n\nWe will include the omitted related work concerning query generation in the final version.\n\n> *Q2*: About query generation. \n\nWe apologize for not describing this part clearly.\nThe query generation model is based on the sequence-to-sequence pipeline, which is initiliazed with the T5-base model. Following Nogueira and Lin (https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf), we feed the passage into the generation model and train the model to generate the query. For fair comparisons to DCE (https://arxiv.org/abs/2208.04232), we use the queries released by Nogueira and Lin (https://git.uwaterloo.ca/jimmylin/doc2query-data/raw/master/T5-passage/predicted_queries_topk_sampling.zip) to expand the passages of MS-MARCO. \nIn parallel, for BEIR, we use the T5-base model fine-tuned on MS-MARCO to generate 5 queries for each document within the BEIR benchmark. During inference, we use the top-k (k=10) sampling and set the max output length to 64 to generate queries. We will provide a clearer description about query generation in the final version. \n\nUsing the query generation model to generate queries can be a time-consuming process. For instance, generating queries for the BEIR dataset took approximately 15 days using 8 V100 GPUs. In light of restricted computational resources, we undertook separate optimization for both the generation model and the retrieval model. In the same vein, we didn't explore impacts of different generation strategies and ways to enhance query diversity. We leave them as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a document expansion method, which targets generating document-related queries for document expansion. The idea is intuitive.\n2. The experimental results on MS MARCO and BEIR show the effectiveness of the proposed method.\n3. The proposed method can alleviate the exploration bias between training and inference, which derives from query generation.", "reasons_to_reject": "1. Some related work, such as query generation and negative sampling should be discussed.\n2. The novelty of this paper aims to alleviate the exploration bias between training and inference due to query generation. However, the query generation method is not jointly optimized. ", "questions_to_authors": "How do you generate document-related queries? And how do you guarantee the diversity of generated queries?", "paper_topic_and_main_contributions": "The paper presents a study on the shortcomings of the dual-encoder architecture in dense retrieval systems. While the dual-encoder has become the standard approach, it falls short of capturing the interactions between queries and documents effectively. This paper proposes CAPSTONE, which expands the document with a real query for better representing the documents. However, the training and inference processes may contain exploration bias, which causes by the quality of generated queries. To overcome the challenge, the authors propose a novel curriculum sampling strategy. During training, pseudo queries are introduced to progressively enhance the relevance between the generated query and the real query. This strategy allows the retrieval model to learn and extend its attention from focusing solely on the document to jointly considering both the document and query.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {"Missing_References": "Query generation:\n[1] Zero-shot neural passage retrieval via domain-targeted synthetic question generation.\n[2] Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision.\n[3] Promptagator: Few-shot Dense Retrieval From 8 Examples.\n"}}]

Author rebuttal:

Meta-review: All the reviewers have agreed that this paper has tackles a pain in dual-cross-encoders.

However, some concerns have also been raised such as some important details are lacking and generally during the rebuttal period, the authors have provided these details and addressed these concerns.

I recommend the authors incorporate these details to the paper.
Decision: Accept-Main