Query-as-context Pre-training for Dense Passage Retrieval
Authors: Xing W
Guangyuan Ma
Wanhui Qian
Zijia Lin
Songlin Hu
Venue: EMNLP 2023 Main (2023)
Keywords: Query prediction
Dense passage retrieval
Pre-training

Abstract: Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training. These methods simply consider two passages from the same document to be relevant, without taking into account the potential negative impacts of weakly correlated pairs. Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue. Query-as-context pre-training assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair. These passage-query pairs are then used in contrastive or generative context-supervised pre-training. The pre-trained models are evaluated on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. Experimental results show that query-as-context pre-training brings considerable gains for retrieval performances, demonstrating its effectiveness and efficiency.

Query-as-context Pre-training for Dense Passage Retrieval
Xing 2u Guangyuan Ma Wanhui Qian
Zijia Lin", and Songlin Hul
Institute of Information Engineering, Chinese Academy of Sciences
School of Cyber Security, University of Chinese Academy of Sciences Du Xiaoman Financial, Kuaishou Technology
(wuxing, maguangyuan, husonglin| @iie.ac.en, qianwanhui @ @uxiaoman.ccom linzijia07 @tsinghua.org.cn
Abstract
A good day at Pipeline means encounter with fellow surfers who can be as friendly pitbulls with migraines, and waves that can shatter boards into kindling. And then there's the reef. At Pipeline there can be 10 foot waves blasting over just three feet of water, so if you fall the reef get caught inside the break, you're lucky if you come out merely sliced up.
Passage 1.
Recently, methods have been developed to im- prove the performance of dense passage re- trieval by using context-superviser pre-training. These methods simply consider two passages from the same document to relevant, with- out taking into account the potential negative impacts of weakly correlated pairs. Thus, this paper proposes query-as-contex pre-training, a simple yet effective pre- gre-trainin technique to alleviate the issue. Query-ase auerr--s--ontextt pre- training assumes that the query derived from a passage is more likely to be relevant to that passage forms a passage-query pair. These passage-query pairs are then used in contrastive or generative context-supervise
Passage This idyll still be found the island of Lanai. Formerly a pincapple plantation, Lanai is almost entirely owned by billionaire entrepreneur David Murdock, and its only notable commercial hub square of Cook pine trees.
tiny village built around
Figure I: An example of low-relevance passages within a document from the MS-MARCO corpus. The two passages are weakly correlated in content.
pre-training. The pre-trained models are evaluated on large- scale passage retrieval benchmarks and out-of- domain zero-shot benchmarks. Experimental results show that amery-as-conters pre-training brings considerable gains for retrieval perfor- mances, demonstrating its effectiveness ef- ficiency.
Wang al., 2022) use PLMs to encode queries and passages into a shared semantic space. The se- mantic relationships between query and passage representations are then measured by dot prod- uet or cosine similarities. Pre-training and fine- tuning techniques have been developed to improve the performance of dense retrieval models. Pre- training processes for dense retrieval aim im- prove the text representation modeling ability of the encoder through auxiliary self-supervised or context-supervised tasks.
1 Introduction
Passage retrieval is the process retrieving rele- vant passages from a large corpus in response to a query, which is useful in a variety of downstream applications such as web search (Fan et al., 2021; Guo et al., 2022; Lin et al., 2021a), question an- swering (Karpukhin et al., 2020: Lee et al., 2020; Zhu et al., 2021) and dialogue systems (Gao et al., 2022a; Yu al., 2021). The success of pre-trained language models (PLMs) (Devlin et al., 2018; Liu et al., 2019) has led to the development of more powerful PLMd- dense and sparse passage re- trieval approaches.
context-supervised pre-training (Gao and Callan, 2021b; Wu et al., 2022) assumes that two passages' within the same document contextual or related each other and can therefore be used for contrastive learning or contextual decoding. However, context-supervised pre-training ignores the fact that the passages within a document may be weakly related or even irrelevant in many cases. As shown in Figure 1, two passages within document from the MS-MARCO corpus (Nguyen et al., 2016) are not directly related
PLM-based dense retrieval methods (Xiong et al., 2020; Lu et al., 2021: Hofstätter et al., 2021; Gao and Callan, 2021b; Ren et al., 2021b; Ma al., 2022; Liu and Shao, 2022; Wu et al., 2022:
"The first two authors contributed equally to this work. *Corresponding authors.
'A passage refers to long text span consisting of consee- utive sentences within a much longer document.

---

passage/ /p
query
Query Generation via TS
Pre- Pre-Training
Pre- Pre-Training
qp pair
document
(1) Context Context-superviser supervised Pre-training
(2) Query-as Query-as-context Pre-training
Figure 2: A comparison of context-superviset pre- pre-training and enery-as-contedt pre-training.
in content. According to our statistic results via human annotation, only 35.5% of passage pairs in the training data of coCondenser (Gao and Callan, 2022) have high correlation. For statistical details, please refer to Appendix A. These weakly correlated irrelevant passages do not align with the assumptions which context-superviied pre-training is based, and likely to be detrimental coptext--upervised pre-training.
two randomly selected passages from a document, the query-as-context method generates a training pair by combining a passage with a predicted query, as illustrated in Figure 2.
There are several advantages to using the query- as-context setting. Firstly, the query is more likely to be related to the passage because it is gener- ated from the passage. Additionally, the of passage-query pairs in supervised downstream re- trieval training is consistent with using passage- query pairs in pre-training, which helps to bridge the gap between the two processes. Finally, since the passage-query pairs are generally shorter than the previously used passage- paaagge pairs, it speeds up the pre-training process and reduces the training overhead.
In contrast to dense retrieval, sparse retrieval is based on the "bag-of-wordss assumption and rep- resents passages and queries as sparse term-based vectors. PLM-based sparse retrieval (Nogueira and Lin, 2019; Dai and Callan, 2019; Mao et al., 2020; Bai et al., 2020; Formal et al., 20211 b,a; Mallia et al., 2021: Shen al., 2022) uses PLM to improve sparse vectors. One representative technique is Query Prediction (Nogueira and Lin. 2019). which predicts set of relevant queries to enrich the pas- sage's content and thus alleviates the mismatch problem. Query prediction has been shown to be effective in sparse retrieval, but has not yet been ex- plored in the context of dense retrieval, especially in the pre-training process. This raises the ques- tion of whether the query prediction technique can benefit the pre-training process tailored for dense retrieval.
To verify the effectiveness of proposed query-as-context pre-training, we conduct exper- iments on large-scale web search benchmarks: MS-MARCO Passage Ranking (Nguyen et al., 2016), TREC Deep Learning (DL) Track 2019 (Craswell et al., 2020a) and Track 2020 (Craswell et al., 2020b). We also evaluate quconex--comtexi pre-trained models the BEIR (Thakur al.. 2021) benchmark with large set of out-of-domain datasets. Experimental results show that query-as- context achieves considerable gains over compet- ing baselines.
The observation that predicted queries align bet- ter with the content of passage in our statistical analyses (see Appendix A) suggests that query pre- diction could promising way to alleviate the issue weakly correlated passages for context- supervised pre-training. Thus, this paper focuses on exploring query prediction techniques to im- prove context-supervised pre-training methods for dense retrieval. Our proposed method, termed query-as-context pre-training, assumes that query derived from a passage (using a generative model like T5) more likely to be a relevant context to the passage. In contrast to the previous context- supervised methods that create a training pair using
Our contributions can be summarized as follows:
We reveal the previously ignored issue of weakly correlated passage pairs during context-supervised pretraining.
We propose query-as-contl pre-training, simple yet effective pre-training technique alleviate the issue above.
Experiments show that query-as-comme pre- training brings considerable gains and mean- while speeds up pre-training.

---

2 Preliminary: Context-supervised Pre-training
considered irrelevant and form negative pair. These pairs constitute mini-batches for contrastive learning. A common approach for generating an embedding representation a passage is to use the hidden states the [CLS]] position in the last layer of the encoder, i.c., hig Thus, the embedding representations of passages x and y are big (x) and h6 (y), simplified hx and hy. Then, for mini- batch B, the contrastive learning objective w.r. x formulated as:
In this section, we begin by providing an overview of the pre-training corpus. Subsequently, we de- scribe the masked language modeling task, which serves as foundational task of pre-training. Fi- nally, we present two representative contrastive and generative context-supervised pre-training meth- ods, on which our proposed query-as context coentet will be applied.
exp(sim (hx, hy) /T) Σ exp(sim (hx, /T) (5) where T is temperature hyper-parameter and sim (.) is the dot product similarity function.
C = log
Pre-training Corpus Given set of documents, we randomly extract pairs of passages from each document, which forms training corpus fol- lows:
KEB
{ ((x0 yo), Nmm)
(1)
An additional auxiliary decoder is also appended to the encoder, which consists of layers of trans- formers. The auxiliary decoder takes the concatena- tion of the [CLS] representation from the L-th layer, i.e., h and the token representations from the en- coder's s M-th (e.g. 6-th) layer, i.c., { ..... h }. as inputs. Similar MLM, the output of the auxil- iary decoder's 's last layer is then used perform an auxiliary MLM pre-training.
where {x) a pair of passages from the same document.
Masked Language Modeling (MLM) Formally, given a passage with tokens, a special token [CLS] is added to the beginning of the passage, resulting in
[mg, T1; )
Σ CE(O(H)) ), (6)
m
where zn represents the [CLS] token. Then, a cer- tain percentage of positions are randomly selected "mask positions" (mask_ _os) and are replaced with special token [MASK] random token. The masked passage is then passed through text encoder, which commonly consists of L layers of transformer blocks. For the /-th transformer layer the encoder, its outputs are the hidden states of the layer
iCmask_pos
Finally, the total loss of coCondenser is:
L= Lytm
Lave
Loco
For more details, please refer to (Gao and Callan, 2021b).
2.2 CoT-MAE
CoT-MAE (Wu et al., 2022) is a representative generative context-supervised method that uses an asymmetric encoder-decoder structure, with a deep encoder of L layers and a shallow decoder of N layers. It performs MLM training both the en- coder and the decoder simultaneously, For a pair of passages (x, y), suppose is fed into the encoder side and y is fed into the decoder side.
= - 1/ k3
(3)
The output of the last layer is then used to calculate the MLM's target loss
mylm
Σ
CE(((H)),
(4)
iCmosk_pos
where CE is short for cross entropy function and @ is projection of the corresponding hidden states of x; to vocabulary distribution.
On the encoder side, X is reconstructed using only the unmasked tokens in the passage, similar to BERT's MLM process, but with a higher mask rate (e.g. 30%). On the decoder side, is reconstructed using both its unmasked tokens and the contextual passage x. The decoder takes the sentence embed- ding of i.e.. hxx and the word repesentations of masked y as input, which are concatenated as:
2.1 coCondenser
coCondenser (Gao and Callan, 2021b) is repre- sentative contrastive context-supervised method, For coCondenser, two passages from document are considered relevant and form a positive pair, while two passages from different documents are
d {/xx V1, yn
(8)

---

The concatenation d is then passed through the N layers of Transformer blocks, and the hidden states of layer is formulated as:
BM negatives
mined negatives
Retriever
Retriever
d [d di.
d
intiiaiize
inttellzee
Finotune Pipoline
Query-as--ontele Pre- Trained Model
The outputs of the last layer in decoder then used for LM pre-trame pre-training, with the loss defined as:
Figure 3: Illustration of the finettuing pipeline. The guete-as-contex pre-trained model is used to initialize the dual-encoder retrievers.
Cota_mim mlmm
Σ ),(o(d)), (10)
iEmask_poo
3.2 Fine-tuning
The subseript elx denotes the process is context- supervised. Then, we add the losses from both the encoder and the decoder get the final loss:
We fine-tune on the downstream retrieval tasks verify the effectiveness of pre-training. Follow- ing (Gao and Callan, 2021b; Wu et al., 2022), the fine-tuning process on MS-MARCO is based on a two-stage pipeline with hard negative mining (Gao et al., 2022b), as depicted in Figure 3. This pro- cess involves two stages of training, bi-encoder triever and bi-encoder retriever which are both initialized with the query-as-context pre-trained models. The retrievers are trained with contrastive learning on the manually annotated passage-query pairs. For a manually annotated passage-query pair (p+ (tt.g*),, the representations of the passage and the query form a positive example (h hg+)), When training retriever 1, for query g the negative sam- ples {pT} include in-batch negative passages and BM25 mined hard negative passages. When train- ing retriever hard negatives are also mined using well-trained retriever and combined with the other negative passages to create the negative sam- ples {p }. Both stages are optimized using the InfoNCE loss.
L = mmmmm +
Lecta_mmm
(11)
For more details, please refer to (Wu et al., 2022).
3 Query-as-context Pre-training
In this section, first introduce the details of query-as-coniext pre-training, and then introduce the fine-tuning process the pre-trained models on the retrieval tasks.
3.1 Pre-training
Pre-training is conducted on large scale of docu- ments without annotations. For each document D, we extract set of passages with a maximum length, {Xo, X1, }..). Following (Nogueira and Lin, 2019). for each passage x,, we use a fine-tuned T5 model for generating queries. We apply nucleus sampling with topp.00.55 and =2p to produce multiple queries for promoting diversity.
C - log exp(sim (hyt Mypt /T)
Specially, each passage x; will be fed into the fine-tuned T5 model, and generate C candidate queries, {q:))=1 During training. we will ran- domly select one of the candidate queries to use as the context for the passage:
exp(sim (hat, hp) /T) pE(pt.p-)
(12)
where is temperature hyper-parameter fixed 1 and sim (,) is dot product similarity function. Following (Thakur et al., 2021), we train the retriever with MS-MARCO negatives for the out- of-domain evaluation BEIR benchmarks
= (umple((qu,))_1)
The passage and sampled query form a training pair yih, which can be used to replace the original pair used in Equation 1. Specifically, the passage-query pair directly used for contrastive pre-training of coCondenser. For COT-MAE, fed the passage into the encoder, and query into the decoder for generative pre-training. Model imple- mentations for coCondenser and CoT-MAE have been introduced Section 2.1 and 2.2.
Experiment
In this section, provide details on the pre- training and fine-tuning processes. Then we present the experimental results.
htttpss//sbert.nnt/dttaaae asmarco-harr--neativve jsonl.

---

4.1 Pre-training
for evaluation following previous work (Gao and Callan, 2021b; Wu et al., 2022). We evaluate our performance on MS-MARCO using MRR 10, Re- call 50, and Recall IK.
Query-as-contes Dataset Following (Gao and Callan, 2021b; Wu et al., 2022), the pre-training dataset is collected from the MS-MARCO passages corpus, which contains 3.2 million documents. We use NLTK to split each document into sentences, and group these sentences into passages of more than 144 consecutive tokens. For each passage, we generate candidate queries via public T5 model During pre-training, we select a batch of pas- sages at each step and randomly choose a candidate query as context for each passage to form a rele- vant pair.
TREC Deep Learning (DL) (Craswell et al.. 2020a,b) tracks provide test sets with more elab- orate annotations to evaluate the real capacity of ranking models. We evaluate the 2019 and 2020 test sets. The 2019 test set contains 43 annotated queries and the 2020 test set contains 54 annotated queries. We evaluate our performance on TREC with NDCG@ 10.
Implementation
We reuse a widely adopted eval- uation pipeline (Gao and Callan, 2021b; Wu et al., 2022; Gao et al., 2022b), with a common random seed (42) to support reproducibility. Note that, as we focus on improving the pre-training tech- nique, we do NOT use any enhanced methods, such as distillation from a strong re-ranker (Ren et al., 2021b; Santhanam et al., 2021) or multi-vector rep- resentation (Khattab and Zaharia, 2020), which can lead further improvements. The fine-tuning is only trained on the MS-MARCO dataset and eval- uated on the dev set and TREC DL 2019/2021 test sets. It's trained on 8 Tesla V100 GPUs using the Adam W optimizer with learning rate of 2e-5, a global batch size of 64, and weight decay of 0.01. The passage length is also set to 144, the negative depth is set 200 and the number of negative pas- sages for one query in the fine-tuning iteration is 15.
Model Implementation Following (Wu al., 2022), the encoder for CoT-MAE is initialized with a pre-trained 12-layer BERT-base model, while the decoder initialized from scratch. We pre- train the model using the AdamW optimizer for a maximum of 50k steps, with a learning rate of 4e-4, a batch size of 16k, and linear schedule with a warmup ratio of 0.1. We use 16 Tesla V100 GPUs to train the model for 60 hours, and then discard the decoder, leaving only the encoder for fine-tuning. Following (Gao and Callan, 2021b), the encoder for coCondenser initialized from a pre-trained 12-layer Condenser (Gao and Callan, 2021a) model. The training is conducted on Tesla V100 GPUs for 120,000 steps over 90 hours using the AdamW optimizer with learning rate of Ic-4, a global batch size of 2k, and weight decay of 0.01. Once the pre-training is finished, the Condenser head is discarded, resulting in model with the same architecture as BERT buse for fine-tuning.
4.3 Baselines
Our bascline methods include the sparse retrieval method and the dense retrieval method, as shown in Table 1. Results of sparse retrieval baselines are mainly from (Qu et al., 2020), including BM25, docTáuuery (Nogueira and Lin, 2019), DeepCT (Dai and Callan, 2019) and GAR (Mao et al., 2020). Results of dense retrieval baselines are mainly from (Gao and Callan, 2021 bb; Liu and Shao, 2022: Ren et al., 2021b; Ma al., 2022), including ANCE (Xiong al., 2020), SEED (Lu al., 2021), TAS-B (Hofstätter et al., 2021), RetroMAE (Liu and Shao, 2022), SimLM (Wang et al., 2022) and etc. We compare the query-as -sceteet performances with their baselines on both retriever 1 and retriever 2.
4.2 Fine-tuning
Datasets and Evaluation We fine-tune the pre- trained coCondenser and CoT-MAE on MS- MARCO passage ranking (Nguyen et al., 2016), TREC Deep Learning (DL) Track 2019 (Craswell et al., 2020a) and 2020 (Craswell et al., 2020b) tasks for evaluation.
MS-MARCO (Nguyen al., 2016) is bench- mark dataset that contains real user queries col- lected from Bing search and web pages, and in- cludes approximately 8.8 million passages in to- tal. The training set consists of around 500,000 annotated query-document pairs, while the dev set contains 6,980 annotated queries. Since the test set is not publicly available, the dev set is used
4.4 Main Results
As shown in Table 1, the results demonstrate that query-as-context pre-training leads to improved
"https://hugingf aacpggggingaaee..ooo http.////occuuryaaalllitiit prefix-t5-base-vi

---

MS-MARCO MRR@ R@50
TREC DL
TREC DL NDCG@
Model
R@ Ik
NDCG@
Sparse retrieval
BM25
18.7
59.2
85.7
51.2
47.7
DeepCT (Dai and Callan, 2019)
24.3
69.0
91.0
57.2
dquery (Nogueira and Lin, 2019)
21.5
64.4
89.1
64.2
Dense retrieval
NPRINC (Lu et al., 2020)
31.1
97.7
ANCE (Xiong et al., 2020)
33.0
95.9
64.5
64.6
SEED (Lu et al., 2021)
33.9
96.1
TAS-B (Hofstätter ct al., 2021)
34.0
97.5
71,2
69.3
COIL (Gao et al., 2021)
35.5
96.3
70.4
ColBERT (Khattab and Zaharia, 2020)
36.0
82,9
96.8
COSTA (Ma et al., 2022)
36.6
84.1
97.3
67.8
Condenser (Gao and Callan, 2021a)
36.6
97.4
69.8
66.5
RocketQA (Qu al., 2020)
37.0
85.5
97.9
PAIR (Ren et al., 2021a)
37.9
86.4
98.2
SimLM (Wang al., 2022)
39.1
98.6
71.4
69.7
RetroMAE (liiu and Shao, 2022)
39.3
98.5
68.1
70.6
LED (Zhang et al., 2022a)
39.6
86.6
98.3
70.5
67.9
coCondenser (Gao and Callan, 2021b)
38.2
86.5
98.4
71.7
68.4
coCondenser (02)) retriever w/ cuert--s-coontex( (120K) retriever
37.0
86.0
98.5
68.2
68.8
37.4
87.3
98.6
69.2
coCondenser (1 (120K) retriever 2
38.8
87.8
98.8
71.1
68.4
w/ query-as-coontext (120K) retriever
39.4
88.6
99.0
73.1
71.8
CoT-MAE (Wu al., 2022)
39.4
87.0
98.7
70.9
70.4
CoT-MAR (50K)- retriever
37.2
85.7
98.2
65.7
66.5
w/ query-as- context (50K) retriever
38.6
87.7
98.6
67.7
67.8
CoT-MAR (50K) retriever 2
38.8
87.3
98.6
70.7
69.7
w/ query-as-connext (50K) retriever 2
40.2
88.8
98.8
71.5
72.7
Table I: Main results on MS- MS-MARCO passage ranking and TREC DL datasets. denotes our reproduction using publicly available codes. The score that is better in comparison is marked in bold.
performance.
pared to the original paper. However, when query- as-context pre-training applied, there an obvi- ous improvement of 1.4pp on MRR@ 10, reaching 40.2. Even compared to the 1200k model's per- formance in the original paper, we still achieve non-trivial improvement of 0.8pp. To the best of our knowledge, this is the new state-of-the-as result for single vector pre-trained (not reranker- distilled) dense retriever. On both TREC DL 19 and 20 test sets, there are improvements of 0.8pp on DL 19 and 3pp on DL 20. In addition, qucry-as- context pre-trate tre-training also improves the MRR 10, R @ 50, and R @ 1k scores of retriever 1.
coCondenser When reproducing coCondenser, the pre-training steps extend to 10k steps. The main evaluation metric, MRR @ 10 on the MS- MARCO passage ranking dataset, of retriever improves by 0.6pp compared to the original pa- per(Gao and Callan, 2021b). When query-as- context pre-training is used, there is further im- provement of 0.6pp on MRR@ @ On both TREC DL 19 and 20 test sets, there are improvements of 2pp on DL 19 and 3.4pp on DL 20. addi- tion, query-as-context pre-training also improves the MRR @ 10 and R@5 50 scores of retriever 1.
Overall, the query-as-context pre-training ap- proach is effective, improving both contrastive and generative context-supervised pre-training. This is due
When reproducing CoT-MAE, for ef- ficiency, adopt much larger batch size than in (Wu et al., 2022), which allows us reduce the number of training steps from 1200k 50k. This results faster training, but somehow lower perfor- mance on the AS-MARCO MRR 10 metric com-
COT-MAE
main reasons: (1) Pre-trained models can provide better parameters initialization for both retriever and retriever 2: (2) A better retriever can used to mine more effective hard negatives,

---

coCondenser w/o
CoT-MAE
from 1 to 5 slightly improves performance on the MS-MARCO dataset and leads to good improve- ment on the TREC DL 19 and 20 test sets. For COT-MAF, using queries lead to an increase on the MS- MS-MARCO dataset and TREC DL 20 test set, while slight performance decrease in the TREC for coCondenser, increasing the number of queries
Dataset
w/
tree-covid
0.632
0.703
0.646
0.665
nfcorpus
0.333
0.330
0.319
0.340
nq
0.531
0.548
0.513
0.546
hotpotga
0.538
0.583
0.512
0.572
fiqa
0.319
0.322
0.288
0.326
arguana
0.389
0.447
0.312
0.416
webis-touche2020
0.213
0.204
0.202
0.212
eqadupstack
0.310
0,341
0.312
0.337
DL 19 test set. However, further increasing the number of candidate queries will generally bring about a decline in performance. A proper num- ber of queries retains their correlation the pas- sages, thus yielding higher performance query- as as-context pre-training.
quora
0.866
0.864
0.781
0.859
dhpedia-entity
0.373
0,386
0.355
0.406
scidocs
0.133
0.145
0.132
0.151
fever 0.728
0,664
0.707
0.688
climate-fever
0.204
0,199
0.173
0.220
scifact
0.599
0.648
0.591
0.642
Average
0.441
0.456
0.417
0.456
5.2 Impact of Mixed Context
Table Out-of-domain evaluation on BEIR benchmark. The score that is better in comparison is marked in bold.
We further explore the effect of mixing the two kinds of contextual pairs, passage-query and passage-passage. In a training step, we randomly choose to use either the passage-query or passage- passage pair as input with the same probability. As shown in Table 4, mixing does not improve the effect for coCondenser and CoT-MAF, despite increasing the diversity of context. The decrease aligns with the human-annotated correlation results in Appendix A. The passage-passage pairs have a higher proportion of low correlation pairs, com- bining passage-query and passage-passage pairs will be less effective than using passage-query pairs alone. This also indicates that for pre-training tai- lored for intensive retrieval, the relevance of train- ing pairs is more crucial than diversity.
which further improves the training of retriever
4.5 Out-of-domain Evaluation
We evaluate the out-of-domain performance of query-as-contex- pre-trained models on the zero- shot benchmark BEIR(Thakur et al., 2021). BEIR benchmark contains different open-domain in- formation retrieval tasks from 18 different datasets. We evaluate the models on the 14 publicly available datasets As shown in the table, both the coCon- denser and the CoT-MAE results show non-trivial improvements most datasets when using query- as as-context pre-training. Specifically, using query- as-context pre-training improves the performance of the coCondenser model on different datasets. The improvement in CoT-MAE is more significant, with notable gains observed on 13 datasets.
Related Works
Dense Retrieval Different techniques have been developed to improve dense retrieval, both in fine- tuning and pre-training stages. In fine-tuning stage, attempts includes mining hard negatives (Xiong et al., 2020; Zhan et al., 2021), late interaction (Khattab and Zaharia, 2020), query clustering (Hof- stätter al., 2021), reranker distillation (Lin et al., 2021b: Santhanam et al., 2021), data augmentation (Qu et al., 2020) and jointly learning (Ren et al., 2021b; Zhang et al., 2022b, 2021) In pre-training stages, attempts are divided into two categories. One category focuses on improving the encoder using auxiliary self-supervised auto-encoding tasks (Lu al., 2021; Gao and Callan, 2021a; Liu and Shao, 2022: Zhou et al., 2022). The other cate- gory proposes passage prediction tasks to resem- ble passage retrieval in pre-training (Chang et al., 2020; Gao and Callan, b021b; Ma et al., 2022).
5 Analyses
In this section, we examine the efficiency advan- tage and analyze the impact of different settings query-as-context pre-training.
5.1 Impact of Generated Query Number
During pre-training, using multiple candidate queries leads to better diversity as each passage is paired with different candidate query in each epoch. Therefore, we explore the effect of the num- ber of generated queries. As shown in Table 3,
'The current state-of-the-art models on the BEIR bench- mark reach higher scores as they are pre-trained on the WIKI dataset. Due to the high cost of pre-training, we directly eval- uate the models pre-trained on the MS-MARCE datasct and leave the exploration on the WIKI dataset in future work.

---

MS-MARCO
TREC DL
TREC
Model
Query Number
Retriever-1 MRR
Retrieven-2 MRR@10
R@
R@50
R@
NDCG@N
NDCG@
37.7
87.6
98.6
39.3
88.5
98.9
72.3
71.1
37.4
87.3
98.6
39,4
88.6
99.0
73.1
71.8
coCondenser
10
37.6
87.0
98.5
39.1
88.5
98.9
71.1
70.9
20
37.7
87.2
98.6
39.4
88.3
99.0
71.5
70.3
38.3
87.4
98.5
39.9
88.7
98.7
71.7
70.8
38.6
87.7
98.6
40.2
88.8
98.8
71.5
72.7
CoT-MAE
10
38.5
87.2
98.6
39.7
88.7
98.8
72.5
71.7
20
38.3
87.5
98.6
39.7
88.5
98.8
72.2
69.9
Table Impact of the number of generated queries. The that better in comparison is marked in bold.
MS-MARCO
TREC
TREC
Model
Mixed
Retriever-1 R@:00
Retriever-2 MRR 10
DL NDCG@
MRR @
RΦ
R@500
R@
NDCG@
X
37.4
87.3
98.6
39.4
88.6
99.0
73.1
71.8
coCondenser
37.4
86.7
98.4
39.1
88.1
98.8
71.2
71.3
X
38.6
87.7
98.6
40.2
88.8
98.8
71.5
72.7
CoT-MAR
36.9
85.6
98.1
38.4
87.1
98.5
72.4
70.0
Table
Effect of mixing passage-query and passage-passagg pairs in pre-training.
The most related methods in this category are (Gao and Callan, 2021b) and (Wu et al., 2022). (Gao and Callan, 2021b) introduces context-supervised contrastive pre-training process, with the hypothe- sis that passages from the same document are closer than those from different documents. (Wu et al., 2022) introduces context-supervised generative masked auto-encoding task via the decoder-side re- construction task assisted by contextual embedding. Our work is on the basis of these two methods.
the fine-tuning process for dense retrieval, while are working the pre-training process.
7 Conclusions
In this work, we propose query-as-context pre- training, simple yet effective technique to allevi- the previously ignored issue of weakly corre- lated pairs during sunexx-supervised pre-training. Extensive experiments well validate its effective- ness and efficiency.
Query Prediction Query Prediction is a tech- nique originally introduced to the IR community to expand passages. It can significantly improve the performance of BM25 by generating additional queries and appending them to passages before building the inverted index (Nogueira and Lin, 2019). Query prediction has also been used to learn better sparse (Mallia et al., 2021) or dense (Li et al.. 2022) representations for documents. sce- narios where data is scarce, query prediction can be used for domain adaptation by generating syn- thetic queries on target domains for model training (Ma et al., 2020). To reduce noise in the generated data, a cross-encoder
Limitations
A passage is more likely to have a high correla- tion with its corresponding generated query than another randomly selected passage from the same document. However, limited by the capabilities of the T5 model. there are still large number of un- related passage-query pairs. We believe that more powerful large language models have the potential to further alleviate this problem, which left our future rescarch.
also be used for pscudo- labeling ( (Wang al., 2021). The most related work ours is (Li al., 2022), which encodes each doc- ument with set of generated pseudo-queries to obtain query-informed document representations. However, (Li et al., 2022) focuses on improving
References
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang. Lifeng Shang. Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020. Sparterm: Learning term-based sparse representation for fast text retrieval. arXiv preprint arXiv-2010.00766.

---

Wei-Cheng Chang, Felix Yu, Yin-Wen Chang, Yim- ing Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-baree large-scale retrieval. arXiv preprint 1tti:220020333333
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022b. Tevatron: An efficient and flexible toolkit for dense retrieval. arXiv preprint arXiv:2203 05765.
Jiafeng Guo, Yingiong Cai, Yixing Fan, Fei Sun, Ruging Zhang, and Xueqi Cheng. 2022. Semantic models for the first-stage retrieval: A comprehensive review. ACM Transactions on Information Systems (TOIS), 40(4):1-42.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020a. Overview of the tree 2019 deep learning track. arXiv preprint arXiv:2003.07820
Nick Craswell, Bhaskar Mitra. Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. 2020b. Overview of the trec 2020 deep learning track. ArXiv, 0oss/200300822.0
Sebastian Hofstääter, Sheng-Chich Lin, Jheng-Hong Yang. Jimmy Lin, and Allan Hanbury. 2021. Ef- ficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pages 113-122.
Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint 11Xiv:190.116637
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint a 100004805
Yixing Fan, Xiaohui Xie, Yingiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li, Ruqing Zhang, Jiafeng Guo, and Yiqun Liu. 2021. Pre-training methods in informa- tion retrieval. arXiv preprint arXiv:21 11.:2111..1853
Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39-48.
Thibault Formal, Carlos Lassance. Benjamin Pi- wowarski, and Stéphane Clinchant. 2021a. Splade v2: Sparse lexical and expansion model for informa- tion retrieval. arXiv preprint arXiv:2109.1008 10066
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2020. Learning dense representations of phrases at scale. arXiv preprint arXiv:2012. 12624.
Thibault Formal. Benjamin Piwowarski. and Stéphane Clinchant. 2021b. l. Splade: Sparse lexical and ex- pansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288 2292.
Zehan Li, Nan Yang, Liang Wang, and Furu Wei. 2022. Learning diverse document representations with deep query interactions for dense retrieval. arXiv preprint arXiv:2208.044 04232..
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021a. Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures on Human Language Technologies, 14(4):1-325.
Jianfeng Gao, Chenyan Xiong, Paul Bennett, and Nick Craswell. 2022a. Neural approaches to con- versational information retrieval. arXiv preprint arXiv:2201.05176.
Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 202 In-batch negatives for knowledge distillation with coutley- teachers for dense retrieval. Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-200 173.
Luyu Gao and Jamie Callan. 2021a. Condenser: a pre-training architecture for dense retrieval. arXiv preprint arXiv:2104.088: 0.8533...
pages 163-
Luyu Gao and Jamie Callan. 2021b. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. arXiv preprint arXiv:2108.0S:: 055500.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1uXiv:1907.1166
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2843 2853, Dublin, Ireland. Association for Computational Linguistics.
Zheng Liu and Yingxia Shao. 2022. Retromae: Pre- training retrieval-oriented transformers via masked auto-encoder. arXiv preprint arXiv:2205.1205 12035.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. arXiv preprint arXiv:2104.07186.
Jing Lu, Gustavo Herández. Ábrego, Ji Ma, Jianmo Ni, and Yinfei Yang. 2020. Neural passage retrieval with improved negative contrast. arXiv preprint arXiv:2010.122 12523..

---

Shuqi Lu, Di He. Chenyan Xiong, Guolin Kc, Waleed Malik, Zhicheng Dou, Paul Bennett, Tin-Yan Liu, and Arnold Overwijk. 2021. Less is more: Pre- train a strong siamese encoder for dense text re- trieval using a weak decoder. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2780-2791.
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiao- long Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. 2022. Lexmae: Lexicon-botlenecked pre- training for large-scale retrieval. arXiv preprint arXiv:2208.1477
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Tryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evalua- tion of information retrieval models. arXiv preprint AxXiv:2104008660
Ji Ma, Ivan Korotkov, Yinfei Yang. Keith Hall, and Ryan McDonald. 2020. Zero-shot neural passage retrieval via domain targeted synthetic question generation. arXiv preprint arXiv:2004.1450:
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2021. Gpl: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. arXiv preprint arXiv:21
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022. Pre-train a discriminative text encoder for dense retrieval via contrastive span prediction. arXiv preprint arxiv:2244. 10641.
Liang Wang, Nan Yang, Xiaolong Huang. Binxing Jiao, Linjun Yang. Daxin Jiang. Rangan Majumder, and Furu Wei. 2022. Simlm: Pre-training with represen- tation boutleneck for dense passage retrieval. arXiv preprint arXiv:2207.00 025578.
Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021, Learning passage impacts for inverted indexes. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1723-1727.
Xing Wu, Guangyuan Ma. Meng Lin, Zijia Lin. Zhongyuan Wang, and Songlin Hu. 2022. Contextual mask auto-encoder for dense passage retrieval. arXiv preprint arXiv:2208.0766
Yuning Mao, Pengcheng He, Xiaodong Liu, Ye- long Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020. Generation-auggennte retrieval for open-domain question answering. arXiv preprint arXiv:2009.0855:
Lee Xiong, Chenyan Xiong, Ye Li, Kwok Frok-Fung Tang. Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.0083 .00808.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. In CoCo@ NIPs.
Shi Yu, Zhenghao Liu, Chenyan Xiong, Tao Feng, and Zhiyuan Liu. 2021. Few-shot conversa- tional dense retrieval. In Proceedings the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 829- 838.
Rodrigo Nogueira and Jimmy Lin. 2019. doc2query to docttttiquery. Online preprint, 6.
From
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020. Rocketqa: An opti- mized training approach to dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2010.08191.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo. Min Zhang, and Shaoping Ma. 2021. Optimiz- ing dense retrieval model training with hard nega- tives. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 551--512.
Ruiyang Ren. Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021a. Pair: Lever- aging passage-centric similarity relation for im- proving dense passage retrieval. arXiv preprint #xXiv:218.066022
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Adversar- ial retriever-ranker for dense text retrieval. arXiv preprint 11000611
Kai Zhang, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, and Daxin Jiang. 2022a. LED: lexicon-enlightenn dense retriever for large-scale retrieval. CoRR, abs/2208. 13661.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji- Ri-Rong Wen. 2021b. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranning. arXiv preprint arXiv:21 10.07367.
Yanzhao Zhang. Dingkun Long, Guangwei Xu, and Pengjun Xie. 20226. Hlatr: Enhance multi-stage text retrieval with hybrid list aware transformer reranking. arXiv preprint arXiv:2205.1055
Keshav Santhanam, Omar Khattab, Jon Saad- Falcon, Christopher Pouts and Matei Zaharia. 2021. Colberty2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:21 12.01488.
Kun Zhou, Xiao Liu, Yeyun Gong, Wayne Xin Zhao, Daxin Jiang, Nan Duan, and Ji-Rong Wen. 2022. Master: Multi-task pre-trained bottlenecked masked

---

autoencoders are better dense retrievers. preprint arXiv:2212.07841.
arXiv
Yunchang Zhu, Liang Pang, Yanyan Lan, Huawei Shen, and Xueqi Cheng. 2021. Adaptive information seek- ing for open-domain question answering. arXiv preprint arXiv:2109.00744)
A Statistically Analysis of Weakly Correlated Passages
We randomly select 200 documents from the MS- MARCO dataset and randomly select a passage from each document. Then we construct the con- textual pairs in two ways:
1. Random passage-passage pair: Referring to coCondenser (Gao and Callan, 2022), we randomly select another passage within the same document the context for the passage.
2. Generated passage-query pair: Referring to the out-of-shelve docTsSueery (Nogueira and Lin, 2019), query prediction technol- ogy generate query as the context for the passage.
Random passage-passan
Generated passage-query
Pairs
We asked the annotators to label whether the ran- dom contexts or generated queries are strongly re- lated the corresponding passages. We manu- ally annotate the 200 passage-pass passggg pairs and passage-query pairs as high-correlation or low- correlation respectively. To eliminate preference bias, we divide 6 annotators into two groups. One group annotates 100 passage-passage pairs and 100 passage-query pairs, while the other annotates the remaining pairs. The correlation of each pair is voted by the annotation results of three annotators. The statistical results are shown in Table 5.
Correlation rate
35.5%
56.5%
Table 5: Correlation statisties of human annotation sults of different contextual pairs, each with 200 pairs. The score that is better in comparison is marked in bold.
Only 35.5% of the passage-passage pairs are highly correlated, compared to 56.6% of the passage-query pairs. Therefore, we suggest that the generated query is more relevant context than the randomly sampled passages. However, due the limited ability of the base-sized T5 model, nearly half of the generated queries are still not quite exact or strongly correlate the correspond- ing passage. We will further explore the potential ability to utilize large language models to gener- ate more precise and semantic correlate queries for improving the performance boundaries of dense passage retrieval pre-training.


Reviews:
[{"review_id": "ok7Anvt2CM", "reviewer": "Reviewer_DMcN", "summary": "", "questions": "1. How were the values of topp and topk determined for T5 fine-tuning?\n2. The performance could vary depending on how the T5 model is pre-trained. Additionally, what about exploring the use of other generation models besides T5?\n\n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We will answer your questions as follows.\n\n**Q1.** The criteria for dividing passages need to be more accurate. Instead of simply dividing by token length, shouldn't the division be based on semantic units?\n\n**A1.** Dividing by semantic units is an intriguing option for fine-grained context-supervised pre-training. However, some aspects may prevent such usage.\n\n1) Processing efficiency: Dividing pre-training corpus as semantic-related spans is not an easy task, especially when dealing with millions of documents from Wikipedia or Web Pages like MS-MARCO. Dividing by token length shows better processing efficiency.\n\n2) Effectiveness: The boundary of semantic units may not be very clear sometimes. Existing semantic processing tools may bring bias to the processed corpus.\n\nThus for processing efficiency and effectiveness, a popular option for dividing passages in unsupervised context-based pre-training systems [1,2] is still directly cutting them by token length. Thanks for your insightful advice on semantic units, we will seek to solve these issues in future works.\n\n[1] Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval. ACL (1) 2022: 2843-2853\n\n[2] Towards Unsupervised Dense Information Retrieval with Contrastive Learning. CoRR abs/2112.09118 (2021)\n\n**Q2.** How were the values of top-p and top-k determined for T5 fine-tuning?\n\n**A2.** We basically follow the guidance on Huggingface [1] for nucleus sampling. We tested several combinations of top-p and top-k in preliminary experiments and found that $top_p=0.95$ and $top_k=25$ bring diverse queries and a stable generation process.\n\n[1] https://huggingface.co/blog/how-to-generate\n\n**Q3.** The performance could vary depending on how the T5 model is pre-trained. Additionally, what about exploring the use of other generation models besides T5?\n\n**A3.** We use the widely adopted generative T5 model as a query expansion model [1] for reproducibility. We will also explore using more generation models besides T5 in our future works, such as the recent powerful ChatGPT-like large language models. Thanks for your advice.\n\n[1] From doc2query to docTTTTTquery[J]. Online preprint, 2019, 6: 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strongness \n1. They proposed a simple context-supervised pretraining technique by utilizing query-passage pairs. \n2. The results of the experiment show that the method delivers significant enhancements.\n3. They also examined the impact of the number of generated queries and the mixed context.", "reasons_to_reject": "Weakness\n1. The criteria for dividing passages need to be more accurate. Instead of simply dividing by token length, shouldn't the division be based on semantic units?\n", "questions_to_authors": "1. How were the values of topp and topk determined for T5 fine-tuning?\n2. The performance could vary depending on how the T5 model is pre-trained. Additionally, what about exploring the use of other generation models besides T5?\n\n\n\n", "paper_topic_and_main_contributions": "This paper proposes a pre-training technique designed to address the problem of weakly correlated pairs of passages within the same document. The proposed method assumes that a query derived from a passage is likely to be more relevant to that passage, thereby forming a passage-query pair.\nThe experimental results demonstrate that the method yields substantial improvements, thereby showcasing its effectiveness and efficiency.\n\n\n\n\n\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6qmm0B8S7i", "reviewer": "Reviewer_nEeh", "summary": "", "questions": "1) Query generation has been widely used in DSI as below. Comparing the case of DSI, what distinguishable effects are expected by applying the query generation for pretraining dense retrieval? \n- Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A neural corpus indexer for document retrieval. CoRR, abs/2206.02743.\n- Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the gap between indexing and retrieval for differentiable search index with query generation. CoRR, abs/2206.10128\n\n2) The last version of CoT-MAE is CoT-MAE-v2 that uses multi-view representation and decoding. Similar improvements could be made over CoT-MAE-v2 too? \n\n3) In Table 3, the use of single query is most dominant, and increasing number of queries does not so make substantial difference. Can you check the diversity across generated queries? How to control the diversity of the generated queries? \n\n4) In Table 3, it would help us to check the effect of single query when providing the case of the zero query number. \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We will answer your questions as follows.\n\n**Q1.** Comparing the case of DSI, what distinguishable effects are expected by applying the query generation for pretraining dense retrieval?\n\n**A1.** DSI is a typical T5-based generation retrieval architecture. It receives queries as inputs and directly generates the document IDs. Thus DSI learns a direct mapping function from query to doc IDs, where queries partially come from query generation.\n\nIn our method, query generation expands more accurate and fine-grained contexts compared to original context-based pre-training. It does not learn the mapping function like DSI but blends the expanded contexts for better passage representation pre-training. \n\nThus we conclude the distinguishable effects as follows.\n\n1) Different training objects. DSI aims at generation retrieval, while our method aims at context-based representation pre-training.\n\n2) Different retrieval methods. DSI learns a direct mapping function from query to doc IDs, while our method retrieves passages with sentence representations.\n\n3) Different query usages. DSI uses query generation directly in one fine-tuning stage (Or it does not distinguish pre-training or fine-tuning stage), while we use query generation in the pre-training stage, without changing the fine-tuning stage.\n\n\n**Q2.** Similar improvements could be made over CoT-MAE-v2 too?\n\n**A2.** Yes, query-as-context pre-training expands fine-grained contexts at the training data level, which can be applied \nto existing context-based pre-training methods like CoT-MAE-v2. We expect to observe similar improvements for CoT-MAE-v2, but due to the limited time for the response period, we will leave this exploration to our future works. Note that our paper is still self-contained as we focus on exploring context-based dense retrieval methods, while CoT-MAE-v2 uses both dense and sparse retrieval.\n\n**Q3.** In Table 3, the use of single query is most dominant, and increasing number of queries does not so make substantial difference. Can you check the diversity across generated queries? How to control the diversity of the generated queries?\n\n**A3.** The diversity of the generated queries is controlled through nucleus sampling by adjusting $top_p$ and $top_k$, referring to [1]. The current setting has already ensured diversity, and please kindly refer to the examples in Q5. We will open-source all our corpus with generated queries soon.\n\nDifferent numbers of queries per passage do not influence the results significantly. We hypothesize that this is due to the large amounts of pre-training corpus (3.2 million). Even if we only use one query per passage, the amount of queries is still enough for effective pre-training.\n\n[1] https://huggingface.co/blog/how-to-generate\n\n**Q4.** In Table 3, it would help us to check the effect of single query when providing the case of the zero query number.\n\n**A4.** Zero query means that we do not use query-as-context for pre-training. Thus zero query is the baseline result in Table 1. Please kindly refer to Table 1 for the baseline results coCondenser (120k) and CoT-MAE (50k). Moreover, we will add the results of zero query from Table 1 to Table 3 in our next version. Thanks for your question.\n\n**Q5.** About presenting the generated query samples for some passages.\n\n**A5.** Here is an example of passage and generated queries. Due to limited space for the author's response, we only present one example here. And we will present more as an Appendix of the paper in our next release.\n\n| Passage                                                                                                                                                                                                                                                                                                                               |\n|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated. |\n\n| Queries                                                                                              |\n|------------------------------------------------------------------------------------------------------|\n| What were the biggest contributions of the manhattan project?                                        |\n| What was the importance of communication between scientists and each other for the mandela project?  |\n| Why was the madamson project important?                                                              |\n| How did the manhattan project work?                                                                  |\n| Why was the mandela project important to the history of science?                                     |\n| Why was the new york project important?                                                              |\n| Why the manhattan project was so important to society?                                               |\n| Why was the manhattan project important to the scientific world?                                     |\n| Why the manhattan project was so important?                                                          |\n| What did the manhattan project have in common with the atomic bomb?                                  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While query generation for pretraining has been widely used in DSI, the paper extensively applies the query generation on two existing models (i.e., coCondenser and CoT-MAE) in the dense passage retrieval, which makes meaningful and novel contribution in the literature. Different from the existing doc2query that focuses on document expansion, this work applies the query generation for the contrastive learning. In the experiment results, although the proposed method does not achieve the SOTA performance without improving CoT-MAE-v2, the performances are consistently improved on the standard datasets. ", "reasons_to_reject": "- The proposed query generation and its resulting pretraining is quite similar to the works introduced in DSI (i.e., NCI and DSI-QG), which makes the technical part less novel. Comparing to the existing NCI (or DSI-QG), the novelty and value of this work needs be clearly presented. \n\n- The authors criticize that the use of a passage-passage pair in the same document is not optimal for pretraining the dense retrieval, and propose the query-as-context as an alternative approach. However, rather than the proposed alternative view, the proposed query-passage and the existing passage-passage pairs can be integrated in a complementary manner, based on the multi-task learning. Not all passage-passage pairs are problematic in pretraining, as some of the pairs would be relevant. The authors’ critic on the usefulness of the existing passage-passage pair is largely unvalidated. In particular, the experiments in Section 5.2, without mixing two types of pairs, the results of only passage-passage pair need to be presented. Experiments for the effect of combining two pretraining losses derived from passage-passage and passage-query pairs may be required. \n\n", "questions_to_authors": "1) Query generation has been widely used in DSI as below. Comparing the case of DSI, what distinguishable effects are expected by applying the query generation for pretraining dense retrieval? \n- Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A neural corpus indexer for document retrieval. CoRR, abs/2206.02743.\n- Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the gap between indexing and retrieval for differentiable search index with query generation. CoRR, abs/2206.10128\n\n2) The last version of CoT-MAE is CoT-MAE-v2 that uses multi-view representation and decoding. Similar improvements could be made over CoT-MAE-v2 too? \n\n3) In Table 3, the use of single query is most dominant, and increasing number of queries does not so make substantial difference. Can you check the diversity across generated queries? How to control the diversity of the generated queries? \n\n4) In Table 3, it would help us to check the effect of single query when providing the case of the zero query number. \n", "paper_topic_and_main_contributions": "This paper proposes the use of a pair of query-passage for pretraining the dense retrieval, referred to as query-as-context pretraining, motivated by criticizing that a pair of passage-passage in the same document, which is used on the previous context-supervised pretraining, is weakly related or not relevant. The proposed query-as-context pretraining is applied to coCondenser and CoT-MAE, which shows improvements on the standard datasets. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Section 3.1, in Appendix, it would be helpful to present the generated query samples for some passages. "}, "extra_text": {"Missing_References": "- Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A neural corpus indexer for document retrieval. CoRR, abs/2206.02743.\n- Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the gap between indexing and retrieval for differentiable search index with query generation. CoRR, abs/2206.10128\n- Xing Wu, Guangyuan Ma, Peng Wang, Meng Lin, Zijia Lin, Fuzheng Zhang, Songlin Hu, CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrieval, arXiv:2304.03158, 2023\n\n"}}, {"review_id": "Rt0IRRyYM0", "reviewer": "Reviewer_qcxv", "summary": "", "questions": "A. Why are some numbers on TREC DL for your own model implementations missing?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We will answer your questions as follows.\n\n**Q1.** Why are some numbers on TREC DL for your own model implementations missing?\n\n**A1.** Sorry for the missing results. We accidentally ignored them in Table 1. We will fill in the below missing results on TREC DL immediately in our next release. \n\n| Model                                    | MS-MARCO |      |      | TREC DL 19 | TREC DL 20 |\n|------------------------------------------|----------|------|------|------------|------------|\n|                                          | MRR@10   | R@50 | R@1k | NDCG@10    | NDCG@10    |\n| coCondenser (120K) - retriever 1         | 37.0     | 86.0 | 98.5 | **68.2**       | 68.8       |\n| w/ query-as-context (120K) - retriever 1 | **37.4**     | **87.3** | **98.6** | 68.1       | **69.2**       |\n| CoT-MAE (50K) - retriever 1              | 37.2     | 85.7 | 98.2 | 65.7       | 66.5       |\n| w/ query-as-context (50K) - retriever 1  | **38.6**     | **87.7** | **98.6** | **67.7**       | **67.8**       |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written and easy to follow. The idea of using pseudo queries as context for pre-training is simple and clear.\n2. Experiments on popular passage retrieval datasets show that the proposed method is effective and outperforms other competitive methods.\n", "reasons_to_reject": "1. The technical contribution is not much. Generated pseudo queries have been widely used in the context of passage retrieval, including doc2query style sparse retrieval, zero-shot and few-shot dense retrieval. I acknowledge that this is the first paper to combine pseudo queries with coCondenser / CoT-MAE framework, but it is quite straightforward, and the overall contribution is limited.\n2. The proposed method requires training a query generator with labeled data, while methods like coCondenser work with unlabeled data only. This will limit the applicability of the proposed method.\n", "questions_to_authors": "A. Why are some numbers on TREC DL for your own model implementations missing?\n", "paper_topic_and_main_contributions": "This paper proposes a new method for context-supervised pre-training for dense passage retrieval. The main idea is to use generated pseudo queries as context for pre-training. It can be combined with existing frameworks like coCondenser or CoT-MAE. The proposed method is evaluated on the MS-MARCO passage retrieval dataset and out-of-domain zero-shot BEIR benchmarks. Experimental results show that the proposed method is effective compared to baselines.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to generate queries from a passage, and use them to pre-train a dense retriever. The approach is an alternative to using passage-passage pairs for pre-training. The experiments demonstrate the positive effects using such a pre-training approach.

The idea is interesting, although it is a straightforward extension to the exiting work. The experiments successfully demonstrate that this is a reasonable way to create training datasets for dense retriever.

The authors have answered most of the questions raised by the reviewers in their rebuttal.
Decision: Accept-Main