MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers
Authors: Kun Zhou
Xiao Liu
Yeyun Gong
Xin Zhao
Daxin Jiang
Nan Duan
Ji-Rong Wen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-task Pre-training
Dense Retrieval

Abstract: Dense retrieval aims to map queries and passages into low-dimensional vector space for efficient similarity measuring, showing promising effectiveness in various large-scale retrieval tasks. Since most existing methods commonly adopt pre-trained Transformers (\eg BERT) for parameter initialization, some work focuses on proposing new pre-training tasks for compressing the useful semantic information from passages into dense vectors, achieving remarkable performances. However, it is still challenging to effectively capture the rich semantic information and relations about passages into the dense vectors via one single particular pre-training task. In this work, we propose a multi-task pre-trained model, MASTER, that unifies and integrates multiple pre-training tasks with different learning objectives under the bottlenecked masked autoencoder architecture. Concretely, MASTER utilizes a multi-decoder architecture to integrate three types of pre-training tasks: corrupted passages recovering, related passage recovering and PLMs outputs recovering. By incorporating a shared deep encoder, we construct a representation bottleneck in our architecture, compressing the abundant semantic information across tasks into dense vectors. The first two types of tasks concentrate on the semantic information of passages and capturing relationships among them within the pre-training corpus. The third can capture the knowledge beyond the corpus from external PLMs (\eg GPT-2). Extensive experiments on several large-scale passage retrieval datasets have shown that our approach outperforms the previous state-of-the-art dense retrieval methods.

MASTER: MULTI-TASK PRE-TRAINED BOTTLE- NECKED MASKED AUTOENCODERS ARE BETTER
DENSE RETRIEVERS
Anonymous authors Paper under double-blind review
ABSTRACT
Dense retrieval aims to map queries and passages into low-dimensional vector space for efficient similarity measuring, showing promising effectiveness var- ious large-scale retrieval tasks. Since most existing methods commonly adopt pre-trained Transformers (e.g., BERT) for parameter initialization, some work fo- cuses proposing new pre-training tasks for compressing the useful semantic in- formation from passages into dense vectors, achieving remarkable performances. However, it is still challenging to effectively capture the rich semantic information and relations about passages into the dense vectors via one single particular pre- training task. In this work, propose multi-task pre-trained model, MASTER, that unifies and integrates multiple pre-training tasks with different learning objec- tives under the bottlenecked masked autoencoder architecture. Concretely, MAS- TER utilizes a multi-decooer architecture to integrate three types of pre-training tasks: corrupted passages recovering, related passage recovering and PLMs out- puts recovering. By incorporating a shared deep encoder, we construct repre- sentation bottleneck in our architecture, compressing the abundant semantic in- formation across tasks into dense vectors. The first two types of tasks concentrate on capturing the semantic information of passages and relationships among them within the pre-training corpus. The third one can capture the knowledge beyond the corpus from external PLMs (e.g., GPT-2). Extensive experiments on several large-scale passage retrieval datasets have shown that our approach outperforms the previous state-of-the-ar dense retrieval methods.
INTRODUCTION
Recent years have witnessed the great success of dense retrieval methods (Karpukhin et al.
2020
Qu et a 2021 Xiong et al.] (221)) in industrial applications, e.g., web search (Brickley et al. 2019
Qiu al. 2022) and question answering (Karpukhin et al., 2020: Izacard & Grave 2021) ) These methods typically encode queries and passages into low-dimensional dense vectors and utilize the vector similarity between them to measure semantic relevance. In real-world applications, the dense vectors of large amounts of passages will be pre-computed. Then the approximate nearest neighbor (ANN) search techniques (Johnson al. 202T) can be incorporated for efficient retrieval.
To generate high-quality dense vectors, pre-trained language models (PLMs) (Devlin et al. 2019 Liu al. 2019) have been widely adopted the backbone of the query and passage encoders. However, general PLMs (e.g., BERT (Devlin et al.] 2019)) may not be the best for dense retrieval, as their produced native dense representations (usually the [CLS] emhedding) are not designed on purpose to generalize the information from the input text. To solve it, recent studies (Gao & Callan 2021a Lu et al., 2021: Sachan et al.) 2021) adopt pre-training techniques to endow the [CLS] embedding the capacity of compressing the semantic information of the input text. They either rely on the autoencoding task that utilizes the (CLS] embedding to recover the corrupted text (e.g., masked replaced tokens) (Liu & Shao 2022; Wang et al., 2022 Wu et al. 2022). leverage the contrastive learning objective to capture the relations among passages (e.g., co-occurrence) (Ram)
al.
2022 Sachan al. 2021), outperforming general PLMS in this task.

---

Despite the success, it is obvious that neither the autoencoding nor contrastive learning pre- task is optimal to fully exploit the useful characteristics into the dense embedding for the retrieval task, as either mainly relies on limited specific information or relation from the corpus. From this point of view, adopting the multi-task pre-training framework that jointly learns various su- pervised signals from different tasks is promising. However, due to the divergences of input formats and learning objectives among different tasks, an arbitrary integration of these tasks is inappro- priate, which may even cause detrimental gradient interference, leading performance degrada- tion (Kendall et al. 2018 Yu et al. 2020).
pre-training
To address this problem, we consider integrating multiple pre-training tasks in a unified format and reducing the divergence of different training objectives. Since most of the NLP tasks can be re- formulated as the text-to-text format (Xie et al. 2022, Raffel et al., 2020), we can also reconstruct the available pre-training tasks into such a format. Recently, the idea of bottlenecked masked au- toencoder (BMAE) (Liu Shao 2022 Wang et al. 2022] Wu et al., 2022) has been proposed to pre-train dense retrievers, which typically adopts an encoder-decoder architecture, consisting of deep encoder to generate the dense vector of the input texts and a shallow decoder that relies on the dense vector to recover an pearressvvel--maskee text. In this way, an information bottleneck is con- structed. The deep encoder should force the dense vector to reserve as much useful information as possible that is beneficial for recovering the text in the shallow decoder. Inspired by it, we consider unifying multiple different pre-training tasks into the BMAE format, iee, taking texts as the input of the encoder and recovering itself or its related text in the decoder. For example, passage relations (e.g., co-occurrence), we can utilize a passage as the encoder's 's input and lever- age its dense vector to help recover an aggressive masked related passage in the decoder. Such unified way is promising to solve the central issue of the multi-task pre- pre-training derived from the divergences of input formats and learning objectives among different tasks, and capture variety of semantics or relations in different tasks to pre-train effective dense vectors.
capture the
Based on the above motivation, we propose MASTER, a multi-task pre-trained bottlenecked masked autoencoder, that adopts an multi-decode architecture to integrate diverse pre-training tasks in the BMAE format. For each pre- pre-training task, we devise a task-specific decoder to accomplish it, and all these decoders should rely on the output dense vector from the shared deep encoder to guide the decoding process. In this way, we construct multiple information bottlenecks to enforce the deep encoder to generate more informative dense vectors, leading to compressed high-quality representa- tions. To learn sufficient useful semantics and relations, we devise three types of pre-training tasks: corrupted passages recovering, related passages recovering, and PLMs outputs recovering, respec- tively, a total of five tasks for pre-training. The first two types of tasks focus on compressing the semantic information of passages and modeling the relationships among passages within the corpus. The third type of tasks forces the dense vector of the input passage to recover the output text from other public generative PLMs like GPT-2 (Radford et al. 2019), which are capable of capturing the semantic information and relations beyond the corpus to further enhance the dense vectors.
To verify the effectiveness of our approach. we conduct extensive experiments on several text re- trieval datasets, e.g., MS-MARCO Passage Ranking (Nguyen et al. (016)). TREC Deep Learning Track (Craswell et al., 2020. 202T), Natural Questions (Kwiatkowski et al., 2019) and BEIR zero- shot retrieval benchmark [hakur et al. (202T). Experimental results show that our approach can achieve new state-of-the-an thr- performances in dense retrieval. We will make the code and model checkpoints publicly available.
2 RELATED WORK
Dense Retrieval. Recent years have witnessed the remarkable progress of dense retrieval (Karpukhin et al. 2020 Zhan et al., 2020; Hong et al., 2022 Ram et al. 2022). Different from tra- ditional sparse retrieval methods (e.g., BM25 (Robertson et al. (209)), dense retrieval approaches typically map queries and documents into low-dimensional dense vectors via a dual-encoder archi- tecture and then utilize vector distance metrics (e.g., dot product and cosine similarity) as the rele- vance scores. Such a way is supported by the efficient approximate nearest neighbor (ANN) search engines, e.g.. FAISS (Jhnsso et al. 2021). For effectively training dense retrieval models, existing work typically leverages pre- trained Transformers (Liu et al. 2019; Devlin et al., 2019) to initialize the dual encoders and then samples high-quality negatives when fine-tuning the encoders with con-
2

---

trastive learning objectives. Early work (Karpukhin al. 2020 Min et al. 2020 mainly relies on in-batch random negatives and hard negatives mined by BM25. Afterward, a line of work (Qu et al. 2021 Xiong et al. 2021) finds that sampling top-k ranked documents by trained dense retrivver as hard negatives is more beneficial for fine-tuning. However, a common problem for such top-k negative sampling strategies is that they are easy to select false negatives, which impedes better performances. To alleviate the problem, current studies have explored several practical directions, including knowledge distillation with rerankers (Qu et al. 2021
Ren et al. 2021b]
b; Lu et al. 2022),
more effective pre-training methods (Zhou et al. 2022 Xu et al. 2022). denoising techniques Mao
al. 2022 Hofstätter et al. 2021) and so on.
Pre-training for Dense Retrieval. PLMs (Devlin et al. 2019 Sanh et al. 2019 Sun et al. 2020) have become the de facto backbone NLP models for their remarkable performances on various tasks. However, the fact is that these models pre-trained with general tasks (e.g., masked lan- guage model) without any prior task knowledge. Therefore, existing work finds that they are not ready use for dense retrieval (Gao & Callan 2021a] 2022), especially in low-data situations (Xul iet al. 2022). To solve this issue, several studies (Gao & Callan 2021a Lu et al.] [021] Xu et al.) 2022) are proposed to make the output sentence embedding more informative and discriminative. A type of work relies on the explicit relations between text pairs and designs the pre-training tasks
based on the contrastive learning objective (Lee et al. 2019 Chang et al. 2020 Ma et al., 2022). e.g., inverse cloze task and contrastive span prediction. Another line of work focuses on compress- ing the textual semantic information into the [CLS] embedding. These methods prefer leverage the masked autoencoder architecture that incorporates deep encoder and shallow decoder, forcing the [CLS] embedding of the input text from the encoders to recover itself (Liu Shao 2022 Wu
et al. 2022 or related texts (Wang et al. 2022).
3 PRELIMINARY
In this section, we introduce the task definition of this work and present the typical fine-tuning process of dense retrieval.
Task Definition. Given a query q, the dense passage retrieval task aims to retrieve the most relevant top-k passages {p:}<= from a large candidate pool P. To achieve this goal, the dual-encoder architecture is widely used. It consists of query encoder Eg and a passage encoder Ep, mapping the query 9 and passage p into k-dimensional dense vectors h2 and hy, respectively. Then, the semantic relevance score of q and will be computed using dot product as
s s(g.p) = hg hp-
Existing work mostly adopts preitrained Transformers (e.g., BERT (Devlin et al., 2019)) as the two encoders, using the representations of the [CLS] token as the dense vectors. In this work, we aim to propose more effective multi-task pre-training framework specially for the dense retrieval task, which learns to compress more useful information into the [CLS] representations. Formally, given pre-training corpus and Transformer encoder, focus devising several tasks to preitrain the parameters of it. Then, the pre-trained Transformer will be used as the backbone of the query encoder F., and passage encoder F.p. and can be fine-tuned on downstream dense retrieval tasks.
Fine-tuning Dense Retrievers. In the fine-tuning stage, the learning objective is to pull the rep- resentations of a query 9 and its relevant passages P together (as positives), while pushing apart irrelevant ones P
P P (as negatives). Therefore, high-quality negatives are critical to the ef- fectiveness of dense retrievers. Existing work commonly leverages the BM25 negatives (Karpukhin et al. 2020) or the top-k ranked negatives mined by well-trained dense retriever (Qu et al. 2021 D Then, the optimization objective can be formulated as:
Xiong et al. 2021). denoted D=.
0* arg min
Σ Σ Σ 1(s(q.d*), s(g.
4 d+ d-eD-
where I(.) is the loss function. Besides, as the top-k hard negatives may contain false negatives, Ren
recent studies (Qu al.) Ren et al. 20.1 Lu et al. (002)) have adopted knowledge distillation strategies to solve it. They rely on pre-le cross- crosserncoder rerankers to produce soft labels on D and minimize the KI. divergence between the dual encoders' outputs and the soft labels.
3

---

Corrupted Passages Recovering
Masked Language Model
Shallow Decoder
Shallow Decoder
lov
(Missted Keywordss Prediction)
/Compiementory Most Prediction)
h''
Isslad
bANS-
Ap'
pUASE
bassel
hpp
Related Passages Recovering
Deep Encoder
Shallow Decoder
(Neighbaring Passage Recovering)
hp'
XL5)
IMAS
PLMs Outputs Recovering
Random
Mask
Whot
Nobood
Icep
seoge
Shallow Decoder
Shallow Decoder
(Doc2query Outouts Recovering)
(GPT-2 Outputs Recovering)
hp'
hp'
Figure 1: The overview of our MASTER. We incorporate a bottlenecked multi-decoder architecture, and design three types of pre-training tasks, requiring totally five decoders for specific tasks.
4 APPROACH
In this section, we present MASTER, an approach to pre-train an effective dense retriever. We first introduce the bottlenecked model architecture (consisting of deep encoder and multiple shallow decoders), then describe our adopted three types of pre-training tasks unified as the bottlenecked masked autoencoding manner. Figure shows the overview of our approach.
4.1 BOTTLENECKED MULTI-DECODIE ARCHITECTURE
To tre-train the dense retriever for compressing useful information into the dense vectors, we borrow the idea of bottlenecked masked autoencoders (BMAA) using an ancoder-decoder architecture (Liu & Shao, 2022; Wang et al.) [022; Wu et al.) 2022). In this way, a deep Transformer encoder is to compress the input text into dense vector, and shallow decoder to recover the input or other related text based on it. To enrich the informativeness of the output dense vectors, we devise a multi- decoder architecture that incorporates five decoders corresponding to different pre-training tasks to capture diverse semantics and relations.
Concretely, the deep Transformer encoder shares the same architecture as BERT (Devlin et al. 2019), and can be initialized with its pre-trained parameters. Given passage p from the pre- pra-tinin corpus, we leverage the deep encoder to encode it, and select the output representation of the [CLS] token its dense vector hp. Following existing work (Gao & Callan 2021a Lu al., 2021). employ a masked language model task to pre-train the encoder. Formally, a certain percentage a% of tokens from p will be masked to obtain p', and the encoder needs predict them
LMLM
Σ
- log (otpp(t/p':O0p))
(3)
NEM,
where Mp denotes the masked tokens p/. Og denotes the parameters of the encoder. The multiple shallow decoders all adopt the 2-layer bi-directional Transformer encoder architecture, and share the embedding matrix and language modeling head with the deep encoder. For each decoder, its input an aggressive masked text x' (masking rate 3 2 50%) that requires to be recovered. Besides, the dense vector h from the encoder will be inserted into the decoder to replace the original [CLS] token embedding. In this way, the learning objective of each decoder is formulated
LD
Σ
- log p(t,l( hy: Og, @p)
(4)
L.M.M
where Myr denotes the masked tokens in x', OD denotes the parameters of the decoder. Such way can build the information bottleneck that the multiple decoders must rely on the h, to help recover its input text, forcing hy reserve useful information from the input.
4

---

4.2 MULTI-TASE
Based on the architecture, we devise multiple pre-training tasks, to help the output dense vectors from the encoder capture more useful information. Concretely, we adopt three types of tasks to capture the semantic information within passages, relations with other passages, and knowledge from other PLMs, namely corrupted passages recovering, related passages recovering and PLMs outputs recovering, respectively.
Corrupted Passages Recovering, Given a passage from the pre-training corpus, the corrupted passages recovering tasks (CPR) first mask its contained tokens to compose the inputs of the encoder p' and decoder D' according to the mask rates a% and 3% respectively. Then, the output dense vector h from the encoder will be leveraged to help the shallow decoder to recover into p. Such way is helpful to compress important semantic information from the passage into the dense vector. To achieve it, we design two pre-training tasks by utilizing special masking mechanisms for the decoder, namely masked keywords prediction (MKP) and complementary mask prediction (CMP),
For MKP, we aim to mask as more keywords as possible in the decoder, as they may reflect im- portant semantic information of the passage. Specifically, we rely on the widely-used TE-IDF weights (Ramos et al., 2003) to obtain a masked probability distribution about words in the pas- sage, where keywords with low frequencies would receive larger probabilities to be masked. In this way, the input masked passage PмKp of the decoder will lose most keywords, which will force the dense vector hy to well reserve their information for recovering.
For CMP, given the passage p. we leverage a complementary mask mechanism in the decoder that masks the unmasked tokens from the input of the encoder p'. As result, the incomplete inputs of the encoder and decoder will be complementary. and the dense vector hy should accurately remember all the unmasked input information from p' for recovering 1'MMP-
Finally, the pre-train pre-training objective of the CPR tasks is given by combining the above two tasks as:
LCPR
-
MKP log p(u PLAKP, b θg, OD
- -logp(t hy; Oe CMP
(5)
4.MMKR
G.Mcsa
where ММKP and MCMP denote the masked tokens in PlKE and p.MP respectively, and O and ODMP are the parameters the two specific decoders, respectively.
Related Passages Recovering. The related passages recovering task (RPR) aims to model the semantic relationships between related passages. In this work, we focus on the commonly-used and easily-obtained co-occurrence relation from the pre-training corpus. Based on this motivation, collect the passage pairs (Pi,Pi+1) that are neighbouring spans in a document, and devise the neighbouring passage recovering task (NPR).
In NPR, given a neighbouring passage pair (pi, 1)), we rely on the mask rates a% and ,%% to mask their tokens for composing the inputs of the encoder p and decoder P respectively. Next, the output dense vector of pt from the deep encoder is utilized to help the decoder recover P/+11 Such way will encourage the dense vector retain the information that is useful recover the neighbouring passage, capturing the intrinsic token-level correlations across the two passages. Be- sides, we also rely on the TF-IDF weights of words mask more keywords in the decoder as MKP, which further increases the difficulty of this task and forces the dense vector to focus more on the key information. The learning objective of the RPR task can be defined as:
TRPR
Σ
- log Mogg((tel/un hy; Ox, NPR).
(6)
E.E.
where MNPP and @PP denote the masked tokens in p/+1 and the parameters of the decoder specially for the NPR task, respectively. Note that existing work (Lee et al. 2019 Ma et al.) 2022) has also considered the neighbouring relations and mostly adopts the contrastive learning objective to capture it. In fact, contrastive learning mainly aims to characterize the passage-leve semantics and arbitrarily pushes apart irrelevant passages, even if they semantically relevant. As comparison, the NPR task can capture more fine-grained token-level characteristics, and such a text covering task much safer to not hurt the semantic relevance between relevant but not neighbouring passages.
5

---

PLMs Outputs Recovering. The above tasks able capture the semantic information and relations within the unsupervised pre-training corpus. We further consider to learn the knowledge from other PLMs, to capture more rich information beyond the corpus. Based on this idea, design the PLMs outputs recovering tasks (POR) that aim to recover the outputs of two generative PLMs, consisting of the doc2query outputs recovering (DOR) and GPT-2 outputs recovering (GOR) tasks.
Given a passage p. we leverage a public well-trained doc2query model (Nogueira & Lin 2019) to generate & relevant queries { and concatenate them into long sentence S(g), as the generated queries have shown effectiveness in previous dense retrieval methods (Nogueira et al., 2019). Be- sides, we also use p as the prompt to guide the popular autoregressive GPT-2 model (Radford al. 2019) to generate a long sentence S(g), as GPT-2 has shown surprising performance in generating intormative long text. Then, we aggressively mask the tokens in ((g) and 8(g) according to the mask rate 8%, obtain the inputs 5/g) and ('g) two task-specific decoders. Similar to above tasks, the two decoders also rely on the dense vector hy to recover the generated texts, and the pre-training objective of the POR tasks is the combination of the two tasks as:
PPOR
Σ MEE.MDORR
log patssl(((asnnn. hp; DOR)
Σ
logp(tils()) hp; Og. ODRR)
(7)
&.M.GOO
where MDOO and .MGOR denote the masked tokens in ((g) and <6)>> respectively, and ODO and Do are the parameters of the two specific decoders. respectively. In this way, we can enhance the dense vector capture richer semantics from other PLMs, and learn more information in the documents corpus. Note that such way is similar to the knowledge distillation process that transfers the learned knowledge from PLMSs into the dense vector by forcing it to predict the PLMs' outputs.
included
4.3 LEARNING
During pre-training, we optimize the parameters in the deep encoder and the multiple shallow de- coders using the above pre-training tasks, denoted as:
Liotl = LMLM + LCPR + LRPR + LPOR
(8)
During fine-tuning, utilize the pre-trained deep encoder as the backbone of the query and pas- sage encoders. Following the pipeline in previous dense retrieval methods (Gao & Callan [022] WWang et al. 2022 Wu et al.) 2022). we first train the Retriever, using the in-batch negatives and BM25 hard negatives. Then, we utilize Retriever, to mine hard negatives from large-scale passage pool, and leverage these negatives and in-batch negatives to train the Retrieverg. Next, we train cross-encoder reranker model based on the mined negatives from Retrieverz. Finally, we distil the knowledge from the reranker into the Retrieverassii by using it to produce soft labels for both positives and mined negatives from Retriever,. Note that our pre-trained encoder is used to initialize the Retriever,, Retrievery and Retrieveraisul-
EXPERIMENT
5.1 EXPERIMENTAL SETTING
Datasets and Evaluation. We conduct ex-
periments on several text retrieval datasets:
Table 1: Statistics of the text retrieval datasets.
MS MARCO Passage Ranking (MS-MARCO)
"#Passage" means the number passages.
(Nguyen et al. 2016). TREC 2019 Deep Learn-
Dataset
Train
Dev
Test
#Passage
ing Track (TREC-19) (Craswell et al.) 2020).
MS-PAS S02.939
6,980
8,841,823
TREC 2020 Deep Learning Track (TREC-20)
TREC-19
200
8,841,823
Craswell et al.] 2021). and Natural Questions
TREC-20
200
8,841,823
(NQ) (kwiatkowski et al. 2019). The statistics of the above datasets are shown in Table [ MS-
NQ
58,880
8,757
3,610
21,015,324
TQ
60.413
8.837
11,313
21,015,324
MARCO consists real queries collected from
Bing search engine.NQ is an open domain question answering dataset.

---

Table 2: Experimental results on three web search datasets. The best and second-best methods are marked in bold and underlined, respectively. The inn the column of "with KD?" means that the model has used knowledge distillation.
MS-MARCO
TREC-19
TREC-20
Model
with KD?
MRR@10 10
R@50
R@ 1k
nDCG@ 10
nDCG@10 10
BM25 Yang et al. 2017
18.5
58.5
85.7
51.2
47.7
DeepCT Dar Callan 2019a
24.3
69.0
91.0
57.2
docTSquery (Noguerra & Lin 2019
27.7
75.6
94.7
64.2
ANCE [Xiong al.. 2021
33.0
95.9
64.5
64.6
STAR (Zhan al. 2021
34.7
68.3
TAS-B (Holstatier al. 2021
34.0
97.5
71.2
69.3
RocketQA
37.0
85.5
97.9
RocketQAv2 [Ren al.
202
38.8
86.2
98.1
AR2 (Zhang al. 2022
39.5
87.8
98.6
ERNiE-Search (Lu ct
2022)
87.7
98.2
COIL Gao et al. 202 la
35.5
96.3
70.4
ColBERT (Kattab & Zäharia
2020
36.0
82.9
96.8
ColBERT 2 Saathanam et al.
2022
39.7
86.8
98.4
SERD aL et al_ 2
33.9
96.1
RetroMAB (Lii & Shao
2022
35.0
97.6
Condenser Gao & Callan
202
36.6
97.4
69.8
coCondenser (Gao & Callan
2022
38.2
86.5
98.4
71.7
68.4
CoT-MAE (Wu
2022
39.4
87.0
98.7
70.4
PAIR (Ren et at. 2021a
37,9
86.4
98.2
SimLM (Wang ct ut. 2022
41.1
87.8
98.7
71.2
69.7
MASTER
41.5
88.6
98.8
72.7
71.7
5.2
MAIN RESULTS
Performance on Web Search Datasets. Table shows the experimental results on three web search benchmarks, i.e., MS-MARCO, TREC-2019 and TREC-2020. First, distillation strategy, the best baselines are both pre-training dense retrieval methods, i.e., CoT-MAE and SimLM, even outperforming methods using multiple representations. It indicates that proper pre-training strategies are helpful to the downstream dense passage retrieval tasks. Second, SimLM mostly outperforms other baselines. It employs bottlenecked architecture that learns to compress the input information into a dense vector, and adopts a replaced language modeling objective to pre- train it. Such way
that with without
effective to force the dense vector reserve the important semantics.
Besides, we can see that our approach outperforms all the baselines in terms of all metrics on all datasets. Our approach adopts multi-task pre- pre-training framework that unifies five tasks recover- ing of corrupted passages, related passages and PLMs outputs, based on a bottlenecked one-encoder multi-decode architecture. this way, we can foree the output dense vector from the encoder be more informative and functional to accomplish these tasks, leading to better representative capacity.
Performance on Open Domain QA Datasets. Table 3 shows the experimental results open domain QA datasets, NQ. For a fair com- parison, only report the performance of Retriever, without performing knowledge dis- tillation our approach. First, we can also see that pre-training dense retrieval methods mostly outperform other methods. It further indicates the effectiveness of pre-training techniques in open domain QA tasks. Besides, coCondenser and SimLM perform better than other methods, the reason is that they both adopt bottlenecked architecture to compress the information into the dense vectors. Finally, we can see that our approach outperforms all the baselines. As comparison, our approach can enhance the in- formativeness of dense vectors by integrating
Table 3: Results on NQ. Following (Wang et al. 2022), we report the performance Retriever,
without knowledge distillation for our MSATER.
NQ
Model
R@20 20
R@ 100
BM25 (Yang et al. (017)
59.1
73.7
DPRangle (Karpukhin al. ANCE (Xiong et al. 2021
2020)
78.4
85.4
81.9
87.5
RocketQA (Qu el al.
2021
82,7
88.5
Rocket (Ren el al.
2021b
83.7
89.0
Condenser (Gao & Callan
202
83.2
88.4
PAIR (Ren al. 2021a
83.5
coCondenser (Gao Callan
2022
84.3
89.0
SimLM (Wang et al. 2022
84.3
89.3
MASTER
84.6
89.4

---

Table Zero-shot dense retrieval nDCG@ @ performances on BEIR benchmark. Results with *
from our reproduction.
Dataset
BERT
LaPraDoR
SimCSE
DiffUSE
SEED
Condenser
SimLM
MASTER
TRECSCOVID
0.649
0.495
0.524
0.4992
0.612
0.754
0.637
0.620
BioASQ
0.262
0.239
0.264
0.258
0.297
0.317
0.350
0.354
NFCorpus
0.257
0.283
0.250
0.259
0.256
0.278
0.323
0.330
NQ
0.438
0.415
0.412
0,412
0.425
0.459
0.477
0,516
QotpotQA
0.478
0.488
0.502
0.499
0.528
0.537
0.581
0.589
FiQA-2018
0.266
0.240
0.244
0.292
Signal- (RT)
0.216
0.245
0.264
0.260
0.246
0,258
0.257
0.252
TREC-NEWS
0.362
0.206
0.368
0.363
0.335
0.353
0.326
0.409
Robus104
0.364
0.310
0.353
0.343
0.348
0.352
0.368
0.405
ArguAna
0.357
0.503
0,436
0.468
0.347
0.375
0,421
0.395
Touché-2020
0.270
0.178
0.178
0. 168
0.180
0.223
0.292
0.320
CQADupStack
0.284
0.326
0.295
0,305
0.285
0.316
0.332
0.327
Quora
0.782
0.843
0.848
0.850
0.849
0.855
0.773
0.791
DBPedia
0.298
0.328
0,304
0.303
0.324
0.331
0,345
0.399
SCIDOCS
0.115
0.145
0.125
0.125
0.117
0.136
0.145
0.141
FEVER
0.684
0.518
0.651
0.641
0.653
0.682
0.657
0.692
Climate-FEVER FEVER
0.205
0.172
0.222
0.200
0.176
0.163
SciFact
0.504
0.483
0.545
0.523
0.556
0.570
0,588
0.637
Avg. Performance
0.376
0.358
0.377
0.372
0.377
0,403
0.407
0,429
Table 5: Comparison with different pre-training dense retrieval methods in three stages of our fine- tuning pipeline on the dev set MS- AA-O. Results with * are from our reproduction.
coCondenser
CoTMAE
SimLM
MASTER
Model
MRR @ 10
R@ 1k
MRR@ 10
R@I 1k
MRR@N 10
R@ 1k
MRR 10
R@ 1k
Retrieveri
35.7
97.8
36.8*
98.3*
38.0
98.3
38.3
98.8
Retriever,
38.2
98.4
39.2
98.7
39.1
98.6
40.4
98.8
Retrieverassil
40.2
98.3
40.4
98.7
41.1
98.7
41.5
98.8
multiple pre-training tasks, which compress the semantic information within passages, model the relations between passages, and learn the knowledge from other PLMs.
Zero-Shot Evaluation. We evaluate the zero-shot retrieval performance of our approach BEIR benchmark (Thakur et al.) 2021). It contains 18 datasets, covering dense retrieval tasks across dif- ferent domains, e.g., question answering, fact checking, bio-medical retrieval and news retrieval. Following Thakur et al. (2021), we fine-tune our approach in MS-MARCO training set and evalu- it the BEIR benchmark using the official evaluation toolkit to show the zero-shot abilities. nDCG@ @ 10 is chosen as the evaluation metrics. As shown in Table we can see that our approach outperforms most of baselines in all 18 datasets, and the average performance also surpasses all baselines significantly. Since our approach incorporates multiple pre-training tasks for learning the dense representations, such a way can enrich the informativeness of them and help better adapt into different domains and retrieval tasks.
5.3 FURTHER ANALYSIS
Performance in Three Stages of Fine-tuning Pipeline. To further investigate the effectiveness of our approach, we show the performances of MASTER and other pre-training dense retrieval methods in each stage of fine-tuning pipeline. Here, the models in the three stages are all initialized by corresponding pre-trained parameters of these methods. As shown in Table we can see that the performances all pre-training methods are consistently improving with the process of the three-stage training, and the models in the third stage Retrieverasil perform the best. In addition, our approach also consistently outperforms all other pre-training methods in all the three stages. also indicates the effectiveness of our proposed multi-task pre- tre-niannin strategy, which can produce higher-quality dense vectors for downstream retrieval tasks.
Ablation and Variation Study. Our proposed approach incorporates multi-decoder architecture
and three types of tasks for pre-training. To verify the effectiveness of each part, we conduct the
8

---

Table 6: Ablation and variation study of our approach. We report the means and variances of
MRR@ @ 10 of the retriever, and retriever, on the dev set of MS- MARCO.
Model
MASTER
w/o CPR
w/o RPR
w/o POR
+Shared-Dec
SimLM
Retriever
38.3 (0.016)
37.7 (0.016)
37.6 (0.007)
37.6 (0.016)
37.4 (0.016)
38.0 39.1
Retrievera
40.4 (0.016)
39.9 (0.007)
39.8 (0.009)
39.8 (0.002)
39.1 (0.069)
Table 7: Performance comparison w.r.t. different training data size on the dev set of MS-MARCO.
MRR@
R@ lk
Model
5%
10 %
20% %
50%
100%
5%
10 %
20 %
50%
100%
SimLM
31.5
32.6
33.8
35.4
38.0
97.2
97.5
97.8
98.2
98.7
MASTER
33.0
34.3
35.4
36,6
38.3
97.8
97.9
98.4
98.7
98.8
ablation and variation study on the dev set of MS-MARCO to analyze their contributions. We remove the CPR, RPR and POR tasks individually, and propose variants that adopts a shared decoder to deal with the multiple tasks. As shown in Table we can see that all the ablation and variation models will lead to the performance degradation. It indicates that all the pre-training tasks and our multi-decoder architecture are useful to improve the performance. Besides, after removing any type of pre-training tasks, Retriever, still outperforms the SOTA method, SimLM. It further shows the promising effectiveness multi task pre- pre-trainin training for dense retrieval tasks.
Performance w.r.. Different Pre-training Steps. As pre-training approach, the num- ber of pre-training steps will affect the perfor- mance downstream tasks. In each step, we optimize the model parameters using a batch of pre-training data by gradient descent algo- rithm. However, too many pre- pre-training steps conecconguinng and costly. Here, in- vestigate the performance convergence speed of approach during pre-training. As shown in Figure 2 we can see that our model performs well with few pre-training steps, especially that the retriever, of our method achieves the 39.1 metric (the same as SimLM) af- ter 10k steps. It shows that our approach is more effective to pre-train effective dense vec- tors, with no need for too many pre-training steps.
41
39
MIRR@10
37
Retriever-1 of MASTER
38
Retriever-2
Retriever- of Siml M
Retriever-2 of SimLM
33
120
Pre-training Steps (k)
Figure 2: Performance comparison w.rt. different number of pre-training steps MS-MARCO.
on MRR @ 10
Few-Shot Learning. In our approach, as we have pre-trained the backbone via a multi-task man- ner, the pre-trained dense vectors be easily adapted into downstream tasks with less data. To validate this conjecture, we reduce the training data size into 50%, 20%, 10% and 5%, and com- pare the MRR@ @ 10 and R@ 1k of our approach with the SOTA pre-training method SimLM. As shown in Table 7 we can see that the performance substantially drops when less training data is used. Additionally, our approach is consistently better than SimLM in all cases, especially in an extreme sparsity level (5%). It indicates that MASTER is better pre-trained to effectively adapt to downstream dense retrieval task.
CONCLUSION
In this paper, we proposed MASTER, a multi-task pre-trained bottlenecked masked autoencoder for dense retrieval task. In our approach, we adopted bottlenecked multi-decoder architecture to integrate a variety pre-training tasks, and devised three types of pre-training tasks about corrupted passages recovering, related passage recovering and PLMs outputs recovering. The three types of tasks focused on compressing the semantic information within the passages, modeling relations among passages, and learning the knowledge from external public generative PLMs. respectively, leading to more informative and effective dense vectors. Experimental results have shown that our approach outperforms several competitive baselines.

---

REFERENCES
Dan Brickley, Matthew Burgess, and Natasha F. Noy. Google dataset search: Building search engine for datasets an open web ecosystem. In Proceedings of www 2019, pp. 1365-1375,
2019. UtLtttps:: //dctps://do..o http///iioogg.../////oo 1 001111333855553 3313558.33333338333333 3313685
Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang. and Sanjiv Kumar. Pre-training tasks for umbedding-basee large-scale retrieval. In Proceedings of ICLR 2020, 2020. URL
mA4FDr
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the TREC 2019 deep learning track. CoRR, abs/2003.07820, 2020. URL Btttttss://arx orq/ abs 2000//2003.0m 07820
brq/abs/ae
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview the TREC 2020 deep learning track. CoRR, abs/2102.07662, 2021.
07662
Zhuyun Dai and Jamie Callan. Context-aware sentence/passage term importance estimation for first stage retrieval. CoRR, abs/1910.100877 2019a. httt::://arxv..or///aaaaa
10687
Zhuyun Dai and Jamie Callan. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of SIGIR 2019, pp. 985-988, 2019b. URL httph::////// /ttp:s://dd.... .org/
10. 11 145/3331184 3331303 303
Jacob Devlin, Ming-Wo Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings NAACI. 2019, pp. 4171- 4186, 2019. URL
Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval. In Proceed- ings of EMNLP 2021, pp. 981-993, 2021a. URL http:s///aaa.atttooooo.oo//////
emnlp ma maain... 75
Luyu Gao and Jamie Callan. Is your language model ready for dense representation fine-tuning? CoRR, 02//2104.822. 2021b. Ut/.a/aaess//arrrr...rrr//////2a
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. In Proceedings of ACL 2022, pp. 2843-2853, 2022. URL
org/10.18653/ /2212222222222 acc acl-long....o.... 2003
Luyu Gao, Zhuyun Dai, and Jamie Callan. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of NAACL. 2021, pp. 3030-3042, 2021a. URL
atttps://aclantho ogy org. httg:/2221.aaacmmaaaaa nwa.c-aallaaaaa in..mmiinnn...... 241
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for
NLP (RepL4NLP-2021) pp. 316-321, 2021b.
repl4nlp-1..
Sebastian Hofstätter, Sheng-Chich Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Effi- ciently teaching an effective dense retriever with balanced topic aware sampling. In Proceed- ings of SIGIR 2021, pp. 113-122, 2021. URL http:pppe//odo..rrro/001111////1
3462891
Wu Hong, Zhuosheng Zhang, Jinyuan Wang, and Hai Zhao. Sentence-aware contrastive learning for open-domain passage retrieval. In Proceedings ACL 2022, pp. 1062-1074, 2022. URL
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of EACL 2021, pp. 874-880, 2021. URL https:
//aclanthol wtw.thhoool...ogg///a.o eacl -aa main...
10

---

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. 1EEE Transaction's on Big Data, 7(3):535-544 2021. URL h 10.09/
TEDATA 2019 2921572
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Pro- ceedings of EMNLP 2020, pp. 6769-6781, 2020. URL https:://cclaantholooyyy 22000. emnnp-mmmmmm ma..a.i..55555
Alex Kendall, Yarin Gal, and Roberto Cipolla. weigh losses for scene geometry and semantics. In Proceedings of CVPR 2018, pp. 7482- 7491, 2018. URL BRLLhtttp///peeacceee
Multi-task learning using uncertainty to
thecvf. com content_etttrtttee 1app__2c88/tml_ Kendall Multi-Tase Laarss__Larrnin___a_e
2018 1at_a_221___pe______11_1
Omar Khattab and Matei Zaharia. Colhert: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of SIGIR 2020, pp. 39-48, 2020. URL https:
7/doi. 10. 1111111433330 3397271 3111//33377..1111111 3401075
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wa Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Trans- actions of the Association for Computational Linguistics, 7:452-466, 2019. URL https:
//aclanthology org 0019-11990000 1026
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of ACL 2019, pp. 6086-6096, 2019. URL
https://ack antho ogy. wrtpp/9-11119100000
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer, and Veselin Stoyanov, Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019,
1117
Zheng Liu and Yingxia Shao. Retromac: Pre-training retrieval- transfommers via masked auto-encoder. CoRR, abs/2205 2022. UKLLattpee//o....rr//0008888888
2205. 12035
Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie- Yan Liu, and Arnold Overwijk. Less is more: Pretrain a strong Siamese encoder for dense text retrieval using a weak decoder. In Proceedings of EMNLP 2021, pp. 2780-2791, 2021. URL attts://aclanthonn
org/2021 emnlp ma main. 220
Yuxiang Lu, Yiding Liu, Jiaxiang Liu, Yunsheng Shi, Zhengjie Huang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, Shuaiqiang Wang, Dawei Yin, and Haifeng Wang. Ernie-search: Bridging cross- encoder with dual-encoder via self on-the-fly distillation for dense passage retrieval. CoRR, abs/2205.09153. 2022. URL https:: //t::///oo..... wwwooor///........ 10. 48550/a /0.55////vv... 2205.09153
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. Pre-train discriminative text encoder for dense retrieval via contrastive span prediction. In Proceedings of SIGIR 2022,
pp. 848-858, 2022. URL httph:tt////t htttp:://dci.org: 10.
3531772
Kelong Mao, Zhicheng Dou, and Hongjin Qian. Curriculum contrastive context denoising for few- shot conversational dense retrieval. In Proceedings of SIGIR 2022, pp. 176-186, 2022. URL
/tdoi.
11411115//55 34773447744... 3531961
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of EMNLP 2020, pp. 5783--8757 2020. URL.
https: http//claanttoooo ho ogy. wwtpoog/22222 2mtp:/222mm..mmmmmmama. emnlp-ma 466
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and
Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Proceed- ings of the Workshop Cognitive Computation: Integrating neural and symbolic approaches 2016, volume 1773, 2016. http:htt://cce wttp://eerwwaaaaeee owo/oo///ooo.o 11w//1111111111111 paper9.pdf
1011///1111377/// CCCONIPS 2016

---

Rodrigo Nogueira and Jimmy Lin. From doc2query to doctttitquery https: cs www.wrloo. ca/ htt.a///aaaaaa///aaaaaaaaoaaooaa 2019 doc.ITTqqueyy.................
2019. URL
Lin
Rodrigo Frassetto Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. CoRR, 0bs//194400875, 2019. URL
08375
Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, Qiaoqiao She, Jing Liu, Hua Wu, and Haifeng Wang. Dureader retrieval: A large-scale chinese benchmark for passage retrieval from web search en- gine. CoRR, abs/2203.10232 2022. URL https:://do w.t./1...../0/.a...... wtt.xxiii..........
10232
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of NAACI. 202/, pp. 5835-5847, 2021. URL
htttps://aclantholooog 2ttp://claanntooooooooooo/////2 org/ naacl -m ma 466
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever,
et al. Language models are unsupervised multitask Iearners. 2019. URL https: 7/con www.oppeaa.cc.o.eee.oooo htwpee//bbetter-aaggeeeeeeee Saanguage-modelsss models/ models/LLnguagee modele_mdeelll_eeeeeeeeeeeeelele are__
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21:140:1-140:66 2020. URL http:
html
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. Learning to retrieve passages without supervision. In Proceedings of NAACL 2022, pp. 2687-2700, 2022. URL
naacl ma main. 193
Juan Ramos et al. Using tf-idf determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pp. 48, 2003.
Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. PAIR: Leveraging passage-centric similarity relation for im- proving dense passage retrieval. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2173-2183, 2021a. URL /tttps://aclaniee htt/a/aclaatttoooyyy. org/2011. Eindings-acl 191
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. Rocketqav2: A joint training method for dense passage retrieval and passage re- ranking. In Proceedings of EMNLP 2021, pp. 2825-2835, 2021b. URL htttps:///dot / 10. N0.18653/v1/2022
ma mmain... 224
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-33) 2009.
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamil- ton, and Bryan Catanzaro. End-to-end training of neural retrievers for open-domain question answering. In Proceedings ACL 2021, pp. 6648-6662, 2021. doi: 10.11865//1/2221.acll
519. UtRL:tttss///aLLLtttooooooooorr
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019. URL http://
arxiv.c waciivorrg/aae abs 0ttpb///991.1111100
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of NAACL 2022, pp. 3715-3734, 2022. URL /ttppb.......111.166//////222220 naacl -ma i 272
12

---

Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang- ERNIE 2.0: A continual pre-training framework for language understanding. In Proceedings of AAAI 2020. pp. 8968 8975, 2020. [RRLttt::///ss www.aaa.rrriiinnnenn wwtpee..ppp/AA....... AAAI/
article/view/cc
Nandan Thakur, Nils Reimers, Andreas Rückléé Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663, 2021.
226.224644400663 08663
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bow- man. GLUE: A multi-task benchmark and analysis platform for natural language understand- net ing. In Proceedings of ICLR 2019, 2019. URL https://opeerevellt wtt.://wwwweeterummmmo forum?id
rJ4km2R5t.7
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval. CoRR, abs/2207.02578, 2022. URL httts::/dci..rgee
22077.2578
Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. Contextual mask auto-encoder for dense passage retrieval. CoRR, abs/2208.07670, 2022. URL https:
/ //doi. 48550//rxiiv 2208. 07670
Tianboo Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michíhiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir R. Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. CoRR, abs/2201.05966, 2022.
URL BRLLhtpss//aaxii org. /twp/201..00000 05966
Lee Xiong, Chenyan Xiong. Ye Li, Kwok-Fun Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proceedings of ICLR 2021, 2021. URL
net forum?
id=zeFrfgy2ln
Canwen Xu, Daya Guo, Nan Duan, and Julian J. McAuley. Laprador: Unsupervised pretrained dense retriever for zero-shot text retrieval. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 3557-3569, doi. wrt.ool////..o. 186530011/11331//101 /ttp////222222////// findings-acl.225
2022. URL
Peilin Yang. Hui Fang, and Jimmy Lin. Anserini: Enabling the of lucene how.inooooooootmiinnnteiiiiiiiii retrivaal research. In Proceedings of SIGIR 2017, pp. 1253-12553 1256, 2017. URL/attss://abi.ozqg
1145/307713 3080721
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Proceedings of NeurIPS 2020, 2020.
URL bttpss.//procceedngss.eee neurips cc httppppar//22///aaat...
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. Repbert: Contextualized text
embeddings for first-stage retrieval. CoRR, abs/2006. 15498, 2020. URL https://arxiv.
hog//bss200s 1.e.15444 15498
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. In Proceedings of SIGIR 2021, pp. 1503-1512, 2021.
URL http::ttt://d/tt www./orggooooo 100. 11....144/34400 44/444443445555 3462880
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. Adversarial retriever-ranker for dense text retrieval. Proceedings of ICLR 2022, 2022. URL https:
MR MR7XubbUFB
Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Lan Luo, Ke Zhan, Enrui Hu, Xinyu Zhang, Hao Jiang, Zhao Cao, Fan Yu. Xin Jiang, Qun Liu, and Lei Chen. hyperlink-inducee pre-training for passage retrieval in open- domain question answering. In Proceedings of ACL 2022, pp. 7135-7146, 2022.
URL htRlhttps::// //ttss:://doo. http://oooooog///////// www.oro..0... htt.g//.111633//0 18653/6653777 /2022 2cl.222.ccl--
13

---

Table Experimental results on four NLU tasks from GLUE.
Model
CoLA
MRPC
STS-B
QQP
BERT
59.1
87.7
87.8
89.7
Ours 60.7
89.1
88.0
89.8
A BASELINES.
We compare our approach with variety of methods:
2017 is widely-used sparse retriever based on exact matching. DeepCT Daii & Callan 2019b) and doe docT5qqers T5query (Nogueira & Lin 2019) enhance BM25 with neural models.
BM25 Yang al.
2021p. TAS-B (Hofstätter et al. 2021 and STAR ZZann al.) 2021 are dense retrieval methods that adopt top-k hard negatives to improve training. RocketQA (Qu et al.) 2021]. AR2 (Zhang et al. 2022) and E.NNTE-search (Lu et al. 2022) utilize knowledge distillation technique that leverages a teacher model to guide the training of the dual-encoder retriever.
ANCE (Xiong al.
COIL (Gao et al.) 2021b). ColBERT (Khattab & Zaharia
2020 and ColBERTv2 (Santhanam
et al. 2022) utilize multiple representations for text retrieval.
SEED (Lu al. 2021). RetroMAE (Liu & Shao 2022), Condenser (Gao & Callan 2021b
PAIR (Ren et al. 2021a),
coCondenser (Gao & Callan 2022). CoT-MAE (Wu et al.) 2022) and
SimLM (Wang et al. 2022 design special pre-training tasks to improve the backbone models.
B IMPLEMENTATION DETAILS.
During pre-traine pre-training, we leverage BERT-base to initialize the shared encoder, and the multiple de- coders are randomly initialized two-layer Transformers. Following previous work (Gao & Callan 2022 Wu et al. 2022, Wang et al. 2022), we leverage the passages in MS-MARCO and NQ dataset the pre-training corpus of them, respectively. The pre-training steps setting to 120k. During fine-tuning, we also follow SimLM that first train Retriever, using BM25 negatives, then train Retriever, using the hard negatives mined by Retriever,, finally utilize cross-encoder based reranker D to perform knowledge distillation on the hard negatives mined by Retrieverz, to train Retrieveraiaii- Note that our pre-trained deep Transformer encoder leveraged to initialize the pa- rameters of Retriever, Retriever, and Retrieverasail- Our all other prreerpprraettess are the same as SimLM (Wang et al. 2022). All the experiments in this work are conducted on NVIDIA Tesla A100 GPUs
C NATURAL LANGUAGE UNDERSTANDING TASKS
In approach, have integrated multiple pre-training tasks for learning, our model is able to capture diverse knowledge from these tasks. In this part, we aim to evaluate if our pre-training methods also benefit for other tasks, iee, natural language understanding (NLU). We select the single-sentence and similarity tasks from the GLUE benchmark (Wang et al.) 2019) (i.e., CoLA, MRPC, STS-B and QQP), which focus on predicting the acceptability, similarity and paraphrase of sentences from different domains (e.g., news and misc). We fine-tune our pre- pre-trained trained model on these following the suggestions of the original BERT paper (Devlin)
tasks. and all the syper--arameters et al. 2019). As shown in Table g we can see that our approach can also improve the performance of BERT on these NLU tasks. It indicates that our multi-task pre-training can also enrich the useful knowledge about NLU tasks for the PLM.
D HYPER-PARAMETER TUNING.
Our approach has two important Syper-parameters, namely the masked rates of the deep encoder and multiple decoders, as they control the information bottleneck our approach. In this part, we
Following SimLM, we also leverage ELECTRA-bass initialize the reranker and train it using the mined negatives by Retrieverz-
14

---

Table 9: Performance comparison w.r.t. different masked rates in the encoder and decoder. We report MRR @ 10 of the Retriever, and Retriever, on the dev set of MS-MARCO,
Model
30%
En-50% De
15% En-50% De 50% En-50% De 30%
En-30% De
30% En-70% De
Retrieven:
38.3
37.9
37.6
37.5
38.0
Retriever: 40.4
39.9
39.7
39.8
39.9
set the masked rate in the encoder to be 15%, 30% and 50%, and that in decoders to be 30%, 50% and 70%. Table [ shows the evaluation results. First, we can see that our model is robust to these settings. Besides, when the masked rates of the encoder and decoders
different Syper-parameee are set to 30% and 50% respectively, our model performs slightly better than others. Therefore, we apply 30% and 50% the masked rates of the encoder and decoders.
15


Reviews:
[{"review_id": "oVd4Tx-Yvu", "reviewer": "Reviewer_JgYb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed multiple pre-training tasks to be used with the bottlenecked masked autoencoder architecture, designed specifically for dense retrieval. The three types of tasks are corrupted passages recovering, related passages recovering, and pre-trained language model (PLM) outputs recovering. The tasks are integrated by formulating each tasks in a unified text-to-text format. Each pre-training task has its own task-specific shallow decoder. The claimed benefits is that the Transformer encoder is forced to compress the information into a dense vector. The empirical results show the proposed method outperforms competitive baselines.", "summary_of_the_review": "The paper proposed carefully designed pre-training tasks, a way to integrate multiple pre-training tasks and show competitive results empirically.", "strength_and_weaknesses": "Strengths\n1. The paper proposed integrating multiple pre-training tasks via a unified text-to-text format. The bottlenecked masked autoencoder also allows task-specific decoders.\n2. The results are competitive.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iDRRi3mgz7J", "reviewer": "Reviewer_3P7A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to pre-train a dense retriever with multiple tasks (five in total), which are all related to masked language modelling (multi-decoder). Results on MS-Marco show moderate improvement with respect to baselines.\n", "summary_of_the_review": "This paper proposes a new pre-training for IR - even if the authors frame it as \"multi-task\", it is still heavily based on a masked language modelling. Results are good but not so impressive when looking at the SimLM baseline (which shares this idea of corruption loss) and especially at the quite expensive pre-training procedure.\n", "strength_and_weaknesses": "Strengths:\n\n- Unifying/studying the pre-training of information retrieval \n- Improved performances with respect to the state-of-the-art.\n- The proposed pre-training is simple to set up, and might be a basis for future pre-training\n\nWeaknesses:\n\n- it is not obvious why the fact that using a more homogeneous task is better: There is a bold statement p.2: \"due to the divergences of input formats and learning objectives among different tasks, an arbitrary integration of these tasks is inappropriate, which may even cause detrimental gradient interference, leading to performance degradation\" - how is your setting more appropriate? Why having multiple tasks be detrimental - I checked the references and they do not provide any reason (but it is true that the magnitude of the respective losses has to be set as in ref. Kendall et al. 2018).\n- no comparison on the BEIR dataset (which would show how robust the pre-training + training procedure is)\n- apart from an ablation, there is no further analysis of the 5 proposed losses, although they are combined directly with no discussion on why they should have the same importance.\n- No code is provided\n- Using DocT5 (POR loss) shows very moderate improvement in comparison to the cost: it relies on a query generation model (trained on MS Marco which is used for the experiments here...) and document generation (which is costly). I fail to see why it justifies the 0.4 drop in MRR@10 (especially since the variance of the results is not really known)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JNaMaG6kQL", "reviewer": "Reviewer_jK4d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a multi-task pretraining objective for dense retrieval models. By leveraging the bottlenecked masked autoencoder architecture, the encoder produces a single-vector embedding, which is used to condition the decoder models on three different types of pretraining tasks, which are all trained in parallel: recovery of corrupted (input) passages, recovery of related passages, and recovery of the outputs from an external pretrained LM.\nThe model is tested on standard web search and open domain QA datasets, and achieves state-of-the-art performance.", "summary_of_the_review": "The paper proposes a simple method that reaches state-of-the-art in dense retrieval. However, the ablations are not sufficient to convincingly show that the modeling contributions are actually responsible for this success. Hence, the paper would need a major revision before it can be accepted.", "strength_and_weaknesses": "Strengths:\n\n* simple model that works well\n* paper is easy to understand\n\nWeaknesses (**UPDATED AFTER REBUTTAL**):\n* the ablation results are not convincing:\n    * the improvements from each pretraining objective are relatively small\n    * ~could it be that the results are simply better because you effectively train on more examples the more pretraining objectives you have? Figure 2 suggests that convergence might not have been reached, and hence more pretraining objectives would mean that you actually do more training steps (in the sense of training examples) than what the x-axis shows. When removing each of the 5 pretraining objectives, you could control for this by instead training for 1/5 = 20% longer.~\n    * there is no qualitative analysis that shows us that the different pretraining objectives actually teach the model distinct \"skills\". This could be done by training a model with only a single pretraining objective and then doing an error analysis for each.\n* ~in your comparisons to SimLM, how do you make sure that the small differences in performance are not actually due to factors of variation other than the pretraining objective? Are all the model details the same between the two models? How do you make sure that both models receive equal treatment in terms of hyperparameter tuning?~\n* the model is of limited novelty; neither the bottlenecked model architecture, nor the multi-task pretraining idea are novel. There is some novelty in the concrete pretraining tasks, but their effect is unclear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject