SIMLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Year: 2023

Abstract:
In this paper, we propose SIMLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA (Clark et al., 2020), to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SIMLM only requires access to an unlabeled corpus and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets and show substantial improvements over strong baselines under various settings. Remarkably, SIMLM even outperforms multivector approaches such as ColBERTv2 (Santhanam et al., 2021) which incurs significantly more storage cost. Our code and model checkpoints are available at https://github.com/ microsoft/unilm/tree/master/simlm.

heading: 1 Introduction
text: Passage retrieval is an important component in applications like ad-hoc information retrieval, opendomain question answering (Karpukhin et al., 2020), retrieval-augmented generation (Lewis et al., 2020) and fact verification (Thorne et al., 2018). Sparse retrieval methods such as BM25 were the dominant approach for several decades, and still play a vital role nowadays. With the emergence of large-scale pre-trained language models (PLM) (Devlin et al., 2019), increasing attention is being paid to neural dense retrieval methods (Yates et al., 2021). Dense retrieval methods map both queries and passages into a low-dimensional vector space, where the relevance between the queries and passages are measured by the dot product or cosine similarity between their respective vectors.
Like other NLP tasks, dense retrieval benefits greatly from a strong general-purpose pre-trained language model. However, general-purpose pretraining does not solve all the problems. As shown in Table 1, improved pre-training techniques that are verified by benchmarks like GLUE (Wang et al., 2019) do not result in consistent performance gain for retrieval tasks. Similar observations are also made by Lu et al. (2021). We hypothesize that, to perform robust retrieval, the [CLS] vector used for computing matching scores should encode all the essential information in the passage. The next-sentence prediction (NSP) task in BERT introduces some supervision signals for the [CLS] token, while RoBERTa (Liu et al., 2019) and ELECTRA do not have such sequence-level tasks.
In this paper, we propose SimLM to pre-train a representation bottleneck with replaced language modeling objective. SimLM consists of a deep encoder and a shallow decoder connected with a representation bottleneck, which is the [CLS] vector in our implementation. Given a randomly masked text segment, we first employ a generator to sample replaced tokens for masked positions, then use both the deep encoder and shallow decoder to predict the original tokens at all positions. Since the decoder only has limited modeling capacity, it must rely on the representation bottleneck to perform well on this pre-training task. As a result, the encoder will learn to compress important semantic information into the bottleneck, which would help
2244
train biencoder-based 1 dense retrievers. Our pretraining objective works with plain texts and does not require any generated pseudo-queries as for GPL (Wang et al., 2022).
Compared to existing pre-training approaches such as Condenser (Gao and Callan, 2021) or coCondenser (Gao and Callan, 2022), our method has several advantages. First, it does not have any extra skip connection between the encoder and decoder, thus reducing the bypassing effects and simplifying the architecture design. Second, similar to ELECTRA pre-training, our replaced language modeling objective can back-propagate gradients at all positions and does not have [MASK] tokens in the inputs during pre-training. Such a design increases sample efficiency and decreases the input distribution mismatch between pre-training and fine-tuning.
To verify the effectiveness of our method, we conduct experiments on several large-scale web search and open-domain QA datasets: MSMARCO passage ranking (Campos et al., 2016), TREC Deep Learning Track datasets, and the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019). Results show substantial gains over other competitive methods using BM25 hard negatives only. When combined with mined hard negatives and cross-encoder based re-ranker distillation, we can achieve new state-of-the-art performance.

heading: 2 Related Work
text: Dense Retrieval The field of information retrieval (IR) (Manning et al., 2005) aims to find the relevant information given an ad-hoc query and has played a key role in the success of modern search engines. In recent years, IR has witnessed a paradigm shift from traditional BM25-based inverted index retrieval to neural dense retrieval (Yates et al., 2021; Karpukhin et al., 2020). BM25-based retrieval, though efficient and interpretable, suffers from the issue of lexical mismatch between the query and passages. Methods like document expansion (Nogueira et al., 2019) or query expansion (Azad and Deepak, 2019; Wang et al., 2023) are proposed to help mitigate this issue. In contrast, neural dense retrievers first map the query and passages to a low-dimensional vector space, and then perform semantic matching. Popular methods include DSSM (Huang et al., 2013), C-DSSM (Shen et al., 2014), and DPR (Karpukhin et al., 2020) etc. 1Also called dual-encoder / two-tower encoder.
Inference can be done efficiently with approximate nearest neighbor (ANN) search algorithms such as HNSW (Malkov and Yashunin, 2020).
Some recent works (Chen et al., 2021; Reimers and Gurevych, 2021; Sciavolino et al., 2021) show that neural dense retrievers may fail to capture some exact lexical match information. To mitigate this issue, Chen et al. (2021) proposes to use BM25 as a complementary teacher model, ColBERT (Khattab and Zaharia, 2020) instead replaces simple dot-product matching with a more complex token-level MaxSim interaction, while COIL (Gao et al., 2021) incorporates lexical match information into the scoring component of neural retrievers. Our proposed pre-training method aims to adapt the underlying text encoders for retrieval tasks, and can be easily integrated with existing approaches.
Pre-training for Dense Retrieval With the development of large-scale language model pre-training (Dong et al., 2019; Clark et al., 2020), Transformerbased models such as BERT (Devlin et al., 2019) have become the de facto backbone architecture for learning text representations. However, most pre-training tasks are designed without any prior knowledge of downstream applications. Chang et al. (2020) presents three heuristically constructed pre-training tasks tailored for text retrieval: inverse cloze task (ICT), body first selection (BFS), and wiki link prediction (WLP). These tasks exploit the document structure of Wikipedia pages to automatically generate contrastive pairs. Other related pretraining tasks include representative words prediction (Ma et al., 2021), contrastive span prediction (Ma et al., 2022), contrastive learning with independent cropping (Izacard et al., 2021), domainmatched pre-training (Oguz et al., 2022) or neighboring text pairs (Neelakantan et al., 2022) etc.
Another line of research builds upon the intuition that the [CLS] vector should encode all the important information in the given text for robust matching, which is also one major motivation for this paper. Such methods include Condenser (Gao and Callan, 2021), coCondenser (Gao and Callan, 2022), SEED (Lu et al., 2021), DiffCSE (Chuang et al., 2022), and RetroMAE (Liu and Shao, 2022) etc. Compared with Condenser and coCondenser, our pre-training architecture does not have skip connections between the encoder and decoder, and therefore forces the [CLS] vector to encode as
much information as possible. RetroMAE (Liu and Shao, 2022) is a concurrent work at the time of writing that combines a bottleneck architecture and the masked auto-encoding objective.

heading: 3 SimLM

heading: 3.1 Pre-training
text: For pre-training, we assume there is a collection of passages C = {xi}|C|i=1, where x denotes a single passage. Since our motivation is to have a general pre-training method, we do not assume access to any query or human-labeled data.
The overall pre-training architecture is shown in Figure 1. Given a text sequence x, its tokens are randomly replaced with probability p by two sequential operations: random masking with probability p denoted as x′ = Mask(x, p), and then sampling from an ELECTRA-style generator g denoted as Sample(g,x′). Due to the randomness of sampling, a replaced token can be the same as the original one. The above operations are performed twice with potentially different replace probabilities penc and pdec to get the encoder input xenc and decoder input xdec.
xenc = Sample(g, Mask(x, penc)) xdec = Sample(g, Mask(x, pdec)) (1)
We also make sure that any replaced token in xenc is also replaced in xdec to increase the difficulty of the pre-training task.
The encoder is a deep multi-layer Transformer that can be initialized with pre-trained models like BERT (Devlin et al., 2019). It takes xenc as input and outputs the last layer [CLS] vector hcls as a representation bottleneck. The decoder is a
2-layer shallow Transformer with a language modeling head and takes xdec and hcls as inputs. Unlike the decoder component in autoregressive sequenceto-sequence models, the self-attention in our decoder is bi-directional. The pre-training task is replaced language modeling for both the encoder and decoder, which predicts the tokens before replacement at all positions. The loss function is the token-level cross-entropy. The encoder loss Lenc is shown as follows:
min Lenc = − 1
|x|
|x|∑
i=1
log p(x[i] | xenc) (2)
Similarly for the decoder loss Ldec. The final pretraining loss is their simple sum: Lpt = Lenc+Ldec. We do not fine-tune the parameters of the generator as our preliminary experiments do not show any performance gain.
It is often reasonable to assume access to the target retrieval corpus before seeing any query. Therefore, we directly pre-train on the target corpus similar to coCondenser (Gao and Callan, 2022). After the pre-training finishes, we throw away the decoder and only keep the encoder for supervised fine-tuning.
Since the decoder has very limited modeling capacity, it needs to rely on the representation bottleneck to perform well on the pre-training task. For the encoder, it should learn to compress all the semantic information and pass it to the decoder through the bottleneck.

heading: 3.2 Fine-tuning
text: Compared to training text classification or generation models, training state-of-the-art dense retrieval models requires a relatively complicated procedure. In Figure 2, we show our
supervised fine-tuning pipeline. In contrast to previous approaches, our proposed pipeline is relatively straightforward and does not require joint training (Ren et al., 2021b) or re-building index periodically (Xiong et al., 2021). Each stage takes the outputs from the previous stage as inputs and can be trained in a standalone fashion.
Retriever1 Given a labeled query-passage pair (q+, d+), we take the last-layer [CLS] vector of the pre-trained encoder as their representations (hq+ ,hd+). Both the in-batch negatives and BM25 hard negatives are used to compute the contrastive loss Lcont:
− log φ(q +, d+)
φ(q+, d+) + ∑
ni∈N (φ(q+, ni) + φ(d +, ni))
(3) Where N denotes all the negatives, and φ(q, d) is a function to compute the matching score between query q and passage d. In this paper, we use temperature-scaled cosine similarity function: φ(q, d) = exp( 1τ cos(hq,hd)). τ is a temperature hyper-parameter and set to a constant 0.02 in our experiments.
Retriever2 It is trained in the same way as Retriever1 except that the hard negatives are mined based on a well-trained Retriever1 checkpoint.
Re-ranker is a cross-encoder that re-ranks the topk results of Retriever2. It takes the concatenation of query q and passage d as input and outputs a realvalued score θ(q, d). Given a labeled positive pair (q+, d+) and n−1 hard negative passages randomly
sampled from top-k predictions of Retriever2, we adopt a listwise loss to train the re-ranker:
− log exp(θ(q +, d+))
exp(θ(q+, d+)) + ∑n−1
i=1 exp(θ(q+, d − i ))
(4) The cross-encoder architecture can model the full interaction between the query and the passage, making it suitable to be a teacher model for knowledge distillation.
Retrieverdistill Although cross-encoder based reranker is powerful, it is not scalable enough for first-stage retrieval. To combine the scalability of biencoder and the effectiveness of cross-encoder, we can train a biencoder-based retriever by distilling the knowledge from the re-ranker. The reranker from the previous stage is employed to compute scores for both positive pairs and mined negatives from Retriever2. These scores are then used as training data for knowledge distillation. With n− 1 mined hard negatives, we use KL (KullbackLeibler) divergence Lkl as the loss function for distilling the soft labels:
Lkl =
n∑
i=1
piranker log piranker piret
(5)
where pranker and pret are normalized probabilities from the re-ranker teacher and Retrieverdistill student. For training with the hard labels, we use the contrastive loss Lcont as defined in Equation 3. The final loss is their linear interpolation: L = Lkl + αLcont.
Our pre-trained SimLM model is used to initialize all three biencoder-based retrievers but not the cross-encoder re-ranker. Since our pre-training
method only affects model initialization, it can be easily integrated into other more effective training pipelines.

heading: 4 Experiments

heading: 4.1 Setup
text: Datasets and Evaluation We use MS-MARCO passage ranking (Campos et al., 2016), TREC Deep Learning (DL) Track 2019 (Craswell et al., 2020a) and 2020 (Craswell et al., 2020b), Natural Questions (NQ) (Kwiatkowski et al., 2019; Karpukhin et al., 2020) datasets for training and evaluation. The MS-MARCO dataset is based on Bing search results and consists of about 500k labeled queries and 8.8M passages. Since the test set labels are not publicly available, we report results on the development set with 6980 queries. The NQ dataset is targeted for open QA with about 80k question-answer pairs in the training set and 21M Wikipedia passages. For evaluation metrics, we use MRR@10, Recall@50, and Recall@1k for MS-MARCO, nDCG@10 for TREC DL, and Recall@20, Recall@100 for the NQ dataset.
Implementation Details For pre-training, we initialize the encoder with BERTbase (uncased version). The decoder is a two-layer Transformer whose parameters are initialized with the last two
layers of BERTbase. The generator is borrowed from the ELECTRAbase generator, and its parameters are frozen during pre-training. We pre-train for 80k steps for MS-MARCO corpus and 200k steps for NQ corpus, which roughly correspond to 20 epochs. Pre-training is based on 8 V100 GPUs. With automatic mixed-precision training, it takes about 1.5 days and 3 days for the MS-MARCO and NQ corpus respectively.
For more implementation details, please check out the Appendix section B.

heading: 4.2 Main Results
text: We list the main results in Table 2 and 4. For the MS-MARCO passage ranking dataset, the numbers are based on the Retrieverdistill in Figure 2. Our method establishes new state-of-the-art with MRR@10 41.1, even outperforming multi-vector methods like ColBERTv2. As shown in Table 3, ColBERTv2 has a 6x storage cost as it stores one vector per token instead of one vector per passage. It also requires a customized two-stage index search algorithm during inference, while our method can utilize readily available vector search libraries.
The TREC DL datasets have more fine-grained human annotations, but also much fewer queries (less than 100 labeled queries). We find that using
different random seeds could have a 1%-2% difference in terms of nDCG@10. Though our model performs slightly worse on the 2019 split compared to coCondenser, we do not consider such difference as significant.
For passage retrieval in the open-domain QA setting, a passage is considered relevant if it contains the correct answer for a given question. In Table 4, our model achieves R@20 85.2 and R@100 89.7 on the NQ dataset, which are comparable to or better than other methods. For end-to-end evaluation of question answering accuracy, we will leave it as future work.
Though SimLM achieves substantial gain for biencoder-based retrieval, its success for re-ranking is not as remarkable. In Table 5, when used as initialization for re-ranker training, SimLM outperforms BERTbase by 0.6% but still lags behind ELECTRAbase.
Next, we zoom in on the impact of each stage in our training pipeline. In Table 6, we mainly compare with coCondenser (Gao and Callan, 2022). With BM25 hard negatives only, we can achieve MRR@10 38.0, which already matches the performance of many strong models like RocketQA (Qu et al., 2021). Model-based hard negative mining and re-ranker distillation can bring further gains. This is consistent with many previous works (Xiong et al., 2021; Ren et al., 2021b). We also tried an additional round of mining hard negatives but did not observe any meaningful improvement.
Based on the results of Table 6, there are many interesting research directions to pursue. For example, how to simplify the training pipeline of dense retrieval systems while still maintaining competitive performance? And how to further close the gap between biencoder-based retriever and crossencoder based re-ranker?

heading: 5 Analysis

heading: 5.1 Variants of Pre-training Objectives
text: Besides our proposed replaced language modeling objective, we also tried several other pre-training objectives as listed below.
Enc-Dec MLM uses the same encoder-decoder architecture as in Figure 1 but without the generator. The inputs are randomly masked texts and the pre-training objective is masked language modeling (MLM) over the masked tokens only. The mask rate is the same as our method for a fair comparison, which is 30% for the encoder and 50% for the decoder. In contrast, RetroMAE (Liu and Shao, 2022) uses a specialized decoding mechanism to derive supervision signals from all tokens on the
decoder side. Condenser is a pre-training architecture proposed by Gao and Callan (2021). Here we pre-train Condenser with a 30% mask rate on the target corpus. MLM is the same as the original BERT pretraining objective with a 30% mask rate. Enc-Dec RTD is the same as our method in Figure 1 except that we use replaced token detection (RTD) (Clark et al., 2020) as a pre-training task for both the encoder and decoder. This variant shares some similarities with DiffCSE (Chuang et al., 2022). The main difference is that the input for DiffCSE encoder is the original text, making it a much easier task. Our preliminary experiments with DiffCSE pre-training do not result in any improvement. AutoEncoder attempts to reconstruct the inputs based on the bottleneck representation. The encoder input is the original text without any mask, and the decoder input only consists of [MASK] tokens and [CLS] vector from the encoder. BERTbase just uses off-the-shelf checkpoint published by Devlin et al. (2019). It serves as a baseline to compare against various pre-training objectives.
The results are summarized in Table 7. Naive auto-encoding only requires memorizing the inputs and does not need to learn any contextualized features. As a result, it becomes the only pretraining objective that underperforms BERTbase. Condenser is only slightly better than simple MLM pre-training, which is possibly due to the bypassing effects of the skip connections in Condenser. Enc-Dec MLM substantially outperforms Enc-Dec RTD, showing that MLM is a better pre-training task than RTD for retrieval tasks. This is consistent with the results in Table 1. Considering the superior performance of RTD pre-trained models on benchmarks like GLUE, we believe further research efforts are needed to investigate the reason behind this phenomenon.

heading: 5.2 Effects of Replace Rate
text: In the experiments, we use fairly large replace rates (30% for the encoder and 50% for the decoder). This is in stark contrast to the mainstream choice
of 15%. In Table 8, we show the results of pretraining with different replace rates. Our model is quite robust to a wide range of values with 30%- 40% encoder replace rate performing slightly better. Similar findings are also made by Wettig et al. (2022).
One interesting extreme scenario is a 100% replace rate on the decoder side. In such a case, the decoder has no access to any meaningful context. It needs to predict the original texts solely based on the representation bottleneck. This task may be too difficult and has negative impacts on the encoder.

heading: 5.3 Effects of Pre-training Steps
text: Since pre-training can be costly in terms of both time and carbon emission, it is preferred to have an
objective that converges fast. Our proposed method shares two advantages of ELECTRA (Clark et al., 2020). First, the loss is computed over all input tokens instead of a small percentage of masked ones. Second, the issue of input distribution mismatch is less severe than MLM, where the [MASK] token is seen during pre-training but not for supervised fine-tuning. In Figure 3, our method achieves competitive results with only 10k training steps and converges at 60k, while MLM still slowly improves with more steps.

heading: 5.4 On the Choice of Pre-training Corpus
text: For a typical retrieval task, the number of candidate passages is much larger than the number of labeled queries, and many passages are never seen during training. Take the NQ dataset as an example, it has 21M candidate passages but only less than 80k question-answer pairs for training. In the experiments, we directly pre-train on the target corpus. Such pre-training can be regarded as implicit memorization of the target corpus in a query-agnostic way. One evidence to support
this argument is that, as shown in Table 7, simple MLM pre-training on target corpus can have large performance gains.
An important research question to ask is: will there be any benefits of our method when pretraining on non-target corpus? In Table 10, the largest performance gains are obtained when the corpus matches between pre-training and finetuning. If we pre-train on the MS-MARCO corpus and fine-tune on the labeled NQ dataset or the other way around, there are still considerable improvements over the baseline. We hypothesize that this is due to the model’s ability to compress information into a representation bottleneck. Such ability is beneficial for training robust biencoder-based retrievers.

heading: 5.5 Case Analysis
text: To qualitatively understand the gains brought by pre-training, we show several examples in Table 9. The BERTbase retriever can return passages with high lexical overlap while missing some subtle but key semantic information. In the first example, the retrieved passage by BERTbase contains keywords like “boy”, “Winnie the Pooh”, but does not answer the question. In the second example, there is no routing number in the BERTbase retrieved passage, which is the key intent of the query. Our proposed pre-training can help to learn better semantics to answer such queries. For more examples, please check out Table 14 in the Appendix.

heading: 6 Conclusion
text: This paper proposes a novel pre-training method SIMLM for dense passage retrieval. It follows an encoder-decoder architecture with a representation bottleneck in between. The encoder learns to compress all the semantic information into a dense vector and passes it to the decoder to perform well on the replaced language modeling task. When used as initialization in a dense retriever training pipeline, our model achieves competitive results on several large-scale passage retrieval datasets.
For future work, we would like to increase the model size and the corpus size to examine the scaling effects. It is also interesting to explore other pre-training mechanisms to support unsupervised dense retrieval and multilingual retrieval.
Limitations
One limitation of SimLM is that it can not be used as a zero-shot dense retriever, since the pre-training framework does not have any contrastive objective. Fine-tuning on labeled data is necessary to get a high-quality model. On the other hand, although SimLM pre-training is quite efficient thanks to the replaced language modeling objective, it still requires extra computational resources to train the model.
Ethical Considerations
If the retrieval corpus contains some offensive or biased texts, they could be exposed to users under certain queries through our dense retriever. To deal with such risks, we need to introduce toxic text classifiers or manual inspection to exclude such texts from the corpus.

heading: A Details on Table 1
text: The numbers for the GLUE benchmark are from the official leaderboard 2. Note that the leaderboard submission from BERT does not use ensemble, so the comparison is not entirely fair. However, this does not change our conclusion that BERT generally performs worse than RoBERTa and ELECTRA on NLP tasks. For the MS-MARCO dataset, we fine-tune all the pre-trained models with BM25 hard negatives only. For BERT and RoBERTa, we use the same hyperparameters as discussed in Section 4.1. For ELECTRA, we train for 6 epochs with a peak learning rate 4 × 10−5 since it converges much slower.
B Implementation Details
The hyper-parameters for our proposed pretraining and fine-tuning are listed in Table 11 and 13, respectively. For supervised fine-tuning, One shared encoder is used to encode both the query and passages. We start with the official BM25 hard negatives in the first training round and then change to mined hard negatives. During inference, given a query, we use brute force search to rank all the passages for a fair comparison with previous works. The generator is initialized with the released one by ELECTRA authors 3, and its parameters are 2 https://gluebenchmark.com/leaderboard 3https://huggingface.co/google/ electra-base-generator
frozen during pre-training. All the reported results are based on a single run, we find that the numbers are quite stable with different random seeds.
For fine-tuning on the NQ dataset, we reuse most hyper-parameters values from MS-MARCO training. A few exceptions are listed below. We finetune for 20k steps with learning rate 5×10−6. The maximum length for passage is 192. The mined hard negatives come from top-100 predictions that do not contain any correct answer.
C Variants of Generators
In the ELECTRA pre-training, the generator plays a critical role. Using either a too strong or too weak generator hurts the learnability and generalization of the discriminator.
We also tried several variants of generators. In Table 12, “frozen generator” keeps the generator parameters unchanged during our pre-training, “joint train” also fine-tunes the generator parameters, and “joint train w/ random init” uses randomly initialized generator parameters. We do not observe any significant performance difference between these variants. In our experiments, we simply use the “frozen generator” as it has a faster training speed.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethical Considerations section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction section
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 2 Section 4.1 setup
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. The datasets we use are well-known and widely used in the research community.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The datasets we use are created for dense retrieval, so it is kind of obvious that it is consistent with their intended use.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We do not collect new datasets.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1 setup
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1 Setup
C 3 Did you run computational experiments? Section 4 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.1 Setup Appendix Section B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.1 Setup Appendix Section B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix Section B.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix Section B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, we would like to increase the model size and the corpus size to examine the scaling effects. It is also interesting to explore other pre-training mechanisms to support unsupervised dense retrieval and multilingual retrieval. Limitations
One limitation of SimLM is that it can not be used as a zero-shot dense retriever, since the pre-training framework does not have any contrastive objective. Fine-tuning on labeled data is necessary to get a high-quality model. On the other hand, although SimLM pre-training is quite efficient thanks to the replaced language modeling objective, it still requires extra computational resources to train the model. Ethical Considerations
If the retrieval corpus contains some offensive or biased texts, they could be exposed to users under certain queries through our dense retriever. To deal with such risks, we need to introduce toxic text classifiers or manual inspection to exclude such texts from the corpus