Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval (2024)

Abstract: Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query. Recent studies have highlighted the potential of a strong generative retrieval model, trained with carefully crafted pre-training tasks, to enhance downstream retrieval tasks via finetuning. However, the full power of pre-training for generative retrieval remains underexploited due to its reliance on pre-defined static document identifiers, which may not align with evolving model parameters. In this work, we introduce BootRet, a bootstrapped pre-training method for generative retrieval that dynamically adjusts document identifiers during pretraining to accommodate the continuing memorization of the corpus. BootRet involves three key training phases: (i) initial identifier generation, (ii) pre-training via corpus indexing and relevance prediction tasks, and (iii) bootstrapping for identifier updates. To facilitate the pre-training phase, we further introduce noisy documents and pseudo-queries, generated by large language models, to resemble semantic connections in both indexing and retrieval tasks. Experimental results demonstrate that BootRet significantly outperforms existing pre-training generative retrieval baselines and performs well even in zero-shot settings.

heading: 1 Introduction
text: Document retrieval is an important task with widespread applications, such as question answering (Karpukhin et al., 2020a; Lee et al., 2019) and fact verification (Chakrabarty et al., 2018; Olivares et al., 2023), which aims to retrieve candidate documents from a huge document collection for a given query (Gao and Callan, 2022; Nie et al., 2020). Currently, the dominant implementation is dense retrieval (Xiong et al., 2017; Zhan et al., 2020a),
*Corresponding author.
which encodes the query and documents into dense embedding vectors to capture rich semantics. Generative retrieval. An emerging alternative to dense retrieval in document retrieval is generative retrieval (GR) (Tang et al., 2023; Tay et al., 2022). It employs a sequence-to-sequence (Seq2Seq) architecture to generate relevant document identifiers (docids) for queries. In this manner, the knowledge of all documents in the corpus is encoded into the model parameters, similar to the human cognitive associative mechanism (Anderson and Bower, 2014; Kounios et al., 2001). To achieve this, GR involves two basic operations (Tay et al., 2022): (i) indexing, which memorizes the entire corpus by associating each document with its identifier, and (ii) retrieval, which uses the indexed corpus information to produce a ranked list of potentially relevant docids for a given query.
Using general language models, such as BART (Lewis et al., 2019) and T5 (Raffel et al., 2020), as the base Seq2Seq model has become a popular choice in GR (Bevilacqua et al., 2022; De Cao et al., 2021; Zhuang et al., 2023). On top of this, some work has designed pre-training objectives for GR. For example, Zhou et al. (2022) proposed indexingand retrieval-based pre-training tasks; document pieces or pseudo-queries are used as input, and docids (e.g., product quantization code) are predicted as output with maximum likelihood estimation (MLE). Similarly, Chen et al. (2022) proposed retrieval-based tasks, which aim to construct and learn pairs of pseudo-queries and docids (i.e., Wikipedia titles) from the corpus. These works demonstrate that applying specialized pre-trained models to GR yields superior results compared to using general language models. Research challenges. While pre-training methods have shown their effectiveness, important limitations remain in the following: (i) The construction process of pre-defined docids is independent from
10303
the pre-training process. This results in a semantic gap between both processes, which could potentially hinder the retrieval performance. (ii) Docids remain unchanged during pre-training. If the initial docids are not suitable, they cannot be further adjusted after the training begins. Consequently, it may become challenging to learn semantics and relationships between documents, impeding the achievement of satisfactory retrieval performance. (iii) Existing pre-training methods do not explicitly consider the interrelations between document-docid or query-docid pairs. The widely-used MLE objective may result in difficulties in distinguishing among similar documents and docids. Therefore, we argue that the model should enhance its discriminative and generalization ability. Approach. To address these challenges, we introduce a general bootstrapped pre-training method for GR, called BootRet. Our objective is to dynamically adjust docids in accordance with the evolving model parameters during pre-training. The key idea is inspired by that the human brain updates the organization of existing knowledge to better match updated goals or contents in learning (Mack et al., 2016). BootRet includes three key steps: (i) Initial docid generation. We leverage the encoder of the initial model to encode documents and then obtain the product quantization code (Ge et al., 2013; Zhan et al., 2021) as the initial docids. (ii) Pre-training. We design two pre-training tasks, i.e., corpus indexing task and relevance predication task. The corpus indexing task aims to memorize corpus information and distinguish among similar documents and docids. We construct pairs of original documents and corresponding identifiers to simulate the indexing operation. To enhance discrimination and generalization, we use a large language model (LLM) to generate noisy documents similar to the originals, creating pairs of noisy documents and identifiers. Besides, we design contrastive losses to help the model memorize and contrast these pairs. The relevance prediction task aims to learn relevance information from the corpus. We construct pairs of pseudo-queries and relevant docids to simulate the retrieval operation. We also use a LLM to generate high-quality pseudo– queries for original documents as input and design a contrastive loss for the model to predict and contrast docids. These two tasks are jointly learned, with the docids remaining fixed throughout this process. (iii) Enhanced bootstrapping. The encoder of the model pre-trained with the above two tasks is further used to encode documents, updating document representations, and then updating the PQ code, i.e., docids. These updated docids are further used to retrain the model based on the pre-training tasks. Steps (ii) and (iii) iteratively update the model parameters and docids. We pre-train BootRet based on two kinds of large scale text corpus, i.e., MS MARCO (Nguyen et al., 2016) and Wikipedia (Wikipedia, 2022). We then fine-tune BootRet on two representative downstream datasets widely used in GR research. The empirical experimental results show that BootRet can achieve significant improvements over strong GR baselines. Contributions. Our main contributions are: (i) We propose a bootstrapped pre-training framework for GR to iteratively update the model parameters and docids. (ii) BootRet demonstrates superior performance in downstream retrieval tasks. For instance, on the MS MARCO dataset, it outperforms the strong pre-training GR baseline, Ultron (Zhou et al., 2022), by 11.8% in terms of Hits@1. (iii) Additionally, BootRet exhibits better zero-shot performance than other general language models.

heading: 2 Related Work
text: Generative retrieval. GR marks a new paradigm in document retrieval that generates identifier strings of documents as the retrieval target (Metzler et al., 2021; Tay et al., 2022). The current design of docids can be categorized into two types. (i) Pre-defined static docids. They remain unchanged during training, such as document titles (De Cao et al., 2021), URLs (Ren et al., 2023), product quantization code (Chen et al., 2023; Mehta et al., 2023). This design is simple and shows decent performance (Chen et al., 2022), but the pre-defined process is independent of training. (ii) Learnable docids. They are optimized jointly with the retrieval task (Sun et al., 2023; Wang et al., 2023). Though these docids are dynamic, their optimization primarily targets retrieval. Nevertheless, docids serve functions in both indexing and retrieval. In addition to widely-used supervised learning approaches (Sun et al., 2023; Zhang et al., 2023; Zhou et al., 2023), recent studies (Chen et al., 2022; Zeng et al., 2023; Zhou et al., 2022) have explored pre-training for GR. However, each study adopts fixed docids, ignoring the potential mismatch between docids and the updated model. In contrast, our work dynamically updates both the docids and
the evolving model to enhance the effectiveness. Bootstrapping. The idea of bootstrapping training methods have garnered significant interest in various natural language processing tasks (Deepika and Geetha, 2021; Song and Roth, 2014; Wu et al., 2009). The approach involves generating new training data or information based on the previous model to iteratively enhance its capabilities. While the techniques are sometimes used in conjunction with supervised learning (Deepika and Geetha, 2021; Song and Roth, 2014), our scenario involves an unlabeled corpus without ideal docids. Thus, we adopt a more unsupervised approach, iteratively refining the GR model and docids. Dense retrieval. Dense retrieval (Gao and Callan, 2022; Karpukhin et al., 2020a) is currently the de facto solution for document retrieval. It focuses on representing documents and queries as dense vectors in continuous spaces, capturing semantic relationships. Efficient vector search is facilitated by approximate nearest neighbor (Xiong et al., 2020) algorithms. Further enhancements include using pre-trained models within a dual-encoder architecture (Nie et al., 2020; Zhan et al., 2020b) and hard negative mining techniques (Karpukhin et al., 2020b; Zhan et al., 2020a). Compared to dense retrieval, GR could achieve end-to-end global optimization. However, its current performance lags behind state-of-the-art methods in dense retrieval.

heading: 3 Method
text: This section introduces the details of the BootRet model proposed in this paper. As shown in Figure 1, (i) Given a corpus D = {d1, . . . , d|D|}, we first construct an initial docid idi for each document di in D. The initial docid set is denoted as I0D. We employ an encoder-decoder language model as the base model, where initial parameters are denoted as θ0. (ii) Then, while keeping I0D unchanged, we carefully design two pre-training tasks. During pre– training, the model parameters are updated from θ0 to θ1. (iii) Subsequently, fixing θ1, we update the docids to I1D, thus completing one iteration. The updated docids can be used to further retrain the model for a next iteration. We define the t-th iteration as updating the model parameters from θt−1 to θt with fixed It−1D in Step (ii), and then based on fixed θt, updating It−1D to ItD in Step (iii).

heading: 3.1 Model Architecture
text: Like previous GR research (Chen et al., 2022; Tay et al., 2022; Wang et al., 2022), we leverage a
transformer-based model comprising: (i) An encoder, a bidirectional encoder to encode documents or pseudo-queries. (ii) An identifier decoder, operating through a sequential generation process to produce document identifiers. We initialize the model with T5-base (Raffel et al., 2020), and the initial model parameter is denoted as θ0.

heading: 3.2 Initial Docid Generation
text: Docids with semantic ties to the document content aid the model’s learning (Tay et al., 2022). For effective bootstrapping, docids need efficient updates based on the model’s progress. Considering these needs, we choose the widely used PQ code (Chen et al., 2023) as the docid.
Specifically, we first encode all the documents to obtain document vectors with the encoder of θ0. Following (Zhou et al., 2022), vectors are evenly divided into g groups. For each group, we apply the K-means clustering algorithm to obtain k cluster centers. Then, the docid can be represented by cluster indices of length g corresponding to the clusters. And we denote the initial docid set as I0D. To facilitate the generation of docids, we include all cluster indices from all groups obtained in the docid generation process as new tokens added to the model vocabulary. I0D will be used for subsequent iterative pre-training and updates.

heading: 3.3 Pre-training Tasks
text: The core idea is to construct pseudo documentdocid pairs and query-docid pairs to simulate the indexing and retrieval operations, respectively. Our two pre-training tasks are: (i) Corpus indexing task. We first construct pairs of original documents and their corresponding identifiers. For original documents, we use a LLM to construct similar but noisy documents D̃. d̃hi ∈ D̃ is the h-th noisy version of di. And we design multiple losses to guide the model to learn the associations between original or noisy documents and their identifiers. (ii) Relevance prediction task. We use a LLM to construct pseudo-queries Q, and pair them with relevant docids. For di, we construct a total of X queries, and qxi ∈ Q denotes the x-th pseudo-query.

heading: 3.3.1 Corpus Indexing Task
text: We introduce the construction of noisy documents and pre-training objectives in detail. Noisy document construction. The noisy documents should maintain semantic consistency with the originals while remaining distinguishable. We
propose leveraging a LLM to effectively achieve this. Inspired by (Raffel et al., 2020), we design the following four prompts to guide LLM generation: • A synonym replacement prompt: “Replace some words in the following document with their synonyms while maintaining the overall semantic meaning: {d}.” • A sentence removal prompt: “Remove one or more sentences from the following document, while maintaining the overall semantic meaning: {d}.” • A sentence shuffling prompt: “Rearrange the sentences in the following document to create a new flow, while maintaining the overall semantic meaning: {d}. • A word masking prompt: Mask some words with [Masked] in the following document, while maintaining the overall semantic meaning: {d}.” Combining these prompts with an original document as the input, LLM generates four noisy documents, sharing the same docid with the original. Pre-training objective. In the t-th iteration, the objective consists of three parts as the follows. • Semantic consistency loss: It aims at maintain-
ing overall semantic consistency between original and noisy documents. Specifically, in a minibatch, there are a total of 4N document-docid
pairs, where N pairs correspond to the original pairs, and each original document has four noisy pairs. This loss LSC(D, D̃; θt−1) is defined as:
N∑
i=1
4∑
h=1
1− sim(Enc(di),Enc(d̃hi )), (1)
where θt−1 denotes model parameters of the previous iteration and sim(·) is the cosine function. • Contrastive losses for corpus indexing: Conditioned on original document-docid pairs, we encourage the model to generate a docid that corresponds to the document rather than the docids of other documents. In the same mini-batch, we aim for the model to generate the docid corresponding to the document with a higher probability than generating others. Inspired by contrastive learning (Khosla et al., 2020), this loss LC1(D, ID; θt−1) is formalized as:
− N∑
i=1 log exp(P (idi | di)/τ)∑N j=1 exp(P (idj | di)/τ) , (2)
where τ is the temperature hyperparameter. P (idi | di) is the generated likelihood probability of idi conditioned on di. Similarly, for
noisy pairs, the loss LC2(D̃, ID; θt−1) is:
− N∑
i=1
4∑
h=1
log exp(P (idi | d̃hi )/τ)∑N j=1 exp(P (idj | d̃hi )/τ) . (3)
The pre-training objective of the corpus indexing task LCI(D, D̃, ID; θt−1) is a weighted sum of the three aforementioned losses, denoted as:
LSC(·) + αLC1(·) + βLC2(·), (4)
where α and β are hyperparameters.

heading: 3.3.2 Relevance Prediction Task
text: We introduce the construction process of pseudoqueries, and the pre-training objective. Pseudo-query construction. To generate highquality pseudo-queries for the original documents, we employ a LLM using the prompt: “Given the following document {d}, generate {X} insightful queries that a reader might have after reading the content. Ensure the queries cover key concepts.” When the prompt is combined with a document di and the required number of pseudo-queries X as input, we obtain well-written pseudo-queries. They share the same docids as the input original document. Pre-training objective. Similarly, we ensure that the model tends to generate relevant docids than irrelevant ones. In the same mini-batch, the loss LRP (Q, IQ; θt−1) in the t-th iteration is:
− N∑
i=1
X∑
x=1 log exp(P (idi | qxi )/τ)∑N j=1 exp(P (idj | qxi )/τ) . (5)

heading: 3.3.3 Joint Learning
text: We jointly pre-train the model with two above objectives and two sequence generation objectives. In the t-th iteration, the overall loss LPre(D, D̃, ID,Q, IQ; θt−1) is :
γLCI(·) + ρLRP (·) + λLID(·) + λLRE(·), (6)
where γ, ρ and λ are hyperparameters; LID(D, D̃, ID; θt−1) is the widely used standard MLE loss based on document-docid pairs:
− |D|∑
i=1
logP (idi |di)− |D|∑
i=1
4∑
h=1
logP (idi | d̃hi ). (7)
Note, Eq. (2) and Eq. (3) ensure that the model’s probability of generating the corresponding docid is greater than generating other docids. Eq. (7)
does not explicitly contrast with other docids. LRE (Q, ID; θt−1) is based on query-docid pairs:
− |Q|∑
i=1
logP (idi | qi). (8)
During training, we construct two types of batch data. One type has original and noisy documentsdocid pairs, optimized using Eq. (4) and Eq. (7). The other type has pairs of the pseudo-query and relevant docid only, optimized using Eq. (5) and Eq. (8). After jointly training during the t-th iteration, θt−1 updates to θt with docids fixed.

heading: 3.4 Enhanced Bootstrapping Strategy
text: Based on the updated θt, we introduce how to update docids It−1D and retrain the model. Docid update. Fixing θt, we use the encoder of θt to encode documents as in Section 3.2, to update docids of the previous iteration It−1D , to ItD. We refer to the version following the initial iteration’s completion (i.e., I1D and θ1) as BootRet-Bs. Retrain the model. To proceed to the next iteration, we retrain the model with ItD as described in Section 3.3. After multiple iterations, we achieve continuous dynamic alignment and enhancement. We refer to this version as BootRet-Mt.

heading: 4 Experimental Settings
text: Pre-training corpus. For pre-training we use two large, publicly available corpora: (i) English Wikipedia, which contains tens of millions of well-written documents and we downloaded this dump (Wikipedia, 2022) for pre-training, and (ii) the MS MARCO Document Collection (Nguyen et al., 2016), which has about 3 million documents extracted from web documents using the Bing search engine. For each corpus, we sample 500K documents and generated four noisy documents and five pseudo-queries, i.e., X , for each document. This results in 2.5M documents and 2.5M pseudo-queries for pre-training. BootRet-BSWiki and BootRet-BSMS denote the model pre-trained on Wikipedia and MS MARCO, respectively. Downstream retrieval datasets. We leverage two representative retrieval datasets. (i) MS MARCO Document Ranking dataset (Nguyen et al., 2016). Following the setup of (Zhou et al., 2022), we sample a subset of 300K documents for experimentation, denoted as MS 300K, containing 360K training queries, 6980 evaluation queries. These
documents do not overlap with the ones used in pre– training. (ii) Natural Question (NQ) (Kwiatkowski et al., 2019) has real questions and Wikipedia documents, having about 228K documents with 307K training queries and 7.8K test queries. Baselines. Following typical GR research (Tay et al., 2022; Wang et al., 2022), we examine three types of baseline: (i) Sparse retrieval baselines: BM25 (Robertson et al., 1995), and DocT5Query (Nogueira and Lin, 2019b). (ii) Dense retrieval baselines: RepBERT (Zhan et al., 2020b), DPR (Karpukhin et al., 2020a), and ANCE (Xiong et al., 2020). (iii) Advanced GR baselines: DSI (Tay et al., 2022), GENRE (De Cao et al., 2021), SEAL (Bevilacqua et al., 2022), DSI-QG (Zhuang et al., 2023), NCI (Wang et al., 2022), Ultron-PQ (Zhou et al., 2022), Corpusbrain (Chen et al., 2022), GenRet (Sun et al., 2023), and NOVO (Wang et al., 2023) . For more details on our baselines, please refer to Appendix A.1. Evaluation metrics. Following GR work (Li et al., 2023; Tay et al., 2022), for NQ, we use hit ratio (Hits@K) with K = {1, 10} as the metric. For MS 300K, we also use mean reciprocal rank (MRR@K) with K = {3, 20} (Li et al., 2023). Implementation details. For pre-training, we use the LLaMA-13b model (Touvron et al., 2023) to generate noisy documents and pseudo-queries. We initialize our model with T5-base (220M) (Raffel et al., 2021). For docids, we set the length g of PQ codes to 24, the number of clusters k to 256, and the dimension of vectors to 768. The hyperparameters for pre-training are set to α = β = λ = 1, γ = ρ = 2 and τ = 0.2. The batch size is 256. The Adam optimizer with a learning rate of 5e-5 is used, and the sequence length of documents is set to 512. The max training step is 500K, with the first iteration occurring at step 100K, followed by iterations every 40K steps thereafter.
For fine-tuning, we use the pre-trained model obtained from the last iteration to generate docids. Models are further fine-tuned with document-docid pairs and labeled query-docid pairs with MLE (Tay et al., 2022). Following (Wang et al., 2022; Zhou et al., 2022), we additionally generate 10 pseudoqueries for each document to enhance training. We set the learning rate as 1e-3, and the max training steps as 30K. Other settings remain consistent with the pre-training stage.
All models are trained on eight NVIDIA Tesla A100 80GB GPUs. For inference, we build a pre-
fix trie (De Cao et al., 2021) for docids and use constrained beam search with 20 beams to decode docids. For more details, please see Appendix A.2.

heading: 5 Experimental Results
text: This section presents the experimental findings.

heading: 5.1 Main Results
text: The comparison between our BootRet and baselines on MS 300K and NQ are shown in Table 1 and Table 2, respectively. We observe: (i) Dense retrieval baselines generally outperform sparse retrieval baselines, indicating that dense vectors capturing rich semantics are more beneficial for retrieval. (ii) Dense retrieval baselines outperform naive GR methods, such as DSI and SEAL, demonstrating the challenge of learning with only labeled data for GR. (iii) DSI-QG and NCI with data augmentation perform better than dense retrieval baselines, suggesting that GR requires more labeled data. (iv) Pre-trained baselines, i.e., Ultron and Corpusbrain, outperform supervised learning GR baselines, highlighting the necessity of pre-training for GR. (v) Our BootRet-MtWiki and BootRet-MtMS outperform base versions and Ultron, demonstrating the effectiveness of bootstrapped pre-training
with dynamic identifiers. (vi) In MS300K, our BootRet-Bs does indeed perform slightly worse compared to strong GR baselines such as UniGen, GenRRL, GenRet, NOVO, and ASI. However, the performance of BootRet-Mt is better than them, which also demonstrates the effectiveness of our approach. Similar conclusions are observed in NQ. (vii) BootRet-BsMS performs better on MS 300K than BootRet-BsWiki , while the opposite is observed on NQ, indicating that the performance of pre-trained models improves when downstream data and pre-training corpora are more similar. Additionally, Table A.3 in the appendix, shows that the performance of GR methods lags behind crossencoder methods, suggesting ample room for exploration in GR.

heading: 5.2 Ablation Study
text: To analyze the impact of each part of BootRet, we conduct ablation study on the Wikipedia pretraining corpus. From Table 3, we observe the following: (i) When not using dynamic identifiers (i.e., the 2nd row), wherein the model solely undergoes repeated pre-training using fixed docids, the performance significantly deteriorates compared to BootRet-MtWiki , affirming the effectiveness of
dynamic identifiers. (ii) When pre-training is not performed, and docids are directly obtained using the initial T5-base model (i.e., the 4th row), the model’s performance is lower than that of Ultron. This underscores the necessity of pre-training for GR. (iii) When pre-training does not involve the retrieval prediction task or corpus indexing task (i.e., the 5th-6th rows), the performance is lower than BootRet-BsWiki (i.e., the 3rd row). This confirms that pre-training should consider both relevance and corpus information. (iv) Not learning the corpus indexing task (i.e., the 6th row) leads to better performance compared to Ultron, indicating that the contrastive loss in the retrieval prediction task enhances the discriminative ability. (v) When the corpus indexing task does not use noisy documents (i.e., the 7th row), the performance is even lower, demonstrating that both noisy documents and contrastive losses contribute to discriminating similar documents and docids. (vi) When not using contrastive losses, i.e., the 8th row, where pre-training solely uses MLE losses (Eq. (7) and (8)) and Eq. (1), there is a significant decrease in performance compared to BootRet-BsWiki , indicating the effectiveness of contrastive losses. The ablation results based on the MS MARCO pre-training corpus show similar trends, as shown in Table 4.

heading: 5.3 Zero- and Low-resource Settings
text: To show whether BootRet can perform well with limited data, we randomly sample 2K, 4K, 6K, and 8K queries from the training set of both datasets. From Figure 2, we observe the following: (i) Under the zero-shot setting, where the model learns solely from the corpus without annotated queries, BootRet-BsWiki initially performs worse than Ultron on MS 300K. However, as fine-tuning with annotated queries progresses, BootRet-BsWiki
quickly surpasses Ultron. This is possibly due to Ultron directly pre-training on the downstream dataset’s corpus, while BootRet-BsWiki ’s pre-training corpus differs significantly from MS 300K. It also indicates that BootRet-BsWiki requires less annotated data to achieve rapid performance improvement. (ii) Under the low-resource setting, both base versions of BootRet exhibit performance gaps compared to BM25, highlighting the importance of annotated data for GR. (iii) Both versions of BootRet-Mt demonstrate better performance over base versions. Additionally, they achieve performance comparable to BM25 at approximately 1.3%, i.e., 5K, queries fine-tuning on MS 300K. Similar trends are observed for all methods on NQ, but all GR models perform worse than BM25.

heading: 5.4 Impact of the Number of Iteration
text: The iteration of updating docids and model parameters is important in our proposed bootstrapping pre-training method. We analyze the retrieval performance of the number of iterations on the downstream task, MS 300K, pre-training on the MS MARCO corpus. In Figure 3, we find that performance generally improves as the number of
iterations increases from 1 to 7, indicating the effectiveness of the bootstrapping pre-training method. However, performance begins to decline gradually after exceeding 7 iterations, possibly due to the model overfitting to the pre-training data.
As shown in Figure 4, it show the retrieval performance of different number of iterations on NQ, which aligns with the trend on MS 300K. The phenomenon that the performance degrades substantially after a certain iteration, is reasonable. Because different datasets have different characteristics and properties. Therefore, the optimal number of iterations may vary. However, the optimal iteration range is similar across datasets. For example, we found that the best performance is achieved around the 7th iteration in MS 300K, while it is around the 6th iteration in NQ (Figure 4). Therefore, for computational efficiency, when generalizing to other datasets, one can initially choose the number of iterations within a similar range.

heading: 5.5 Impact of Noisy Documents
text: To analyze the impact of different prompts for generating noisy documents, we remove noisy documents generated using a certain type of prompt during pre-training to train BootRet-BsMS and evalu-
ate its retrieval performance on the downstream MS 300K dataset. Based on Figure 5, we observe the following. (i) When the noisy documents generated by the shuffling prompt are removed, the performance dropped the most (the red circle), likely due to the significant semantic differences introduced by altering sentence order, reinforced by semantic consistency loss, improving discrimination ability. The sentence removal prompt (the green circle) shows a similar result. (ii) Next, the word masking prompt (the grey circle) yields moderate results, possibly due to the omission of masked token prediction (as the initial T5 is already pre-trained for this task), thereby weakening the masking effect. (iii) Lastly, the synonym replacement prompt (the blue circle) performs the most modestly, possibly because it introduces minimal semantic changes, thus having the same effect as original documents.

heading: 5.6 Visual Analysis
text: To further analyze the bootstrapped pre-training, we conduct visual analysis on BootRet-BsMS and BootRet-MtMS on MS 300K. We sample a query, “ germany gasoline cost” (QID: 194592), from the validation set and visualize the documents corresponding to the decoded docid lists (top 100) generated by BootRet-BsMS and BootRet-MtMS . Specifically, we visualize the query and document repre-
sentations encoded by the encoders of both models. From Figure 6, we observe that compared to BootRet-BsMS (left), BootRet-MtMS (right) exhibits the relevant docid (the blue triangle) closer to the query (the red star), while irrelevant documents (the grey circles) are farther away. Additionally, we observe that irrelevant documents near the query are more clustered in BootRet-BsMS compared to BootRet-MtMS , indicating that dynamic identifiers and pre-training tasks could effectively distinguish between documents.

heading: 6 Conclusion
text: In this work, we proposed BootRet, a bootstrapped pre-training method for GR, addressing the mismatch between pre-defined fixed docids and evolving model parameters in existing pre-training approaches. It dynamically adjusts docids based on the model pre-trained with two tasks. Extensive experiments validate that BootRet achieves superior performance compared to strong GR baselines on downstream tasks, even in the zero-shot setting.

heading: Acknowledgements
text: This work was funded by the Strategic Priority Research Program of the CAS under Grants No. XDB0680102, the National Key Research and Development Program of China under Grants No. 2023YFA1011602 and 2021QY1701, the National Natural Science Foundation of China (NSFC) under Grants No. 62372431, the Youth Innovation Promotion Association CAS under Grants No. 2021100, the Lenovo-CAS Joint Lab Youth Scientist Project, and the project under Grants No. JCKY2022130C039. This work was also (partially) funded by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, https://hybrid-intelligencecentre.nl, project LESSEN with project number NWA.1389.20.183 of the research program NWA ORC 2020/21, which is (partly) financed by the Dutch Research Council (NWO), and the FINDHR (Fairness and Intersectional NonDiscrimination in Human Recommendation) project that received funding from the European Union’s Horizon Europe research and innovation program under grant agreement No 101070212. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

heading: 7 Limitations
text: While BootRet has shown certain results in GR, it still has several limitations. (i) In the relevance prediction task, although we incorporate negative samples, the computational cost limits our ability to conduct comprehensive comparisons beyond batch-level contrasts. Future work could explore integrating dynamic hard negative mining techniques from traditional retrieval methods into GR. (ii) We design prompts with minimal hyperparameters to generate noisy documents. Future research could explore corresponding hyperparameter designs, such as determining the extent of sentence shuffling/removal strategies. (iii) Compared to other GR pre-training methods, our pre-training incurs slightly higher computational costs due to the need to update docids at each iteration. In future work, we can further explore how to trade off iteration costs and performance. (iv) For handling incremental documents, we ignore this issue in this work. For future work, inspired by (Chen et al., 2023), we could adaptively adjust cluster centers in the docid generation process based on the similarity between new and old documents. (v) Scalability is a significant challenge in current GR, requiring targeted solutions. Currently, a few works (Pradeep et al., 2023; Zeng et al., 2023) are exploring this issue. Differently, our work focuses on pre-training for GR which can provide suitable base model for GR. Therefore, the size of our experimental datasets follows that of most current GR works (Li et al., 2023; Sun et al., 2023; Tay et al., 2022; Wang et al., 2022, 2023; Zhou et al., 2022). We leave the scalability issue in the future. (vi) Jin et al. (2023) is concurrent work with ours, proposing to conduct language model indexer pretraining and Docid learning jointly. We do not consider this in the present study.

heading: A Appendix
text: A.1 Baseline Details
The baseline methods are described as follows: Sparse retrieval baselines: (i) BM25 (Robertson et al., 1995) is a widely used strong term-based method. We implement it based on the Anserini toolkit (Anserini); (ii) DocT5Query (Nogueira and Lin, 2019b) expands a document with pseudo– queries predicted by a fine-tuned T5 (Raffel et al., 2020) conditioned on the original document. And then we perform the BM25 retrieval.
Dense retrieval baselines : (i) DPR (Karpukhin et al., 2020a) is a BERT-based dual-encoder model using dense embeddings for texts; (ii) ANCE (Xiong et al., 2020) leverages ANN algorithm and hard negative techniques for training a dual-encoder model; and (iii) RepBERT (Zhan et al.,
2020b) is also a dual-encoder model with brute force searching.
Advanced GR baselines: (i) DSI (Tay et al., 2022) employs semantic structured numbers as docids via a hierarchical k-means clustering algorithm. (ii) GENRE (De Cao et al., 2021) uses document titles as docids. It learns the document-docid pairs. For NQ, it has unique document titles as docids. For MS 300K which might lack of titles, we use a document title generator (Zearing, 2023) to generate high-quality titles for documents. (iii) SEAL (Bevilacqua et al., 2022) uses n-grams as docids, and generates docids based on FM-index. It uses BART-large as the backbone. (iv) DSI-QG (Zhuang et al., 2023) generates pseudo-queries conditioned on the document using docT5query (Nogueira and Lin, 2019b) and pairs them with docids for training. It uses unique integer strings as identifiers. (v) NCI (Wang et al., 2022) employs semantic structured numbers as identifiers. It trains the model using pairs of pseudo-queries and docids, and designs a prefix-aware decoder. (vi) Ultron (Zhou et al., 2022) employs the product quantization code as docids. It starts with pre-training using document piece-docid pairs, followed by supervised fine-tuning with annotated queries and generated pseudo-queries on downstream tasks. (vii) Corpusbrain (Chen et al., 2022) employs unique document titles as docids for Wikipedia during pre-training. For MS MARCO, it might lack of titles; hence, we use the document title generator (Zearing, 2023) to generate titles for documents. It undergoes pre-training using pseudo-queries constructed from documents. For downstream MS 300K, we also generate document titles as docids, and then undergoes fine-tuning on downstream tasks using annotated queries. (Sun et al., 2023) introduces an autoencoder to generate identifiers for documents. This autoencoder learns to compress documents into docids and to reconstruct docids back into documents. It learns jointly with the retrieval task. (viii) NOVO (Wang et al., 2023) selects important words from the document as docids. The model is trained through supervised learning with annotated information. All GR baselines are optimized with an encoder-decoder architecture using MLE.
A.2 Additional Implementation Details For T5-base, the hidden size is 768, the feedforward layer size is 12, the number of selfattention heads is 12, and the number of trans-
former layers is 12. Decoder-only structures like the GPT (Ouyang et al., 2022) series models are left for future exploration.
BootRet and the reproduced baselines are implemented with PyTorch 1.9.0 and HuggingFace transformers 4.16.2; we re-implement DSI, and utilize open-sourced code for other baselines.
For data augmentation during fine-tuning, we leverage the pre-trained model, DocT5Query (Nogueira and Lin, 2019b) to generete pseudoqueries for documents. For MS 300K, we directly use the off-the-shelf pseudo-queries (Nogueira and Lin, 2019a). For NQ, we use the labeled queries to fine-tune DocT5Query. For each document, we generate 10 queries with the first 512 tokens of the document as input and constrain the maximum length of the generated query as 64. During training, we pair these pseudo-queries with docids corresponding to the document, and learn these pairs with standard MLE.
A.3 Additional Comparisons
As depicted in Table 5, for evaluating current GR methods against full-ranking methods, we adopt a cross-encoder baseline, namely monoBERT (Nogueira et al., 2019). Firstly, BM25 retrieves the top 1000 candidate documents, and monoBERT subsequently ranks them. monoBERT concatenates the query and document as input, and utilizes [CLS] for relevance calculation. It is optimized with cross-entropy.
A.4 Case Study
To better explain the changes in identifiers over bootstrapping iterations, we conducted a case study. Specifically, we sampled two documents. Below are their PQs at the initial stage, after training one round (BootRt-Bs), and after training multiple rounds (BootRt-Mt).
As shown in Table 7, we found that as identifiers evolve, the PQs for semantically similar documents
gradually become more discriminative, while still maintaining appropriate similarity. This makes the semantic hierarchy of the docid prefix tree clearer.
A.5 Inference Efficiency Since inference efficiency is critical for practical use, we further evaluate memory costs and inference speed on MS 300K. Table 6 in the Appendix highlights BootRet’s significant reduction in memory and latency compared to DPR. BootRet only needs a prefix tree for inference, resulting in 93% less memory usage than DPR’s index based on dense vectors. Additionally, BootRet outperforms DPR, with latency dropping from 18.35ms to 8.87ms for a 300K corpus. While ANN methods speed up, dual encoder latency may increase with larger corpora. However, the inference speed of BootRet only depends on the prefix tree’s structure.

heading: Future Work
text: Future work could explore integrating dynamic hard negative mining techniques from traditional retrieval methods into GR. (ii) We design prompts with minimal hyperparameters to generate noisy documents. Future research could explore corresponding hyperparameter designs, such as determining the extent of sentence shuffling/removal strategies. (iii) Compared to other GR pre-training methods, our pre-training incurs slightly higher computational costs due to the need to update docids at each iteration. In future work, we can further explore how to trade off iteration costs and performance. (iv) For handling incremental documents, we ignore this issue in this work. For future work, inspired by (Chen et al. , 2023), we could adaptively adjust cluster centers in the docid generation process based on the similarity between new and old documents. (v) Scalability is a significant challenge in current GR, requiring targeted solutions. Currently, a few works (Pradeep et al. , 2023; Zeng et al. , 2023) are exploring this issue. Differently, our work focuses on pre-training for GR which can provide suitable base model for GR. Therefore, the size of our experimental datasets follows that of most current GR works (Li et al. , 2023; Sun et al. , 2023; Tay et al. , 2022; Wang et al. , 2022, 2023; Zhou et al. , 2022). We leave the scalability issue in the future. (vi) Jin et al. (2023) is concurrent work with ours, proposing to conduct language model indexer pretraining and Docid learning jointly. We do not consider this in the present study