How Does Generative Retrieval Scale to Millions of Passages?
Authors: Ronak Pradeep
Kai Hui
Jai Gupta
Adam D Lelkes
Honglei Zhuang
Jimmy Lin
Donald Metzler
Vinh Q. Tran
Venue: EMNLP 2023 Main (2023)
Keywords: generative retrieval
differentiable search index
information retrieval
TL;DR: Evaluating generative retrieval techniques on larger corpora

Abstract: The emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and encoding an entire document corpus within a single Transformer.
Although many different approaches have been proposed to improve the effectiveness of generative retrieval, they have only been evaluated on document corpora on the order of 100K in size.
We conduct the first empirical study of generative retrieval techniques across various corpus scales, ultimately scaling up to the entire MS MARCO passage ranking task with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters.
We uncover several findings about scaling generative retrieval to millions of passages; notably, the central importance of using synthetic queries as document representations during indexing, the ineffectiveness of existing proposed architecture modifications when accounting for compute cost, and the limits of naively scaling model parameters with respect to retrieval performance.
While we find that generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge.
We believe these findings will be valuable for the community to clarify the current state of generative retrieval, highlight the unique challenges, and inspire new research directions.

How Does Generative Retrieval Scale to Millions of Passages?
Ronak Pradeep" 18, Kai Hui', .aii Gupta, Adam D. Lelkes, Honglei Zhuang Jimmy Lin Donald Metzler, Vinh Q. Tran'
Google Research, UUniversity of Waterloo
rpradeep@uwaterloo. ca, (kaihuibj, vqtran Jegooleegooo
Abstract
2021). Dense embeddings are pre-computed for all documents in a corpus and stored in an ex- ternal index, enabling fast approximate nearest neighbor search (Vanderkam et al., 2013; John- son et al., 2021) to retrieve relevant documents. Cross-encoders based on large Transformer mod- els (Nogueira and Cho, 2019; Nogueira et al., 2020; Pradeep et al., 2021b) often function top of these retrieved documents to further refine the top results,
The emerging paradigm of generative retrieval re- re-frames the classic information retrieval problem into
suueenceto-sequeeee model- ing task, forgoing external indices and encod- ing an entire document corpus within a sin- gle Transformer. Although many different ap- proaches have been proposed to improve the effectiveness of generative retrieval, they have only been evaluated the order of 100K in size. We conduct the first empirical study of generative retrieval tech- niques across various corpus scales, ultimately scaling up to the entire MS MARCO passage ranking task with corpus of 8.8M passages and evaluating model sizes up to 11B param- eters. We uncover several findings about seal- ing generative retrieval millions of passages: notably, the central importance of using syn- thetic queries as document representations dur- ing indexing, the ineffectiveness of existing proposed architecture modifications when ac- counting for compute cost, and the limits of naively scaling model parameters with respect to retrieval performance. While we find that generative retrieval is competitive with state- oh-the-art art dual encoders on small corpora, scal- ing to millions passages remains an impor- tant and unsolved challenge. We believe these findings will be valuable for the community to clarify the current state of generative retrieval, highlight the unique challenges, and inspire new research directions.
document corpora
Recently, the emerging paradigm of genera- tive retrieval (De Cao et al., 2020; Tay al., 2022) sought
replace this entire process with a single sequence-to-sequaene Transformer model (Sutskever et al., 2014; Vaswani et al., 2017), showing promising results against dual encoders given sufficiently small corpus size. Since then, various techniques, such (Zhuang et al., 2022b; Bevilacqua et al., 2022; Zhou et al., 2022; Wang et al., 2022; Chen al., 2023), have aimed improve the effectiveness of generative retrieval models, either with alternative document identifier formulations, architecture changes, or training ob- jectives. Such work, however, has only evaluated generative retrieval over relatively small corpora on the order of 100K documents, such as Natu- ral Questions (Kwiatkowski et al., 2019), Trivi- aQA (Joshi al., 2017), or small subsets of the MS MARCO document ranking task (Nguyen al., 2016). Despite these research contributions, sev- cral open questions remain unanswered, including how well current generative retrieval techniques work on larger corpora and which aspects of gener- ative retrieval models proposed so far are vital as we scale the corpora.
1 Introduction
For the last several years, dual encoders (Gillick al., 2018; Karpukhin al., 2020; Ni et al., 2022b; Chen al., 2022) have dominated the landscape for first-stage information retrieval. They model relevance by mapping queries and documents into the same embedding space, optimized via con- trastive learning (Hadsell et al., 2006; Gao et al.,
In this paper, we conduct the first empirical study of generative retrieval techniques over the entire MS MARCO passage-level corpus, evaluating its effectiveness over 8.8M passages. We select popu- lar approaches in recent works and evaluate them first on Natural Questions and TriviaQA to estab- lish a definitive ablation of techniques in con-
"Equal Contribution.
Work completed while a Student Researcher at Google.

---

trolled setup. Our experiments mainly focus on evaluating techniques proposed by Tay et al. (2022), Zhuang et al. (2022b), and Wang al. (2022). We evaluate three main design aspects in our study:
in prior work like Wang et al. (2022). How- ever, the effectiveness improvements vanish as we scale up the naive approach to simi- lar parameter sizes. Following Dehghami et al. (2022), we note that the parameter count is not the entire story and provide more discussion regarding model comparisons and trade-offs in Section A.3.2.
The impact of larget-space document identifiers (DocIDs) exploring Atomic, Naive, Semantic, and 2D Semantic DocIDs.
Document representation, assessing the use of document tokens, ground truth queries, and syn- thetic queries, as per Nogueira and Lin (2019).
3. Increasing the model size is necessary for improved generative retrieval effectiveness. However, for the best Sequential DocIDs, ef- fectiveness does not improve past a certain point - peaking at XL (3B) with a slightly worse score using XXL (1IB) under fixed experimental settings. We find this counter- intuitive to the common conception that model capacity limits the success of generative trieval prior studies of generative retrieval.
Model design techniques including prefix-ave aware aeight-adaptive (PAWA) decoding, constrained decoding, and consisteney loss.
We then scale up the corpus size leveraging the MS MARCO passage ranking task, beginning with a subset of 100K passages before increasing the count to IM and 8.8M passages (the entire set). In- crementally doing so allows us to establish which techniques remain effective as corpus size and dif- ficulty scale.
In the MS MARCO passage ranking task, we found that a scaled model, trained only on synthetic queries and leveraging Naive DocIDs, outperforms all other techniques we considered. On small subset of 100K passages, T5-Base model trained with this strategy achieves 82.4 MRR @ 10 (Sec- tion A.3.1), competitive with GTR-Base (Ni al., 2022b) at 83.2 MRR @ 10. However, on the setting with 8.8M passages, T5-XL model trained with this approach achieves only 26.7 MRR 10.
Finally, explore the effect of model scaling on retrieval effectiveness on large corpora, we evalu- ate set of techniques with promising results the 1..e. (220M) scale (Raffel et al., 2020) and modify the parameterization to consider up to IIB parameters. As the parameter distributions vary be- tween methods, for instance, Atomic DocIDs result in additional parameters equal to the embedding di- mension times corpus size, while the Naive DoeIDs do not cost anything beyond the core Transformer model, we aim to provide some insight into the trade-off of different parameter allocations on a large corpus.
The field of generative retrieval is rapidly evolv- ing. yet matching the effectiveness of top-tier dense retrieval models at scale remains a significant, un- resolved challenge. Our results suggest the need for continued research into generative retrieval and more fundamental advances to the paradigm be- fore we can fully leverage the power of scaling up model parameters. We believe our findings will help the research community better understand the challenges faced while applying generative trieval models larger corpora and inspire new research in this field.
While our experimental findings are nuanced, we summarize the main findings as follows:
1. Of the methods considered, we find synthetic query generation to be the single most critical component as corpus size grows (other than model scaling.) That is, defining the task of generative retrieval solely mapping from synthetic queries to DocIDs is the most ef- fective modeling strategy, with others largely unnecessary.
Methods
We revisit the design details of the generative re- trieval method, using the Differentiable Search In- dex (DSI) (Tay al., 2022) the baseline, and describe multiple techniques introduced in subse- quent works that we ablate and study in this work. For a comprchensive review of related work, we refer the reader to Section A.1 in the Appendix.
2.
As corpus size increases, discussion of com- pute cost crucial. Methods that expand the parameter count beyond what is used by the original pre-trained T5 checkpoint are more effective. We find this crucial detail unstated

---

2.1 Background
argued in Zhuang et al. (2022b) and Wang et al. (2022), there are two kinds of gaps between the index and retrieval tasks. First is the data distribu- tion gap: queries for the retrieval task are short and request specific information, while the documents for the indexing task are long and convey infor- mation. Second is the coverage gap: the model is exposed to the entire corpus during the training of the indexing task, while only positive examples have associated queries in the retrieval task. The latter problem exacerbated in the MS MARCO passage ranking task as only 550K passages have an associated query for training the retrieval task, while the indexing task has to learn to encode all 8.8M passages in the corpus.
DSI (Tay et al., 2022) transforms the retrieval task into sequence-to-sequenee (seq2seq) modeling, us- ing queries as inputs and relevant document iden- tifiers (DoclIss) as generation targets. The corpus, namely the mapping between the document's con- tent and its identifier, is encoded using the parame- ters of the LLM. DSI achieves this by leveraging two seq2seq tasks: indexing and retrieval. During training, the model learns to generate the DocID given the document content (indexing task) or a relevant query (retrieval task). At inference, the model processes query and generates ranked list of identifiers as retrieval results.
2.2 Inputs and Targets
To mitigate this gap, they propose generating synthetic queries for each document using mod- els like docTsquery (Nogueira and Lin, 2019). They then trained a generative retrieval model predict the DocIDs given the corresponding syn- thetic queries. We can also think of these synthetic queries as alternative document representations.
In the framework discussed, DSI learns to encode the mapping between the long-form textual rep- resentation of document and its identifier in its parameters while also learning to fetch the same identifier when it receives a relevant query as input.
Two crucial design choices are how documents are represented (i.e., the inputs in the indexing task) and how document identifiers (DoeIDs) are repre- sented (i.e., the targets in both indexing and re- trieval tasks). One primary consideration is the challenge of encoding long textual sequences with a Transformer (Vaswani et al., 2017)-based LLM, making it hard to index entire documents. Another is that naive identifiers from existing datasets could be sub-optimal due to their lack of semantic mean- ing.
2.2.3 Document Identifiers
In this work, we consider four kinds of different identifiers: the three kinds of document identifiers from DSI (Tay et al., 2022): unstructured atomic identifiers (Atomic DocIDs), naive string iden- tifiers (Naive DocIDs), semantically structured identifiers (Semantic DocIDs), and the 2D Seman- tic DocIDs from Wang et al. (2022).
Atomic DoeIDs. We treat each DocID as a single, or "atomic" token in this setting. The decoder, then, only needs to run for a single decoding step; we then sort the logits of the DocIDs to obtain the ranked document list. Here, each document requires an additional unique token in the model vocabulary. As result, the model's parameters increase by the corpus size times the embedding dimension, becoming potentially expensive for large corpora. When considering millions of documents, we apply two optimizations to make the implementation more feasible. First, the embedding table of the encoder is adjusted to include only the standard T5 vocabulary, while the output projection of the decoder corresponds to DocIDs. Second, we take special care to ensure the output projection is properly sharded across cores to distribute memory cost to allow scaling. We achieve this by setting partitioning rules in the t5x framework (Roberts et al., 2022).
2.2.1 Document Representations
One straightforward idea is to pick a text span from the document as representation. DSI con- siders the first 64 tokens (FirstP) in each docu- ment, whereas Wang et al. (2022) leverages ten randomly selected chunks of 64 consecutive to- kens, technique they call Document As Query (DaQ). We evaluate the FirstP and DaQ approaches individually and in conjunction when working with lengthy documents in Natural Questions and Trivi- aQA datasets. In the case of MS MARCO, which has short passages, FirstP and DaQ are essentially the same, assuming sufficient context length.
2.2.2 Synthetic Query Generation
In training the model for retrieval tasks, the natu- ral baseline uses existing labeled data, i.e., queries from the retrieval dataset as inputs and the Doc- IDs labeled as relevant as targets. However, as

---

2.3 Model Variants
Naive DocIDs. Here, the original document identifier from corpus directly used and treated as a textual string. For example, a five-digit number "42915" is treated as a string and passed through the SentencePiece vocabulary of T5. Note that Naive DocIDs might sometimes capture semantics influenced by the curation pipelines that may reflect inherent relatedness.
Besides alternative ways of constructing model inputs and targets, generative retrieval approaches that build on DSI have also investigated novel modeling components. Here, review three model components introduced by Wang et al. (2022) and Bevilacqua al. (2022).
Prefix-Aware Weight-Adaptive (PAWA) De- coder is proposed as a method for decoding 2D Semantic DocIDs. Unlike a standard Transformer decoder, which uses the same matrix to project the hidden representation of the decoder to the vocabulary space for every position, PAWA decoder uses different projection matrices at each timestep, with the weights of each projec- tion matrix computed adaptively by a separate Transformer decoder. Specifically, in a vanilla decoder, the dense representation h E R from the last decoder layer is projected into the vocabulary space with W C RdxXV[] where denotes the sequence length for decoding. To incorporate the position, the extra decoding module separately processes the input query and the already-decoded DocID tokens to output a projection matrix Waroo E Rdxix[VI, replacing W. This aims to capture that the semantic meaning of DocID token depends on its position in the output sequence as well as on the DocID prefix preceding it. The experiments in this paper use the open-source PAWA decoder implementation provided by the original authors as reference and build out on t5x. For more details, please refer to NCI (Wang et al., 2022) and their codebase.
Semantic DocIDs. Following Tay et al. (2022), Semantic DocIDs aim to imbue document identi- fiers with hierarchical semantic information instead of relying on predefined identifiers. Specifically, after encoding documents into dense vectors, hierarchical k-means algorithm recursively clusters the space into & clusters until individual clusters include more than e documents. Consequently, all document identifiers form a tree, where non-leaf nodes correspond to superclusters, and leaf-node clusters with at most e documents each. Semantic DocIDs are formed by composing these cluster identifiers, each from 0 to k- - 1, tailed by an identifier in the leaf nodes between O and 1. In this work, use the identifiers generated by Wang et al. (2022) for NQ and TriviaQA to perform fair comparison. They achieve this with a 12-layer BERT model. For MS MARCO, we use Sentence
(Ni et al., 2022a), and c- 100. Since the passage-level corpus is in the order of millions, if a cluster ends up with more than 1M documents, we sample 100K documents first, when computing centroids. We used k = 10 clusters at each level, corresponding to the ten digits (0...9).
Constrained decoding can be used to avoid generating invalid document identifiers (Bevilac- qua et al., 2022; Wang et al., 2022). While we have empirically found that roughly less than I in 20 DSI-based generation beams are invalid, we include this method nonetheless, as it is widespread in the literature. In this work, we adopt an exact match approach that leverages a trie to ensure we decode only valid document identifiers.
2D Semantie DocIDs. In the Semantic DocID setting, the same tokens represent different semantic meanings at different positions: we use the same set of tokens between 0 to k k-1 for all identifiers, but they represent semantic clusters at different levels in the tree. To address this, NCI (Wang al., 2022) introduces a 2D variant by adding an extra dimension encode the positions, making the model aware of levels of clustering when decoding the identifier. To allow better modeling with this variant, they additionally introduce a change to the decoder, Weeight-Adaptiv Decoding, which we describe in the next section.
Consistency loss helps alleviate over-fitting by introducing regularization term. The basic idea is that the representations generated by two forward passes with different dropout masks should be similar. Wang al. (2022) incorporate

---

Dataset
#Docs
% Covered by train query set
passages). It is worth noting that most documents in NQ100K and MSMarcol00K have at least one relevant query in the training set. However, as we scale to MSMarcoFULL, the fraction of documents with queries in the training set drastically drops to around 6%, leading to a more practical setup. We summarize the statistics of these datasets in Table 1.
NQ100K (Wang et al., 2022)
110K
98.4%
TriviaQA (Wang et al., 2022)
74K
57.7%
MSMarco 100K
100K
92.9%
MM
IM
51.6%
MSMarcoFULL
8.8M
5.8%
Table The coverage statistics of the benchmark datasets and their training query sets.
NQ100K & TriviaQA. To enable compar- isons, we reuse the documents, the segmented documents, the training/testing splits, and gen- crated qucry sets from Wang et al. (2022). The Natural Questions and TriviaQA datasets have corpora of sizes 109K and 74K, respectively. Note that Wang et al. (2022) refers to NQ100K as NQ320K; we refer to the number of unique documents instead of the labeled training data size. Most documents in the NQ100K dataset have at least one relevant question in the training data, while 58% of the TriviaQA dataset has this property.
this insight into regularization term that augments the generation loss. We investigate the softmax ver- sion as described the NCI paper (Bq. 5 in (Wang et al., 2022)) and a KL-divergence version from an early version of NCI (Eq. 1). They compute the Kullback-Leibler (KL) divergence between the output probabilities of two independent forward passes per position, where Pt, and pi,2 are the probability distributions over the vocabulary space from the two forward passes position i, respectively.
Creg 1 PPLL(P P:2) + (PL ll Pa1) (1)
MSMarco100K, 1M, & FULL. In the same vein as NQ100K and TriviaQA, for MSMarco 100K curate dataset with 100K passages sampled from the full MSMarco passage ranking dataset. Most passages have at least one positive query for training. We also include passages relevant to the queries in the development dataset (for evaluation). MSMarcoIM is 10x larger than MSMarcol00K. As with MSMarco100K, we augment the corpus with passages relevant to development queries. We first include all passages relevant to the 533K and 7K queries from the training dataset and development sets, respectively. This results in 516K and 7K unique passages from each set. We randomly sample passages without a query either set to total a million passages. Finally, MSMLrcoFULL scales up another order of magnitude in corpus size. As a result, only 5.8% of the passages have corresponding query in the training set. We aren't it aware of any previous work that has attempted to apply generative retrieval models dataset of this size and complexity.
While we closely follow the implementation of the Neural Corpus Indexer codebase, we find that these regularization terms lead to training instability and that the model effectiveness often diverges into a NaN loss. As result, we do not include consis- tency regularization in our final experiments.
3 Experimental Setting
We limit ourselves to English retrieval tasks, focus- ing on the behavior of generative retrieval models at varying corpus scales.
3.1 Corpus and Training Data
Following small-scale generative retrieval experiment setups (Tay et al., 2022; Wang al., 2022: Zhuang al., 2022b; Chen et al.. 2023), we start with experiments on the Natu- ral Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017) datasets. To better understand how different model configurations perform at scale and in more practical settings, we also experiment with variants of the MS MARCO passage ranking dataset. The MS MARCO passage ranking collection consists of a corpus of 8.8M passages and a training set of 532K queries. From this dataset, we construct three variants, namely, MSMarco100E (100K passages), MSMarcoIM (IM passages), and MSMarcoFULL (all 8.8M
3.2 Synthetic Query Generation
For NQ100K and TriviaQA, we reuse the gen- erated questions from (Wang al., 2022) with 20 and 15 generated questions for each docu- ment, respectively. For the MSMarco variants, we use docT5query (Nogueira and Lin, 2019) to

---

generate questions, with 40 generated questions per passage. We also train a question-generation model using T5-base using training data from DPR (Karpukhin& et al., 2020), a retrieval dataset derived from NQ (Kwiatkowski et al., 2019). We use this model to generate 40 questions per pas- sage, following the configuration of docTsSuery. We refer to this setting for NQ and TriviaQA as "in-domain D2Q".
2023), and GTR-Base, state-of-the-art dual en- coder (Ni et al., 2022b). For the MSMarco variants we introduce, we provide our own GTR-Base (Ni et al., 2022b) results.
Experimental Results
We report our results in three parts. First, we ab- late all the methods from Scction 2 using T5-Basc on small-scale datasets: NQ100K and TriviaQA. We observe which techniques work best on this small scale with widely studied datasets. Then, we transfer the same techniques and scale up to the entire MS MARCO passage ranking dataset. Fi- nally, to understand whether we can attribute the effectiveness bump from Atomic DocIDs addi- tional model parameters, we fix the corpus to be FSML and scale the model size up to 1 IIB, Across the tables, At. refers to Atomic DocIDs, Nv. Naive DocIDs, and Sm. Semantic DocIDs.
3.3 Evaluation Dataset and Metrics
We evaluate using the validation of each dataset. For NQ100K, TriviaQA, and MSMarco, the set contains 7,830, 7,993, and 6,980 queries, respec- tively. We use the same evaluation set for all three MSMarco variants considered. For each query in the evaluation sets, we use the models to generate ranked lists of documents. We report Recall @ 1, Recall@5, and MRR @ 10 (Mean Reciprocal Rank) for NQ, TriviaQA, and MSMarco, respectively.
3.4 Model Variants
4.1 Ablations over Small Corpora
We evaluate all methods using T5.1.1 back- bone (Raffel et al., 2020). See Appendix A.2 for implementation details, including hyperparameter settings and compute cost. We test variants of la- beled vs. synthetic queries, FirstP vs. DaQ docu- ment representations, and combinations of multiple representation choices. For each model variant, we ablate all versions of document identifiers when applicable. Modifications in model architecture are introduced in stacking fashion, starting with the base model and then adding on the PAWA de- coder, constrained decoding, and consistency loss in this order. Note, we only evaluate the PAWA decoder with 2D Semantic DocIDs as Wang et al. (2022) introduce the decoder to exploit their unique representation hierarchy.
We report our ablations over NQ100K and Trivi- aQA in Table 2. The best combination of our tech- niques (row (7)) sets new state-of-the-ar result on the NCT variant of Natural Questions without using sophisticated modeling techniques, architec- tural changes, and learned DocIDs. The choice of document representation by far dominates the overall effectiveness of the retriever. Using just the training queries provided by the datasets demon- strates poor results due the low coverage of the documents. With FirstP, see considerable im- provements over this setting. The DaQ representa- tion strategy further improves over FirstP, exposing the model to different views of the same document. However, synthetic queries from D2Q are essential to high generative retrieval effectiveness, resulting in gain of more than 7 points. This technique, by far, trumps all other proposed changes.
For model scaling experiments, we mainly inves- tigate whether Atomic DoeIDs are an effective way scale to millions of passages, given the parame- ter cost. As such, we consider larger models with Naive DocIDs and Semantic DocIDs comparable to T5-Base with Atomic DocIDs, which total 7B parameters when scaling to 8.8M DocIDs.
As for other design choices, for this small-scale setup, Naive and Semantic DoeIDs compete on par with each other (varying between task configu- rations), and Atomic DocIDs are consistently the best. We note, though, that on NQ100K, Atomic DocIDs add 80M parameters to T5-Base model that would otherwise be 220M parameters (a 36% increase). Given the comparable effectiveness in the best configuration (row (7)), these extra param- eters may or may not be worth it, but we refer Section A.3.2 for more discussion. Modeling tech- niques from (Wang et al., 2022), i.e. 2D Semantic
For baselines we provide BM25 (Robertson and Zaragoza, 2009) and BM25 with doc2query-TT (Nogueira and Lin, 2019). For Natural Questions and TriviaQA, we also include the previous results reported for the NCI-variant of NQ (i.e., NQ100K). The baselines include state-of-the- the generative retrieval results like NCI and GenRet (Sun et al.,

---

NQ100K
TriviaQA
Model
At.
Nv.
Sm.
At.
Nv. Sm.
Baselines
BM25 (via Wang et al. (2022))
15.1
$6.9
BM25 w/ doc2query-TS (via Wang al. (2022)
59.7
GTR-Buse (via Sun et al. (2023))
56.0
NCI (Wang et al.. 2022)
62.8
65.9
88.8
90.5
GenRet (Sun al., 2023)
68.1
Ours
(1a) Labeled Queries (No Indexing)
50.7
49.2
49.0
60.9
56.7
61.4
(2a) FirstP + Laheled Queries (DST)
60.0
58.4
58.7
71.6
75.2
78.9
(2b) DaQ + Labeled Queries
61.4
60.0
80.4
(3a) DaQ + D2Q + Labeled Queries
69.6
67.9
67.9
88.2
85.7
86.3
(3b) FirsiP DaQ + D2Q Labeled Queries
69.0
68.2
67.2
88.9
86.9
87.4
(4a) 3b PAWA (w/ 2D Semantic DocIDs)
66.3
86.5
(4b)
Constrained Decoding
67.3
87.3
(5) 4b Consistency Loss (NCI)
66.3
86.6
(6a) DaQ Only
17.1
18.4
15.6
41.0
31.3
20.6
(6b) D2Q Only
43.6
42.3
42,9
61.9
57.8
57.1
(6c) 6b PAWA (w/ 2D Semantic DocIDs) Constrained Decoding
43.1
57.7
(7) 3b in-domain D2Q
70.7
69.7
69.5
90.0
88.0
89.2
Table 2: Results on small-scale NQ (Recall@ 1) and TriviaQA (Recall@5) datasets. We present ablation results in a stacking fashion. Rows 6 isolate the importance of D2Q. Last row revises the best method with in-domain D2Q.
DocIDs, PAWA, constrained decoding, and consis- tency loss, do not reliably improve the model over using synthetic queries alone.
of Wikipedia articles: the beginning of Wikipedia documents are informative entity descriptions, and many sentences refer to the entity-which is likely the answer to requested query.
At this corpus scale, our best result uses a mix- ture of FirstP, DaQ, labeled queries, and synthetic queries for training. However, notably, the quality of the synthetic queries are very important, with those from a generator we specifically trained for the aunstinn-answering domain significantly out- performing the query generator trained over MS- MarcoFULL.
As we scale the corpus size, DSI effectiveness drops rapidly, with the best result (D2Q only with Atomic DoclDs) falling off from 80.3 to 55.8 and finally 24.2 as we scale to the whole 8.8M passages. The Semantic DocID setting, too, drops off as we scale to the whole corpus, under- mnderr-performing the Naive DocID setting. We conjecture that this may be due the potentially increased length of Se- mantic DocIDs being more difficult to decode than Naive DocIDs coupled with noisy partitioning of the semantic space (especially when using an off- the-shelf embedding model such as SentenceTS. Base). However, we observe Semantic DocIDs decoded via the PAWA decoder score higher. We hope shed some insight into why this might be in the next section, where examine model size. Constrained decoding only provides marginal value and generally not worth the added complexity.
4.2 Scaling Corpus Size
We now consider variants of the MS MARCO pas- sage ranking task, scaling from 100K to 1M and 8.8M passages. Results are reported in Table 3. Per- haps the most striking observation about the transi- tion to the MS MARCO passage ranking collection is the absolute requirement of synthetic queries for strong retrieval effectiveness. Synthetic queries re- sult in a 2-3x improvement over the original DSI formulation alone. Using only synthetic queries as the indexing task is the most effective and straight- forward training strategy for the MS MARCO col- lection. This result signifies a notable difference in the transition from NQ and TriviaQA to MS MARCO, where FirstP and DaQ did provide sub- stantial value. The reason may be an artifact of the NQ and TriviaQA datasets leveraging a corpus
4.3 Scaling Model Size
How much of the Atomic DocID setting's 's effec- tiveness can be attributed its additional model parameters? On MSMarcoFULLL decoding Atomic DocID tokens adds an overhead of 7B parameters to the otherwise 220M- Mar-maraeeter T5- 15-Base model.

---

MSMarco100K
MM
MSMarcoFULL
Model
At.
Nv.
Sm.
Baselines
BM25
18.4
BM25 (w/ doe2query-TS)
56.6
27.2
GTR-Base
83.2
60.7
34.8
Owrs
(1a) Labeled Queries (No Indexing)
0.0
1.1
0.0
0.0
0.5
0.0
0.0
0.0
(2a)
FirstP/DaQ + Labeled Queries (DSI)
0.0
23.9
19.2
2.1
12.4
7.4
0.0
7.5
3.1
(3b)
FirstP/DaQ D2Q Labeled Queries
79.2
77.7
76.8
53.3
48.2
47.1
14.2
13.2
6.4
(4a) 3b PAWA (w/ 2D Semantic DocIDs)
77.1
50.2
9.0
(5)
4a Consistency Loss (NCI)
77.1
50.2
9.1
(6h)
D22Q only
80.3
78.7
78.5
55.8
55.4
54.0
24.2
13.3
11.8
(4o')
6b PAWA (w/ 2D Semantic DocIDs)
78.2
54.1
17.3
(40)
6b Constrained Decoding
78.6
54.0
12.0
PAWA Semantic DocIDs) Constrained Decoding
78.3
54.2
17.4
Table 3: Results on the development set of the scale variant MS MARCO V1 passage collections, reported
MRR @ I0. Best results per column and results within 0.1 of best are bolded. Note that FirstP here is equivalent to DaQ as MS MARCO input passages fit into the input window. For more metrics (nDCG@20, Hits 1,5,10,20. Precision @ 1,5, 10,20), please Section A.5 of the Appendix.
T5 Scale
Training
Params
Inference FLOPs
MRR@
Basc
D2Q Only + Atomic DocID
7.0B
0.9 10
24,2
Basc
D2Q Only + Naive DocID
220M
1.4 x 10
13.3
Basc
D2Q Only + PAWA (2D Sem.)
761M
6.8 10
17.3
Large
D2Q Only + Naive DoeID
783M
3.5 x 10
21.4
Large
D2Q Only + PAWA (2D Sem.)
2.1B
10
19.8
XI.
D2Q Only + Naive DoelD
2.8B
9.3 x 10
26.7
XXL
D2Q Only Naive DoeID
11B
4.3 10th
24.3
Table effect of scaling up model size sequential DoeID approaches in comparison Atomic DocIDs for the MSMarcoFULL task.
We take the best configuration on MSML from Table 3 and scale model parameters Naive DocID and Semantic DocID (PAWA) to similar sizes for comparison. We report results in Table 4.
Scaling Transformers naively according to de- fault T5 scales while using Naive DocIDs had the highest effectiveness on MSMarcoFULL at 26.7 MRR @ 10. Despite using only ~2.8B pa- rameters, this approach outperforms T5-Base with Atomic DocIDs, which uses 7B parameters while achieving only 24.2 MRR @ 10. However, while pa- rameter count has practical implications regarding the resources required for training and inference (especially TPU/GPU memory), there are other trade-offs to consider, which we discuss in the Ap- pendix A.3.2.
Overall, we observe general trend that as pa- rameter count increases, retrieval effectiveness im- proves. Indeed, both Atomic DocIDs and PAWA Semantic DocIDs had the highest scores in Table 3, which we now attribute to their increased size. No- tice that the difference here only comes out when scaling to MSMarcoFULL, where these parame- ter differences magnify significantly over corpora a smaller scale. However, not all methods are equal. PAWA and 2D Semantic DocIDs (Wang et al., 2022) significantly increase decoding param- eters with its extra decoding stack, yet yield gain over naively scaling the Transformer with Naive DocIDs, underperforming by 4 points around ~800 parameters. This pattern continues hold scaling PAWA to B. parameters. Thus, to con- serve resources, we do not scale PAWA any further.
While Naive DocIDs perform well at T5-XL size, surprisingly, we find that scaling further to XXL. (11B) does not improve effectiveness; in fact, it is detrimental to retrieval effectiveness (24.3 MRR @ 10 XL's 26.7) under the same experi- mental settings and hyperparameter settings, even though model training converges faster. This find- ing is counter-intuitin to most generation tasks and the common conception of generative retrieval re-

---

MSMarcoFULLL Recall @
TREC DL 19 nDCG@
TREC DI 20
M 15 Scule
Training
Params
nDCG@10
nDCG@ 10
Baselines
BM25
22.8
BM25 w/ docTsSquery
33.8
GTR-Bas BTReBasc (Ni et al., 2022b)
110M
89.8
42.0
(75 Scale)
D2Q Only + Atomic DoelI
7.0B
29.7
53.8
Large
D2Q Only Naive DocID
783M
82.0
27.3
49.9
45.2
Large
D2Q Only + PAWA (2D Sem.)
2.1B
73.2
24.8
46.6
50.1
XI.
D2Q Only + Naive Doell>
2,88
84.7
33.2
55.0
52.2
XXL
D2Q Only + Naive DoeID
11B
81.9
30.6
52.0
49.0
Table Additional evaluation of the top five model checkpoints from Table 4 on TREC DL 19 & 20. We decode using 100 beams here. The same metrics reported the original MSMarco validation set, with the addition of Recall @ 100.
lying on model capacity for retrieval effectiveness. To strengthen the evidence from our scaling re- sults, we provide results on TREC DL 19 and 20 (Craswell et al., 2020, 2021) in Table 5. These ad- ditional densely judged evaluation sets, opposed the sparse judgments in the MS MARCO vl Pas- sage Ranking validation set, seek address some the issues with the original task. We see that our original findings generally hold under these new metrics and believe that this addition reaffirms the soundness of our study.
MARCO dataset. Our best result on MS MARCO passage ranking uses only synthetic queries and Naive DocIDs in the target space for the training task, with the model scaled to T5-XL (3B). This model only achieves MRR @ 10 of 26.7. Surpris- ingly, increasing parameters for the same setting up to XXL (IIB) scores worse.
Altogether, our results highlight the unique chal- lenges currently facing the generative retrieval paradigm, the need for closer attention to method comparisons, and more fundamental improvements before fully leverage the power of larger language models in the future.
5 Analysis & Discussion
The results of this work raise major questions, in- cluding: Why are synthetic queries effective? and Which model scaling approach is the best? We present an extended analysis, including additional experiments, to provide more insight these top- ies in Section A.3 of the Appendix. Finally, discuss the limitations of our study and future di- rections for the paradigm in Section 7 and A.4, respectively.
Limitations
As with all empirical studies, has its set of limitations that we urge the reader to consider. Multiple works have come after the experiments in this work, e.g., Chen et al. (2023), and thus, we do not present an exhaustive set of generative retrieval techniques here. For instance, the vast space of identifiers could be in natural language or learned codes. In addition, due to resource constraints, our model scaling experiments not all ablation scenarios in Table 3 are scaled larger model sizes. It could be possible that some missed setups improve more at larger parameteriza- tions, albeit unlikely, as with scaling past BBB. addition, we do not saturate the scaling of Atomic DocIDs because of the extreme parameter require- ments. Finally, since this work focused on the ef- fectiveness of generative retrieval on large corpora, scaling model size for smaller corpora was outside our scope. Investigating the maximum corpus size for which generative retrieval could provide state- of-the-art art effectiveness is a question of practical importance, which we leave for future work.
6 Conclusion
not exhaustive, and
We provide the first empirical study of generative retrieval methods over the large-scale 8.8M passage corpus the MS MARCO passage ranking task. Of the various methods from the literature that we consider in this work (Tay et al., 2022; Zhuang et al., 2022b; Wang et al., 2022), we find that the use of synthetic queries as document represen- tation strategy is the only approach that remained effective and necessary, as we scaled up the corpus size. We highlight the importance of accounting for the compute cost of techniques; keeping the parameter count fixed, we find that naive methods outperform more sophisticated ones on the full MS

---

References
Mitko Gospodinov, Sean MacAvaney, and Craig Mac- donald. 2023. Doe2Query-: When less is more. arXiv:2301.000 03266.
Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee,
Eunsol Choi, and Hannanch Hajishirzi. 2021. XOR QA: Cross- mooss-inggull open-retee retrieval question answer- ing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putattinal Linguisties: Humun Language Technolo- gies, pages 547-564. Association for Computational Linguistics.
Raia Hadsell, Sumit Chopra, and Yann LeCum. 2006. Dimensionality reduction by learning invariant mapping. 2006 IEEF Computer Society Confer- ence on Computer Vision and Pattern Recognition (CVPR 06), 2:1735-1742.
Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ji Ma, Jai Gupta, Cicero Nogueira dos Santos, Yi Tay, and Donald Metzler. 2022. ED2LM: Encoder-decoder to language model for faster document re-ranking inference. In Find- ings of the Association for Computational Linguis- tics: ACI. 2022, pages 3747-3758. Association for Computational Linguistics.
Michele Bevilaequa, Giuseppe Ottaviano, Patrick Lewis, Wen-tau Yih, Sehastian Riedel, and Fabio Petroni. 2022. Autoregressive search engines: Generating substrings document identifiers. arXiv.2204. 10628.
Xiaoyang Chen, Yanjiang Liu, Ben He, Le Sun, and Yingfei Sun. 2023. Understanding differential search index for text retrieval. arXiv:2305.0200 02073.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. IEEE Transactions Big Data, 7(3):535-555 547.
Xuanang Chen, Jian Luo, Ben He, Le Sun, and Yingfei Sun. 2022. Towards robust dense retrieval via lo- cal ranking alignment. In Proceedings of the Thirty- First International Joint Conference on Artificial In- telligence, IJCAI, pages 1980-1986.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading com- prehension. arXiv: 705.03551
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. Text REtrieval Conference (TREC).
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- urul Language Processing (EMNLP), pages 6769- 6781.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022. Overview of the TREC 2022 deep learning track. In Text REtrieval Conference (TREC).
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M Voorhees. 2020. Overview of the TREC 2019 deep learning track. In Text RE- trieval Conference (TREC).
Tom Kwiatkowski, Jennimaria Palomaki. Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a bench- mark for question answering research. Transactions of the Association for Computational Linguisties. 7:453-466,
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020, Autoregressive entity retrieval. arXiv:2010.00904.
Oleg Lesota, Navid Rekabsaz, Daniel Cohen, Klaus Antonius Grasserbauer, Carsten Eickhoff. and Markus Schedl. 2021. modern perspective on query likelihood with deep generative retrieval models. Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval.
Mostafa Dehghani, Yi Tay, Anurag Arnb, Lucas Beyer, and Ashish Vaswani. 2022. The efficiency misnomer. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT
Xueguang Ma, Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2022. Document expansion baselines and learned sparse lexical representations for ms marco v1 and v2. Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 6894 6910. Association for Com- putational Linguistics.
Sanket Vaibhay Mehta, Jai Gupta, Yi Tay, Mostafa De- hghani, Vinh Q. Tran, Jinfeng Rao, Mare Najork, Emma Strubell, and Donald Metzler. 2022. DSI++ Updating transformer memory with new documents. arXiv:2212.09974.
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval continuous space. 18X1008810000000

---

Donald Metzler, Yi Tay, Dara Bahri, and Mare Najork. 2021. Rethinking search: Making domain experts out of dilettantes, SIGIR Forum, 55(1).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the lim- its of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 ( (14)):1667.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. In CoCo@ NIPS
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghu- nandan H. Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Macicj Kula, Ed H. Chi, and Maheswaran Sathi- amoorthy. 2023. Recommender systems with gen- erative retrieval, arXiv:2305.0500
Jianmo Ni, Gustavo Hernandez Abrego, Noah Con- stant, Ji Ma, Keith B. Hall, Daniel Cer, and Yin- fei Yang. 2022a. Sentence-TS: Scaling up sentence encoder from pre-trained text-to-text transfer trans- former. In Findings of the Association for Compu- tational Linguisticx: ACI. 2022, pages 1864-1874, Association for Computational Linguistics.
Adam Roberts, Hyung Won Chung, Anselm Levskaya. Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez. Alex Salcianu, Mare van Zee, Jacob Austin, Sebas- tian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jas- mijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee. Dan Garrette, James Lee- Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel. Mark Omernick, Brennan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scaling up models and data with t5x and seqio. arX1v.220..17189.
Jianmo Ni, Chen Qu Jing Lu, Zhuyun Dai, Gus- tavo Hernandez Abrego, Ji Ma. Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wet Wei Chang, and Yin- fei Yang. 2022b. Large dual encoders are general- izable retrievers. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9844-9855. Association for Com- putational Linguisties.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv: 0ee0..901000000 04085.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre- trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708-718. Association for Com- putational Linguistics.
Stephen Robertson and Hugo Zaragoza. 2009. The prohabilistic relevance framework: BM25 and be- yond. Now Publishers Inc.
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023. Learning to tokenize for generative retrieval. arXiv:2304.0417..
Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docTTTTTquery. Online preprint.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. UrXiv:1904.08377
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence sequence learning with neural networks. arXiv:
Ronak Pradeep, Yilin Li, Yuetong Wang, and Jimmy Lin. 2022. Neural query synthesis and domain- specific ranking templates for multi-stage clinical trial matching. In Proceedings of the 45th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, SIGIR '22, page 2325-2330. Association for Computing Machinery.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer memory as differentiable search index. 2rXii:2202.06991.
Dan Vanderkam, Robert B Schonberger, H. Rowley. and Sanjiv Kumar. 2013. Nearest neighbor search in Google Correlate. Online preprint.
Ronak Pradeep, Xueguang Ma. Rodrigo Nogueira, and Jimmy Lin. 2021a. Vera: Prediction techniques for reducing harmful misinformation in consumer health search. In Proceedings of the 44th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, SIGIR '21, page 2066-2070. Association for Computing Machinery.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Yujing Wang. Ying Hou, Hong Wang. Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia. Cheng- min Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A neural corpus indexer for document re- trieval, arXiv:2206.00 027433.
Ronak Pradeep. Rodrigo Nogueira, and Jimmy J. Lin. 2021b. The Expando-Mono-Duu design pattern for text ranking with pretrained sewuuence-t-sequeeee models. arXiv:2101 101.06677.

---

Yidan Zhang, Ting Zhang, Dong Chen, Yujing Wang. Qi Chen, Xing Xie, Hao Sun, Weiwei Deng. Qi Zhang, Fan Yang, Mao Yang, Qingmin Liao, and Baining Guo. 2023. IRGen: Generative modeling for image retrieval. 10X1v:2303.00126.
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji-Rong Wen. 2022. Ultron An ultimate retriever on corpus with model-based in- dexer. 07X25
Honglei Zhuang. Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma. Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2022a. RankT5: Fine- tuning T5 for text ranking with ranking losses.
arXiv:2210. 10634.
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022b. Bridging the gap between indexing and re- trieval differentiable search index with query gen- eration. 20X1:22006.10228.
Noah Ziems, Wenhao Yu, Zhihan Zhang, and Meng Jiang. 2023. Large language models are built-in au- toregressive search engines. arXiv:2305.09612.

---

A
Appendix
reranker by training a model that takes the con- catenation of the query and document as input and generates a relevance label. Pradeep et al. (2021b); Zhuang et al. (2022a); Hui et al. (2022) have since improved the effectiveness and effi- ciency of generation-basee reranking. These ap- proaches continue to demonstrate strong effective- ness in various retrieval tasks (Pradeep et al., 2021a; Craswell et al., 2022; Pradeep al., 2022).
Due to space constraints, we include related work, additional discussion, analysis, experiments, and implementation details here.
A.1 Related Work
Traditional retrieval models like BM25 (Robertson and Zaragoza, 2009) that rely on the lexical overlap, term frequency heuristics, and inverse document frequency, while reasonably strong on their own, tend to fail at matching documents that have minor word overlap but are semantically related.
Generative retrieval seeks to replace the entire in- formation retrieval process with single sequence- to-sequence model capable of mapping queries directly to relevant document identifiers (Met- zler et al., 2021). Differentiable Search Indexes (DSI) (Tay et al., 2022) first demonstrated the po- tential of this paradigm, where T5 is used to pa- rameterize an end-to-end search system, with the model parameters encoding all information about the corpus. See Section 2 for more information. DSI was shown to outperform dual encoder base- line on the Natural Questions dataset (Kwiatkowski et al., 2019). Zhuang et al. (2022b) explores the effectiveness of DSI and synthetic queries on a 100K passage subset of the MS MARCO passage ranking corpus and XOR QA (Asai et al., 2021). Neural Corpus Indexer (Wang et al., 2022) builds on the success of DSI and introduces a combina- tion of more input variants and architectural ad- ditions, some of which we describe and explore in this work. Many works have explored various document identifier designs, including document substring (Bevilacqua et al., 2022), metadata-based approaches (Zhou et al., 2022; Ziems et al., 2023), and learned quantization (Rajput et al., 2023; Sun et al., 2023). More recently, (Chen et al., 2023) pro- poses distillation approach on top of DSI, learning from the rankings generated by dense retrieval us- ing a multi-tash, training loss. Note, this is not to be confused with "generative retrieval models" as used by Lesota et al. (2021), defined via the cumulative probabilities of generating query terms to provide probabilistic perspective on relevance estimation and to offer nuanced measure of uncertainty.
A popular solution is dual encoders (Gillick et al., 2018; Karpukhin et al., 2020; Chen et al., 2022), where pretrained language model such BERT (Devlin et al., 2019) computes low- dimensional dense representations instead of the high-dimensional sparse representations found in BM25. Fine-tuning pretrained models on the target task can further improve the effectiveness of dual encoders. Based on the success of T5 in various nat- ural language understanding tasks, Ni et al. (2022a) proposes scaling up dual encoders by training T5- style pretrained language models with a two-stage contrastive learning approach on the Semantic Text Similarity (STS) tasks. The Generalizable T5 Re- triever (GTR) (Ni et al., 2022b) extends this idea to information retrieval. The most successful GTR models were pretrained on large-scale question- answering dataset curated from the internet and fine-tuned the MS MARCO Passage Ranking task (Nguyen et al., 2016).
Existing approaches often upply synthetic query generation to improve retrieval effectiveness. Nogueira et al. (2019) first leveraged a vanilla sequence-to-sequeene Transformer to train a model that can map passages to queries that it might be able to answer. Nogueira and Lin (2019), doc2query-T5 further improved the effectiveness of the traditional Transformer by leveraging a T5 model. Ma al. (2022) experimented with similar ideas and showed that query generation is effective across a wide range of corpora and task setups.
Prior to generative retrieval, sequence-to- sequence language models, like T5 (Raffel et al., 2020), were shown to be effective for reranking tasks. In this setup, models assign scores to the top-k results from a first-stage retrieval method, One can then use these scores to rerank the doc- uments. For example, monoT5 (Nogucira et al., 2020) was the first to leverage T5 as pointwise
However, none of these works have explored training or evaluating generative retrieval systems corpora larger than O(100K) documents. Given that the generative retrieval paradigm has extended beyond traditional retrieval into areas such as rec- ommender systems (Rajput et al., 2023) and vi- sion (Zhang et al., 2023), we believe our study will be crucial for an evergrowing community.

---

A.2 Implementation Details
and infereecceevvlluatio.. In addition, we find that the diversity of generated synthetic queries can im- pact retrieval effectiveness.
We use 15.1.1 as implemented by t5x (Roberts et al., 2022). We implement the different setups described in Section 2 in the form of seqio tasks. For the MS MARCO variants, we set the maximum input sequence length to 128 for all experiments. Following the NCI setup, we set it 64 for the NQ100K and TriviaQA experiments. We initialize our models with the pre-trained T5-base model. For the PAWA decoder, we randomly initialize the PAWA model parameters. Following (Tay et al., 2022) for sequential DocIDs, beam search, with 40 beams, is used during inference.
Document coverage gap. In Table 1, for cach dataset considered, we report the coverage of their document corpus by the corresponding la- beled query training set. When comparing MS- Marco100k, IM, and FULL, the query coverage drops from a 92.9% to a 51.6% and 5.8%, re- spectively. Consider rows (2a) and (3b) in Table 3, which only differ by the addition of synthetic queries. In these settings, we observe that MS- MarcolOOK improved by 3.3x while MSMarco improved by 3.9x, even though 1M is a larger cor- pus and may be affected by model capacity, we see with MSMarcoFULL. Similarly, for NQ100k and TriviaQA, which have 98.4% and 57.7% cov- erage, respectively, we observe that swapping La- beled Queries (No Indexing) (row Ia) for D2Q only (row 6b) hurts performance for NQ100k while im- proving performance for TriviaQA (Table 2). Since the D2Q model is trained on MS MARCO, for NQ100K, replacing its labeled training queries with synthetic queries only amounted to 1.6% cov- erage gain, which not worth the domain shift. However, for TriviaQA, this amounted to a 42.3% coverage gain, which is more worth the domain shift.
We revise hyperparameter settings from (Tay al., 2022) to ones have found to empirically perform better, especially for indexing larger cor- pora like FUML. We set the batch size in all our experiments to 512. We train our models with a learning rate of 10-3 and a dropout rate of 0.1. We use 10K learning rate warm-up steps for all runs, except for Atomic DocIDs that use 100K steps. We train our small-scale datasets, NQ100K, TriviaQA, and MSMarcol00ff for 1M steps. For MSMarco IM and MSMarcoFULL, model to convergence or, at most, 9M steps. We use 8 TPUv4 chips for training models at the T5-Base scale. T5-Large, T5-XL, and T5-Base with Atomic DocIDs over MSML use 64 TPUv4 chips. For T5-XXL, we use 128 chips. Our most expen- sive runs took roughly 10-14 14 days to train to con- vergence on MSMarcoFULL.
train our
Query distribution gap. Synthetic query genera- tion effectively closes the query distribution gap between training and evaluation. In Table 2, row (7) first shows the importance of the query distribu- tion by using an in-domain query generation model to improve retrieval performance. To further un- derstand the relationship between retrieval perfor- mance and query distribution gap, we plot the rela- tionhhip between synthetic query similarity vs. val- idation query similarity and retrieval performance (MRR @ 10). For each evaluation query in the MS MARCO validation set, measure the maximum similarity among all synthetic queries generated for the corresponding passage. We used Jaccard simi- larity for its simplicity. For each evaluation query, we evaluate MRR @ 10 using the Atomic DocID variant of row 6b in Table 3. Figure I reports the average MRR @ 10 within each 10pt Jaccard sim- ilarity bucket. We plot two variants using 40 and 100 sampled queries per passage for comparison.
A.3 Discussion
The results of this work raise multiple questions regarding the current state of generative retrieval at a scale that we aim to provide more insight into here.
A.3.1 Why are synthetic queries effective?
Although the use of synthetic queries as docu- ment representation technique is effective in pre- vious works (Zhuang et al., 2022b; Wang et al., 2022; Chen et al., 2023), our experiments high- light its central importance to generative retrieval on a larger, more challenging corpus. We suggest that the effectiveness of synthetic queries mainly comes from augmenting the input distribution dur- ing training to be closer to that observed infer- ence/evaluation time. Mainly, this comes in two forms: mitigating the coverage gap of ground-truth labeled queries and the document corpus and clos- ing the gap between the training query distribution
In general, higher Jaccard similarity correlates with higher MRR 10 scores. That is, the more similar our training queries are to the evaluation

---

We consider using 10, 20, 30, 40, and 100 sampled queries per passage, which we construct by first sampling the full 100 queries and then taking ran- dom subsets of varying sizes. We use a sampling temperature of 1.0 and consider the top 10 tokens at each sampling step. Recent studies show advances in utilizing cross encoders to refine the generated query set of incoherent, unspecific queries im- prove the of D2Q (Gospodinov al., 2023). Accordingly, we also experiment with ranking the 100 sampled queries and taking 1p-10020,,3,44 instead of randomly sampling. We achieve this using a state-of-the
MRR @ 10
90
80
D20-100 100
70
D2Q-40
10 20 30 40 50
60
70
80 90 100
Jaccard Similarity (%)
Figure 1: Jaccard similarity between synthetic queries and validation set queries vs. MRR@ 10 on the MS- Marco 100K subset.
cross-attention re-ranker, RankT5-XL (Zhuang et al., 2022a), to score (gener- ated query, passage) pairs and then take the top-k.
We report results in Figure 2. We consistently find that sampling more synthetic queries improves effectiveness in this setting. Surprisingly, ap- plying RankT5-based selection over the samples hurt scores. This suggests an overall preference for more samples and more diverse samples to improve effectiveness. Using all 100 samples scored the best, increasing MRR @ 10 from 80.3 (Table 3. which used 40 samples) to 82.4, closing the gap with ETR--aase (83.2 MRR @ 10) MS- Marco100K. Exactly why query diversity is im- portant is still up for interpretation, but there could be a couple of possibilities: more diverse samples give a higher chance of at lcast some being close to the target distribution, and more examples could provide type of regularization to the model.
82
80
MRR @
78
76
74
RankT5 log-l
Random &
72
samples
10
20
30
10
l:: # of sampled queries
present the MRR @ scores as we vary the number of synthetic queries per passage. Given 100 pre-generated queries per passage, we compare random-k sampling, top-k selection via RankT5-XL, and using 100 synthetic queries.
Figure 2:
For MSMarco100K,
A.3.2 Which model scaling approach is best?
Much of this paper has considered parameter cost as a proxy for memorization capacity, which has been conjectured, in the past, to be important for retrieval (Tay et al., 2022). However, model com- parisons should not stop at parameter counts this may not correlate with other cost indicators (training speed, FLOPS, etc.) that important to practical applications (Dehghani et al., 2022). While ultimately, the best method to scale genera- tive retrieval models will be the one that unlocks the potential of the paradigm to be competitive on large-scale retrieval tasks, we can provide some first glimpses into what trade-offs are at stake as consider larger models for larger corpora.
ones, the better the retrieval effectiveness. Compar- ing the two settings, higher exposure to synthetic queries typically results better scores across the similarity buckets. Even though the query distri- bution is vital, is worth noting that even on the lowest end of similarity, this setting still has strong retrieval effectiveness. While synthetic query distri- bution is crucial aspect of retrieval effectiveness, it is not singular in determining the end effective- ness, and the generative retrieval model goes far beyond simply detecting lexically similar queries to those seen during training.
Diversity. We provide further analysis regarding the importance of synthetic query diversity. Here, we assume the same MSMarco 100K setting using the Atomic DocID variant of row 6b in Table 3. We vary the number of sampled synthetic queries per passage used for training and observe MRR @ 10.
As a case study, consider T5-Base with Atomic DocIDs compared to T5-XL with Naive DocIDs from Table 4. Both are trained only with synthetic queries and represent the only two viable approaches based on our experiments. The PAWA

---

decoder severely underperforms quality-wise we scale model size, not to mention the FLOP expense of having an extra decoding stack during inference. We further discuss parameter cost, training speed, and inference FLOPs here.
results on the MS Marco passage ranking. On small corpora (100K passages), Atomic DocIDs are the highest quality, efficient option without incurring too many extra parameters. Though we can see that training models to maximize memorization amplify compute trade-offs from our experiments, the field must provide more nuanced discussions of cost trade-offss as it considers more realistic applications of generative retrieval.
Parameters. As corpus size scales, generative retrieval models face fundamental prerequisite in model size to achieve decent performance, as seen in Table 3. Between the three different ways of adding parameters (naive scaling, Atomic Doc- IDs, PAWA decoder), we see quality improvements over the smaller models. As discussed, on a fixed parameter budget basis, Naive DucIDs demonstrate the highest effectiveness on MSMarcoFULL.
A.4 Future Directions
While open problems in generative retrieval have not changed (e.g., how achieve state-of- satee--f-thhan-- re- sults on large corpora, how to update such a model with new documents (Mehta et al., 2022)), we be- lieve that our work also raises new open questions for the field. (1) How do we properly leverage large language models and the power of scaling model parameters to benefit generative retrieval on large corpora? While Tay al. (2022) showed this possi- bility over NQ, the same is not yet observed on MS MARCO even though intuitively expanded model capacity should benefit increased corpus scale. (2) How can we design model scaling recipes and de- rive scaling laws that maximize retrieval perfor- mance? In this work, we only consider default T5 parameterizations, which may or may not be optimal for memorization-hess can we design architectures that interpolate be- tween the compute trade-offs of Atomic DoeIDs and Sequential DocIDs? We look forward to un- derstanding more about these vital problems to the success of generative retrieval in future work.
Training Speed. Applications that require fre- quent retraining value fast total training wall time. We train T5-Base Atomic DocIDs and T5-XL Naive DocIDs on the same hardware (64 TPUv4) and hyperparameter settings. To achieve the opti- mal performance reported in Table 4, T5-XL Naive DocIDs required 14 days while T5-Base Atomic DoeID required only ~7 days. However, at ~>7 days T5-XL Naive DocIDs were quality matched with 15-Base Atomic DocIDs (~~ 24.5 MRR @ 10). making both approaches roughly equal in terms of training wall-time when accounting for quality.
Inference FLOPs. Inference FLOPs can be proxy for serving performance, although imperfect. Here, we see that while Sequential DocIDs can achieve more with fewer parameters, Atomic Doc- IDs are incredibly FLOP efficient during inference. TS-Base with Atomic DocIDs for MUML requires only 9.7% the inference FLOPs of T5-XL. with Naive DocIDs for 90% of the retrieval perfor- mance (Table 4). How is this possible? Atomic DocIDs incur additional compute costs to calculate output projection and softmax over the enor- mous vocab of ~8.8M DocIDs. However, it only has compute this once get a complete ranking of the entire corpus - potentially unique prop- erty of the approach. On the other hand, Sequential DocIDs require d decoding steps to decode a single DocID and k beams to find a ranking of k Doc- IDs (k = 10 for our experiments). Thus, even though Atomic DocIDs require expensive out- put projection, Sequential DocIDs require Odd k) more decoding steps. To scale Naive DocIDs be competitive with Atomic DocIDs, also makes each decoding turn significantly more expensive.
tasks. (3) How
A.5 Additional Results
In the remaining pages, we include extended re- sults. Besides MRR @ 10 in Table 3, we report nDCG@20, 20, Hits @ K, and Precision @K K in Table 6 for our experiments on MSMarco, corresponding different rows in Table indexing with the row id. We can see that these metrics correlate well with MRR 10, and our observations remain consistent with those already described.
Finally, we cannot say which approach is the best as the paradigm has yet to achieve competitive

---

Rows in Table
Dataset
DoeID
nDCG@20
Hits@1
Hits@5
Hits@10
20
P@]
P@5
P@ 10
P@2
2a
MSMarcol00
Atomic
0.0
0.01
0.01
0.03
0.01
0.00
0.00
0.00
MSMarcol001
Naive
26.9
19.23
30.10
33.95
38.21
19.23
6.17
1,97
MSKK
Semantic
24.7
15.52
27.58
33.48
39.00
15.52
5.57
3,40
1.99
Za
MSMareol M
Atomic
4.1
1.33
4.20
6.17
9.17
1.33
0.84
0.62
0.46
2a
MSMarcolM
Naive
14.7
9.13
16.83
19.84
23.12
9.13
3.44
2.03
1.19
2a
MSMarcolM
Semantic
11.2
3,91
12.16
17.77
24.17
3.91
2,46
1.80
1.23
2a
M5MurooUULL
Atomic
0.3
0.19
0.27
0.33
0.42
0.19
0.05
0.03
0,02
2a
FUML
Naive
13.3
3,31
13.75
23.44
31,13
3.31
2.83
2,43
1.62
2a
MSMarcoFFULL.
Semantic
5.5
1.32
5.40
9.01
13.28
1,32
1.09
0.91
0.67
36
MSMarco100K
Atomic
83.0
71.42
90.21
93.34
95.24
71:42
18.86
9.83
5.03
36
MSMarcol00K
Naive
82.2
72.08
88. 88.11
91.45
93.45
72.08
18.52
9.65
4.94
36
MSMarco100K
Semantic
82.0
71.26
88.44
91.58
93.81
71.26
18.49
9.63
4.95
36
MSMarcolM
Atomic
62.0
43.78
70.82
78.84
85.14
43.78
14.61
8.18
4.45
36
M
Naive
59.5
42.61
67.92
75.09
81,02
42.61
14.09
7.83
4.24
36
MSMarco M
Semantic
59.0
41.65
66.73
74.73
81,50
41 65
13.78
7.76
4.26
3b
MS
Atomic
31.1
13.54
36.28
47.78
58.12
13.54
7.35
4.87
2.99
3b
MSMarcoFULL
Naive
23.8
5,90
25.42
42.31
$4,40
5,90
5.19
4.39
2.83
3b
MSMarcoFULL
Semantic
10.8
3.12
11.42
17.32
24.58
3.12
2.30
1.74
1.24
4a
MSMarco100k
Semantic
80.3
69.57
86.63
90.44
92.62
69:57
18.07
9.47
4.87
4a
MSMarcn M
Semantic
56.9
38.97
64.91
73.62
80,27
38.97
13.39
7.63
4.18
4a
MSMarcoFULL.
Semantic
13.3
4,80
14.71
20.56
28.02
4.80
2.97
2.08
1.42
5
MSMarco100K
Semantic
80.4
69.66
86.82
90.59
92.74
69. 69.66
18.11
9.49
4.88
MSMarcoIM
Semantic
57.0
38.98
64.87
73.84
80.47
38.98
13.38
7.64
4.19
MSMarcoFULL
Semantic
13.5
4.84
14,80
20.83
28.62
4,84
2.98
2.11
1.45
MSMarcol00K
Atomic
83.6
72.85
90.33
93.55
95.44
72.85
18.91
9.84
5.05
6h
MSMarea100K
Naive
81.9
71.65
88.24
91.20
93.04
71.65
18.53
9.62
4.92
6b
MSMarco100K
Semantic
81.8
70.96
88.35
91.69
93.91
70.96
18.50
9.66
4.96
6b
MSMarcoIM
Atomic
62.8
44.11
71.56
79.87
86.36
44.11
14.82
8.31
4.52
66
MSMarcolM
Naive
61.7
44,76
69.58
77.12
82.81
44.76
14.52
8.07
4.35
6b
MSMarcol M
Semantic
60.6
42.99
68.85 81
76. 83
82.75
42.99
14.23
8.00
4,33
6h
MS Mrrcoo
Atomic
32.5
14.27
37.26
49.64
60.43
14.27
7.56
5.07
3.11
6b
MSMLrcoFULL
Naive
21.9
5.82
22.42
37.28
50.16
5.82
4.59
3.86
2.62
6b
MSMarcoFULL
Semantic
17.5
6,23
19.01
27.35
36.99
6.23
3.83
2.77
1.88
4a'
1SMarco100K
Semantic
81.7
70.72
88.
91.56
94.20
70.72
18.42
9.62
4.97
4of
MSMarcolM
Semantic
60.9
43.61
68.83
77.23
83.54
43.61
14.23
8.02
4.36
4o
MSMarcoFULL
Semantic
24.4
9.37
27.74
38.28
48.08
9.37
5.61
3.89
2.45
4//
1S0K
Semantic
81.7
70.97
88.11
91.40
93.71
70.97
18.44
9.62
4.95
4//
M
Semantic
60.6
42.95
68.74
76.72
82.91
42.95
14.21
7.98
4.34
W
MSMLrcoFULL
Semantic
17.9
6.36
19.41
27.87
37.66
6.36
3.91
2.82
1.91
10Marco100K
Semantic
24.8
6.56
15.60
35.53
66.66
6.56
3.19
3.65
3.46
MSMarenIM
Semantic
13.1
9.44
14.26
16.32
18.81
9.44
2.86
1.64
0.95
FULL
Semantic
24.5
9.50
27.49
38.25
48.22
9.50
5.57
3.90
2.46
Table 6: Results with additional metries for experiments variants of the MS MARCO passage ranking collection considered in Table 3.


Reviews:
[{"review_id": "xKG3sAVhjF", "reviewer": "Reviewer_vuyp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper and are grateful for your constructive feedback. We hope the following can help address your concerns:\n\n1. Evaluation Metrics: We understand your concern about the use of a single metric for each dataset. In our study, we chose the official metric for each dataset we investigated, unlike some of the prior work in generative retrieval evaluation using some non-traditional IR metrics. However, we also agree that including additional metrics could provide a more comprehensive evaluation. The following table expands on the top-scoring methods in our paper from Table 4 (MSMarcoFULL), and also adds two additional sets widely considered in the IR community (TREC DL19/20):\n\n| Model                              | Params | MRR@10 | Recall@100 | NDCG@10 | NDCG@100 | TREC DL 19 NDCG@10 | TREC DL 19 NDCG@100 | TREC DL 20 NDCG@10 | TREC DL 20 NDCG@100 |\n|------------------------------------|--------|--------|------------|---------|----------|--------------------|---------------------|--------------------|---------------------|\n| Base D2Q Only + Atomic ID          | 7.0B   |   24.2       | 80.47   | 29.73    | 36.35              | 49.75               | 50.21              | 53.75               | 56.01 |\n| Large D2Q Only + Naive ID          | 783M   |   21.4       | 82.03   | 27.26    | 34.32              | 49.92               | 46.1               | 45.16               | 49.18 |\n| Large D2Q Only + PAWA (2D Sem.)    | 2.1B   |     19.8       | 73.21   | 24.79    | 31.3               | 46.57               | 45.39              | 50.13               | 50.56 |\n| XL D2Q Only + Naive ID             | 2.8B   |    26.7       | 84.74   | 33.17    | 39.22              | 54.95               | 51.63              | 52.19               | 54.07 |\n| XXL D2Q Only + Naive ID            | 11B    |   24.3       | 81.87   | 30.56    | 36.96              | 52.05               | 47.67              | 49.03               | 52.1 | \n\n\nThese additional densely judged evaluation datasets, as opposed to the sparse judgments in the MS MARCO v1 Passage Ranking Task, seek to address some of the issues with the original task. We see that our original findings generally hold under these new metrics, and believe that this addition reaffirms the soundness of our study. \n\nFor these top-5 Table 4 runs we additionally have {Recall, NDCG, Precision}@{1, 5, 10, 20, 50, 100} and MRR@100. Additionally, for runs in Table 3, we also have Recall@{1, 5, 10, 20}, NDCG@10, and Precision@{1, 5, 10, 20} values. We do not include them here due to the sheer size of the tables, but we plan to include them in the Appendix for our Camera Ready version. If there is a particular comparison you’re interested in on a certain metric we’d be happy to accommodate it during the discussion period. We hope that this further reinforces our findings for the community, given we are the first to tackle this task at this scale with extensive experimentation.\n\n2. Terminology and Abbreviations: Thank you for your feedback. In the revised version, we will ensure that all terms and abbreviations are clearly defined when first introduced. \n\nWe believe these revisions will address your concerns and significantly improve the quality of our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Extensive experiments\n- Findings that can be useful for the IR community for future research", "reasons_to_reject": "- For a complete evaluation, I'd prefer to see more than a single metric for each dataset.\n- The paper can be hard to follow in some parts: It is full of specific names and abbreviations; moreover, some of them are not defined, such as in the results tables  (e.g., At, Nv) - please define and fully describe them. See below for more.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors experiment with the generative retrieval approach in scale. The paper contains an extensive evaluation across multiple experiments, along with the authors' findings. First, the authors evaluate several existing methods on multiple datasets; then they propose their own method and show its strong performance in small scale; and finally they scale up the corpus size from 100k to several millions of passages.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The term \"docid\" is referred to in the introduction, before its definition in Section 2.1."}, "extra_text": {}}, {"review_id": "N9Q69EQJoZ", "reviewer": "Reviewer_Gmvd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and constructive feedback. We would like to address your concerns as follows:\n\n1. Novelty and Methodology: While we understand your concern about the lack of a new method, the primary aim of our paper is to conduct an empirical study of generative retrieval methods at scale, which is currently poorly understood. We believe that this is an especially important contribution to the field given the relatively large number of newly proposed methods that might overstate the effectiveness of these approaches by omitting larger corpora often found in the information retrieval community. In addition, we find it compelling that even without novel methods, our rigorous ablation is able to produce SOTA results on established tasks (Table 2), outperforming more complicated approaches such as GenRet (Sun et al., 2023).\n\n2. Clarification on statements: We apologize if some statements were unclear, and will update the paper to clarify.\nIn Line 123, \"implicitly\" is used generally to refer to methods that expand the parameter count beyond what is initially used in the original T5 model’s pretraining checkpoint. For example, even though initialized at T5-Base, NCI’s PAWA decoder expands the original 220M parameters to 700M+. This fact is not clearly stated in Wang et al. 2022 and models are still referenced by their pretrained size “T5-Base”, and thus we consider it “implicit”. The same goes with the Atomic ID setting where adding more documents to the same base T5 configuration implicitly results in more parameters, with no direct changes to the architecture.\nAn example of the “common misconception” we refer to in Line 140 can be found in Section 4.4 in the Neural Corpus Indexer paper (Wang et al. 2022), “This implies that the model capacity has a critical impact on the retrieval performance …” and “For a larger corpus, one may need to increase the model size to obtain satisfactory performance.” Through our experiments, it is clear that smaller models (T5-Base) would find it hard to properly handle large corpora. However, just scaling from T5-XL to XXL is not sufficient in enhancing downstream effectiveness, and more advancements in modeling are required to meet the effectiveness of SOTA dual encoders. We will clarify these points in the revised version.\n\n3. Evaluation Metrics: We understand your concern about the use of a single metric for each dataset. In our study, we chose the official metric for each dataset we investigated, unlike some of the prior work in generative retrieval evaluation using some non-traditional IR metrics. However, we also agree that including additional metrics could provide a more comprehensive evaluation. The following table expands on the top-scoring methods in our paper from Table 4 (MSMarcoFULL), and also adds two additional sets widely considered in the IR community (TREC DL19/20):\n\n| Model                              | Params | MRR@10 | Recall@100 | NDCG@10 | NDCG@100 | TREC DL 19 NDCG@10 | TREC DL 19 NDCG@100 | TREC DL 20 NDCG@10 | TREC DL 20 NDCG@100 |\n|------------------------------------|--------|--------|------------|---------|----------|--------------------|---------------------|--------------------|---------------------|\n| Base D2Q Only + Atomic ID          | 7.0B   |   24.2       | 80.47   | 29.73    | 36.35              | 49.75               | 50.21              | 53.75               | 56.01 |\n| Large D2Q Only + Naive ID          | 783M   |   21.4       | 82.03   | 27.26    | 34.32              | 49.92               | 46.1               | 45.16               | 49.18 |\n| Large D2Q Only + PAWA (2D Sem.)    | 2.1B   |     19.8       | 73.21   | 24.79    | 31.3               | 46.57               | 45.39              | 50.13               | 50.56 |\n| XL D2Q Only + Naive ID             | 2.8B   |    26.7       | 84.74   | 33.17    | 39.22              | 54.95               | 51.63              | 52.19               | 54.07 |\n| XXL D2Q Only + Naive ID            | 11B    |   24.3       | 81.87   | 30.56    | 36.96              | 52.05               | 47.67              | 49.03               | 52.1 | \n\nThese additional densely judged evaluation datasets, as opposed to the sparse judgments in the MS MARCO v1 Passage Ranking Task, seek to address some of the issues with the original task. We see that our original findings generally hold under these new metrics, and believe that this addition reaffirms the soundness of our study. \n\nFor these top-5 Table 4 runs we additionally have {Recall, NDCG, Precision}@{1, 5, 10, 20, 50, 100} and MRR@100. Additionally, for runs in Table 3, we also have Recall@{1, 5, 10, 20}, NDCG@10, and Precision@{1, 5, 10, 20} values. We do not include them here due to the sheer size of the tables, but we plan to include them in the Appendix for our Camera Ready version. If there is a particular comparison you’re interested in on a certain metric we’d be happy to accommodate it during the discussion period. We hope that this further reinforces our findings for the community, given we are the first to tackle this task at this scale with extensive experimentation.\n\n4. Terminology and Abbreviations: We apologize for any confusion caused by the use of specific names and abbreviations. In the revised version, we will ensure that all terms and abbreviations are clearly defined when first introduced. \n\n5. Related works: As described in our limitations section, we intentionally omit the study of natural language and learned document identifiers from this work. Given the wide search space and the numerous experiments we already present, we believe this is best suited for future work that can rigorously dissect the relationship between retrieval performance and extractive (e.g. Bevilacqua et al., 2022) or abstractive (generated) document identifiers, natural language tokens or learned tokens, and the quality of the underlying docid generation model.\n\nIn addition, the missing references provided seem to be published in proceedings that appeared after the submission of this paper. Regardless, we'd be happy to include them in our related works in the final version.\n\nThanks again for your time and thoughts! We believe that these changes and additions will significantly improve the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors conduct a detailed empirical study of generative retrieval methods about corpora scales, model scales, modules, etc. \n2. The authors provides some interesting insights and also point out several valuable research directions, which makes a significant contribution to the field. ", "reasons_to_reject": "1. The paper lacks of novelty in terms of methodology. After such a detailed analysis, a better method based on the findings is encouraged. \n2. Some statements in this paper lack detail and analysis. \n   - In Line 123, \"Methods that **implicitly** increase model parameters perform better using the same T5 initialization.\". What does the \"implicitly\" refer to, and what will the corresponding \"explicitly\" refer to? \n   - In Line 140, \"We find this counter-intuitive to the common conception of generative retrieval being limited by model **capacity**\". What does the model capability here refer to, and why is the capability of a larger model (11B parameters) low? \n3. The experiments are not very convincing.\n   - Only one metric is compared on each dataset. Additionally, on large-scale corpora, the \"@10\" metric may be overly strict, and the author should consider adding metrics based on \"@100\". \n   - Some related work (i.e., substring-based docids) has been omitted, which is also a major research branch in the field of generative retrieval, e.g., SEAL (Bevilacqua et al., 2022). I think a comparative analysis and discussion should also be carried out.\n4. In Tables, some abbreviations are used without explanation of their meanings, e.g., At., D2Q. A clear explanation is suggested to provide in summary. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors conduct an empirical study of generative retrieval methods, providing a detailed comparison and analysis of the impacts of modules on corpora of different scales. The authors consequently uncover several findings about generative retrieval methods. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some studies that might be considered for discussion:\n\nLi, Yongqi, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. “Multiview Identifiers Enhanced Generative Retrieval.” In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2023.\n\nTang, Yubao, Ruqing Zhang, Jiafeng Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. “Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies.” In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. ACM, 2023."}}, {"review_id": "ydTLMpblJX", "reviewer": "Reviewer_JoZg", "summary": "", "questions": "What different do you think the findings would be, when the methods are evaluated on other large-scale datasets?   ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time in reading and reviewing our paper! We’re glad you recognize the contributions we put forth in this paper. We believe that this is an especially important contribution to the field given the relatively large number of newly proposed methods that might overstate the effectiveness of generative retrieval approaches by omitting larger corpora often found in the information retrieval community.\n\nRe: Additional datasets. Although given the time required to train the models (see Section A.3.2, i.e. 7-14 days) it is not possible to index additional datasets during the response period, we did want to present some more metrics and evaluations on other datasets to provide a more comprehensive overview.\n\nThe following table expands on the top-scoring methods in our paper from Table 4 (MSMarcoFULL), and also adds two additional sets widely considered in the IR community (TREC DL19/20):\n\n| Model                              | Params | MRR@10 | Recall@100 | NDCG@10 | NDCG@100 | TREC DL 19 NDCG@10 | TREC DL 19 NDCG@100 | TREC DL 20 NDCG@10 | TREC DL 20 NDCG@100 |\n|------------------------------------|--------|--------|------------|---------|----------|--------------------|---------------------|--------------------|---------------------|\n| Base D2Q Only + Atomic ID          | 7.0B   |   24.2       | 80.47   | 29.73    | 36.35              | 49.75               | 50.21              | 53.75               | 56.01 |\n| Large D2Q Only + Naive ID          | 783M   |   21.4       | 82.03   | 27.26    | 34.32              | 49.92               | 46.1               | 45.16               | 49.18 |\n| Large D2Q Only + PAWA (2D Sem.)    | 2.1B   |     19.8       | 73.21   | 24.79    | 31.3               | 46.57               | 45.39              | 50.13               | 50.56 |\n| XL D2Q Only + Naive ID             | 2.8B   |    26.7       | 84.74   | 33.17    | 39.22              | 54.95               | 51.63              | 52.19               | 54.07 |\n| XXL D2Q Only + Naive ID            | 11B    |   24.3       | 81.87   | 30.56    | 36.96              | 52.05               | 47.67              | 49.03               | 52.1 | \n\n\n\n\nThese additional densely judged evaluation datasets, as opposed to the sparse judgments in the MS MARCO v1 Passage Ranking Task, seek to address some of the issues with the original task. We see that our original findings generally hold under these new metrics, and believe that this addition reaffirms the soundness of our study. \n\nFor these top Table 4 runs we additionally have {Recall, NDCG, Precision}@{1, 5, 10, 20, 50, 100} and MRR@100. Additionally, for runs in Table 3, we also have Recall@{1, 5, 10, 20}, NDCG@10, and Precision@{1, 5, 10, 20} values. We do not include them here due to the sheer size of the tables, but we plan to include them in the Appendix for our Camera Ready version. If there is a particular comparison you’re interested in on a certain metric we’d be happy to accommodate it during the discussion period. We hope that this further reinforces our findings for the community, given we are the first to tackle this task at this scale with extensive experimentation.\n\nAs a final note, we do want to add that regardless of the flaws with MS MARCO, it is nevertheless important for retrieval models to work on a variety of possible use cases and be robust to various datasets. Our results show that generative retrieval systems still have a ways to go to perform well on the class of queries present in MS MARCO, a dataset that has been widely studied by the IR field.\n\nAgain, thank you for your comments! These changes will surely strengthen our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is well-written and decently covers various generative ranking models. As an empirical study, it well positions the state of the research in this field, and also suggests possible ways to move forward.\n- The paper studies the large corpus setting, largely ignored in previous studies.\n- The findings are interesting and could be valuable for the researchers in the field.", "reasons_to_reject": "My main concern is that (in terms of large-scale IR collections) the study limits itself to MS MARCO Passage retrieval. As mentioned by the study and also in several previous works, the dataset has its own limitations, and the achieved results might not draw a full picture but reflect the specificities of this dataset. As an empirical study, it would be great to bring in other large dataset with different characteristics (such as TripClick) to be able to reach to stronger conclusions. ", "questions_to_authors": "What different do you think the findings would be, when the methods are evaluated on other large-scale datasets?   ", "paper_topic_and_main_contributions": "Te work conducts an empirical study on generative retrieval - a timely topic with growing interests in the community - with the particular focus on examining the existing methods in a large scale setting. The paper reviews and categorizes various existing methods based on the ways of document representation, synthetic query generation, and creating document identifiers. Moving beyond middle-scale Q/A datasets, the study evaluates the methods on MS MARCO Passage retrieval, leveraging the T5 models with various sizes. The findings highlights the effectiveness of query generation, and suggest possible future directions for this topic. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models\nLesota O., Rekabsaz N., Cohen D., Grasserbauer K., Eickhoff C., Schedl M.\nIn proceedings of the 7th ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR), July 2021"}}]

Author rebuttal:

Meta-review: This paper is an empirical study of generative retrieval techniques, carried out at a scale that is beyond what has been previously studied. The overall finding is that generative techniques are effective at smaller scales, but more research is needed for scaling to much larger settings.

Reviewers found this paper to be well-written, a useful empirical study, and likely of value to parts of the community ("could be valuable for the researchers in the field"). 

The main concerns among reviewers had to do with the diversity of results, as the primary focus was on a single metric on a single large scale dataset. In their rebuttal, the authors have helpfully provided additional results which include additional metrics and additional datasets, and this seems to have allayed the first concern at least. Nevertheless, two reviewers remained ambivalent in terms of excitement, even after taking the rebuttal into consideration. Unfortunately, one of these reviewer was somewhat confused about whether new experimental results could be considered as part of the rebuttal. The policy this year is a departure from past years, so it is understandable that they thought such revisions might be off limits, when in fact new results were within the scope for evaluation.

All reviewers agreed that this paper is worthy of publication. However, the one reviewer who was by far the most confident, who also engaged with the authors during the rebuttal, remained unconvinced to increase their excitement score above ambivalent. On the one hand, they suggest that "this paper will be an excellent and comprehensive empirical study that will benefit the research field of generative retrieval". On the other, they remain concerned that the paper is limited to generative retrieval based on only virtual tokens, and as such, "the impact of this paper is still limited", and they have not been convinced to raise their scores above 3.

Ultimately this paper is somewhat difficult to place. There seems to be consensus that this is a paper that will be interesting and useful to those who work on generative retrieval, despite being primarily an empirical study. That being said, the authors were unable to convince two reviewers to increase their scores, despite providing new results that seem to be precisely what these reviewers were seeking.
Decision: Accept-Main