DiffusionRet: Diffusion-Enhanced Generative Retriever using Constrained Decoding
Authors: Shanbao Qiao
Xuebing Liu
Seung-Hoon Na
Venue: EMNLP 2023 Findings (2023)
Keywords: Infromation Retrieval
Diffusion Model
Generative Retrieval
Model-based Retrieval

Abstract: Generative retrieval, which maps from a query to its relevant document identifiers (docids), has recently emerged as a new information retrieval (IR) paradigm, however, having suffered from  1) the $\textit{lack of the intermediate reasoning step}$, caused by the manner of merely using a query to perform the hierarchical classification, and 2) the $\textit{pretrain-finetune discrepancy}$, which comes from the use of the artificial symbols of docids. To address these limitations, we propose the novel approach of using the document generation from a query as an intermediate step before the retrieval, thus presenting $\underline{diffusion}$-enhanced generative $\underline{ret}$rieval ($\textbf{DiffusionRet}$), which consists of two processing steps: 1) the $\textit{diffusion-based document generation}$, which employs the sequence-to-sequence diffusion model to produce a pseudo document sample from a query, being expected to semantically close to a relevant document; 2) $\textit{N-gram-based generative retrieval}$, which use another sequence-to-sequence model to generate n-grams that appear in the collection index for linking a generated sample to an original document.  Experiment results on MS MARCO and Natural Questions dataset show that the proposed DiffusionRet significantly outperforms all the existing generative retrieval methods and leads to the state-of-the-art performances, even with much smaller number of parameters.

DiffusionRet: Diffusion-Enhaneed Generative Retriever using Constrained Decoding
Shanbao Qiao and Xuebing Liu and Seung-Hoon Na Center for Advanced Image and Information Technology, Department of Computer Science and Artificial Intelligence, Jconbuk National University {joe, liuxubbnn,, nashu@jbnuu.cc. ac. kc.
Abstract
et al., 2022). In generative retrieval, because all the required information and knowledge for indexing and retrieval are maintained in parametric memory, retrieval is conducted efficiently at the inference stage by a decoder, without requiring other inverted index structures or tools for the nearest neighbor search, such as FAISS (Johnson et al., 2019).
Generative retrieval, which maps from a query its relevant document identifiers (docids), has recently emerged trieval (IR) paradigm, however, having suffered from 1) the lack of the intermediate reasoning step, caused by the manner of merely using a query to perform the docids generation or hierarchical classification, and 2) the pretrain- finetune discrepancy, coming from the use of the artificial symbols docids. To address these limitations, propose novel approach of using document generation from query as an intermediate step before retrieval, thus pre- senting diffusion-enhanee generative retrieval (DiffusionRet), which consists of two process- ing steps: 1) diffusiin--ased document genera- tion, which employs the sequence-to-sequeme to-sequence diffusion model to produce a pscudo-document sample from query and is expected be se- mantically close to a relevant document. 2) N-gram-based generative retrieval, which uses another sequence-to-sequenee to-sequence model to gen- erate n-grams appearing in the collection in- dex for linking a generated sample to an orig- inal document. Experimental results on the MS MARCO and Natural Questions datasets show that the proposed DiffusionRet signifi- cantly outperforms all existing generative re- trieval methods, leading to state-of-the-ar per- formance, with much smaller number of parameters.
new information re-
However, the classical methods of existing gener- ative retrieval mostly produce only docids at infer- ence time, much shorter than normal sentences, while no other content is generated (Tay et al., 2022; Zhuang al., 2022; Wang al., 2022). Lacking an intermediate reasoning step, it is un- clear what detailed reasoning steps proceed during decoding-baat based hicrarchical classification 2 or do- cids generation, which is likely far from the im- proved approach based on the chain-of-thought (Wei et al., 2022; Trivedi et al., 2023). Addition- ally, generative retrieval extensively uses artificial symbols of docids for target sequences. As noted in (Ren al., 2023), however, these docid-like symbols are only seen during the finetuning but are unseen the pretraining stage, thus causing the pretrain-fiettune discrepancy, largely limiting the effect of transfer learning, which is arguably important when exploiting pretrained models.
To address these limitations, we establish the document generation stage as key intermediate reasoning step. that directly generates the "actual content" of document relevant to given query. Once document is generated, we use it as a new document- query, generate N-grams restricted to those occurring in the collection, and use them to identify its original source document. To gen-
1 Introduction
Generative retrieval methods have emerged pow- erful IR paradigms (Tay et al., 2022; Zhou al., 2022: Wang al., 2022: Chen al., 2022a; Ren al., 2023; Bevilacqua et al., 2022; Tang al., 2023a), by employing seguence-to-sqqueme mod- els to map a query to relevant docids, showing promising results compared with dense retrieval based on dual encoders (Karpukhin et al., 2020; Ni
Here we primarily focus on the classical type of genera- tive retrieval (Tay et al., 2022; Zhuang et al., 2022; Wang et al., 2022) where the hierarchical clustering process is deployed to assign semantically structured docids, and interprets its gener- ation of a sequence of docids as the hierarchical classification. strongly bused on (Risch et al., 2020), which views the hier- archical document classification as a sequence-to-secn to-squennce task.
'Our code and models are available at https://githu/ com/Miracle081

---

erate such a pseudo query-relevant document, the diffusion model emerges as noticeably promis- ing alternative to the parameter-intensive large lan- guage models (LLM) (Li et al., 2022b), surpassing some GPT variants even with a much smaller pa- rameter scale, as in (Gong et al., 2023; Yuan et al., 2022; Gao et al.. 2022b; Li et al., 2023a; Tang et al., 2023b; Bi et al., 2023). Pursuing the use of the reasoning chain of document-ngrams as an intermediate step, inspired by the aformentioned notable parameter efficiency of diffusion models, we propose two-stage diffusion-cnhance genera- tive retrieval (DiffusionRett model, which consists of two processing steps, as follows:
(Bevilaequa et al., 2022) that shorter n-grams are less informative, we empirically consider long n-grams with length of 30, more likely to be unique to a specific document 3. Using n-gram identifiers, our generative re- trieval model does not suffer from the pretrain- finetune discrepancy because special tokens such docids are not necessary.
To the best of our knowledge, this is the first study where the diffusion model is deployed to generate a realistic document sample for the IR task.
Our contributions are summarized as follows: 1) propose DiffusionRet, two-stage generative retrieval model based on the reasoning chain of document-ngram, without requiring special tokens such as docids, 2) we apply a diffusion model for document generation as a textual version of the semantic inpainting. 3) The proposed RiffusionRet shows a state-of-the-ar performance on the MS MARCO and Natural Questions (NQ) datasets.
Diffusion-based document generation, which extensively applies the sequence- to-sequence diffusion model (Gong et al., 2023; Yuan et al., 2022) to generate pscudo- documents for an input query, similar to text-based semantic inpainting, where full contents of a document are infilled given query as context. Although pseudo- document is not the same as the original one, is expected that it will look like real documents in length, format, and content, and contain semantically enriched content of an original query. By providing a useful intermediate representation with this enriched information, a generated pssudetdocument allows us to perform in-depth semantic matching at the document-to-doccumee level in subsequent retrieval. Furthermore, like (Ren al., 2023), the data distribution mismatch between the training and inference, an issue faced by the classical generative retrieval, is somehow effectively handled by putting the 'intermediate reasoning step". as the subsequent retrieval phrase takes not queries but extended aseudo-documente produced by the diffusion model, whose lengths and formats exhibits the regularity, aligning closely with those observed during indexing phrase.
2 Related Works
2.1 Diffusion Models for Text Generation
Diffusion models (Sohl-Dickstein et al., 2015; Song and Ermon, 2019; Ho et al., 2020; Yang et al., 2022) have emerged as new and prominent ap- proaches exhibiting remarkable performances such as the generation of high-quality images, particu- larly in computer vision tasks (Saharia et al., 2022; Li et al., 2022a: Lugmayr al., 2022). Recently, diffusion models have been extensively studied for text generation tasks (Li et al., 2022b; Gong et al., 2023; He et al., 2022; Yuan et al., 2022; Ye et al., 2023). DiffusionL.M (Li et al., 2022b) presents one of the carlier diffusion models for controlled text generation, but its controlling settings are limited to simplified conditions, such parts-of-speech and syntax trees. DiffuSeq (Gong al., 2023) extends the diffusion model eqquence-to-seqeene tasks, enabling fine-grained controlled generation using a text source as the control condition. More recently, SeqDiffuSeq (Yuan et al., 2022) extended Diffu- sionLM to sequence-to-sequencce setting by equip- ping it with self-conditioning technique (Chen
N-gram-based generative retrieval, which takes pseudo-document resulting from the diffusion model and generates n-grams based (Bevilacqua et al., 2022), another sequence- to-sequence model where decoding is re- stricted the set of FM-indexed n-grams in the collection. Given the experimental evidence the effect of n-gram length in
We do not claim that using long lengths of n-grams leads to the improvement in the retrieval performance. Instead, the n- gram-bused generative retrieval does not need the requirement of the "extensive data augmentation" like query expansion. which is present in SEAL (Bevilacqua et al., 2022), possibly due to our two-stage setting that handles the data distribution mismatch hetween training and inference.

---

al., 2022b; Strudel et al., 2022) and by proposing a novel adaptive noisy schedule. In this study, we extensively utilize SeqDiffuSeq to generate docu- ment samples for the IR problem, because it is a more advanced variant, whereas other advanced dif- fusion models could be deployed in our proposed method.
tive form and employs a prompt learning strategy enable better generalization across different tasks. MINDER (Li et al., 2023b) introduced novel syn- thetic identifiers generated from a passage's ' content and proposed multi-view identifiers based on three different types - titles, sub-strings, and synthetic identifiers (i.e., pseudo-queries) - to facilitate the holistic views of passages.
2.2 Generative Retrieval
For document-level generation, query2doc (Wang et al., 2023) proposes a query expansion method by few-shot prompting LLMs to generate pseudo-documents. However, the performance of query2doc is significantly influenced by the model size, as they mentioned that the model shows the best when combined with the largest LLMs (i.e., 175B) while small models yield only slight im- provements. Unlike query2doc with this depen- dence on LLMs, our diffusion- bassed generation enables to generate high quality pseudo-document with the considerably fewer model parameters.
Generative retrieval (Metzler et al., 2021; Tay et al., 2022; Wang et al., 2022; Sun et al., 2023; Zhuang et al., 2022; Tang et al., 2023a; Zhou et al., 2022; Bevilacqua al., 2022; Chen et al., 2022a, 2023; Ren et al., 2023), also called the Differentiable Search Index (DSI) (Tay et al., 2022), has emerged as a new IR framework, integrating both the index- ing and retrieval in a single text-to-text transformer without requiring an external index. Starting with the pioneering work of DSI (Tay et al., 2022) which trained T5 models to directly generate docids from query, some studies adopted separate query gen- eration modules to store mapping knowledge in the model parameters (Zhuang et al., 2022; Wang et al., 2022).
The work most similar ours is TOME (Ren et al., 2023), which also proposes a two-stage ap- proach by first generating pscudo-documents, How- ever, their second-stage retrieval deployed Web URLS as docids, which are semantically less mean- ingful than our use of n-grams. Moreover, we use diffusion models for document generation, which significantly reduce the parameter size (i.e., 103M parameters), compared their case, which uses either T5-3B or TS-large. The experimental results show that DiffusionRet leads to noticeable improve- ments over TOME the MS MARCO dataset, as discussed in Section 4.
Instead of using a sequence of docids for each document, DynamicRetriever (Zhou al., 2022) proposed method that leverages the document em- beddings resulting from the pretrained dual encoder to initialize the output embeddings of docids in the decoder and then finetunes the encoder-decoder to capture the query-doeid relations.
Numerous studies have used semantically inter- pretable docids, without relying on artificial tokens for docids (Tang et al., 2023a; Bevilacqua et al., 2022: Chen et al., 2022a, 2023; Ren et al., 2023). SE-DSI (Tang et al., 2023a) proposed semantically more meaningful docids based on query genera- tion and adopted rehearsal strategies to better as- sociate a document with its docids. There are also some recent works based on pre-trained language models. SEAL (Bevilacqua et al., 2022) generates n-grams based on the FM-index (Ferragina and Manzini, 2000) to constrain the decoding of the autoregressive model to restrict the search space of the n-grams. CorpusBrain (Chen et al., 2022a) uses the Wikipedia article titles docids to per- form multiple pre-training tasks to encode knowl- edge in the corpus for handling variety of down- tasks. UGR (Chen et al., 2023) exploits n-gram- based identifiers to unify different retrieval tasks into single genera-
3 Method
Figure presents the overall architecture of Dif- fusionRet used to form a document-ngram reason- docu-
ing chain, consisting of 1) diffusion ment generation and 2) ngram-based generative retrieval. Formally, suppose that a query is given x, C = {di,... dx} is collection of documents where d, is i-th document, V is a vocabulary set of tokens, n is the maximum length of the sequence, and is the fixed length of n-grams to be gener- ated. RiffusionRet sequentially performs Diffu(-), the diffusion based document generation, and the n-gram generation, Genngram (.) to compute the
stream knowledge-intenste

---

Documents Collection
Retrieval Document d:
Query x:
Doc !...
acid rain is a broad term that often used to describe several forms of acid deposition. wet deposition is when rain....
Describe what is acid ruin.
Acid rain is broud ierm that is often used describe several forms of acid deposition. wer deposttion is when rain, snow, fog, mist contains high amounts sulfuric and nitric acid.
Seq2Seq Diffusion Document Generation (xf
Scoring
Ro ④)
Doc
Generated n-grams Q:
FM-index
acid rain broad term persistent acid roin reduces tree describe several forms of oold
Generated Pseudo Document q:
N-gram based Generative Retrieval
acid rain is broad term that is often used to describe several forms of acid toxin. by generol, when rain dried, it contains the diverse amount of precipitation..
that often used describe react the
Gen (
deposition is when rain.... water in the oir react ...
Constrained decoding
Figure I: An overall architecture of the proposed DiffusionRet, formulated in Eq. (I): i) Given a query the diffusion- based document generation, denoted as (x) repeatedly applies the reverse denoising function of Eq. (3) and finally the rounding function to generate a pseudo-document q (i.c., Eq. (4)). ii) The N-gram generative retrieval applies Genegram (q) to generate a set of n-grams Q from q, under the restricted space of the FM-index of the collection and then computes the relevance score Rg(d, q) as in Eq (6).
relevance score of document d E C as follows:
i.e., d E R 4, Unlike continuous diffusion models, d is first converted to a list of continuous vectors the embedding spaces using Emb(d), adopted by (Li et al., 2022b):
q Diffu(x)
Q Genngram
Score(d, (d, Re(d,q)
(1)
q (yol d) N (yo; Emd S6I)
(2)
where Diffu(-) is the reverse process of the sequence-to-sequenee diffusion model of (Yuan al., 2022), as described in Section 3.1, Genngram ()) and Re(-) are the constrained decod- ing process and the scoring function of (Bevilacqua et al., 2022), respectively, as described in Section 3.2. The generated document and n grgamms (i.e., and Q) are used as key representations for interme- diate reasoning in DiffusionRet.
where 3r is hyperparameter referring to the amount of noise added at the time step 4, Emb(d) is trainable embedding function defined as Emb(d) = [Emb(d)),-
Emb(d,)), in which de refers to i-th token of d. The diffusion pro- cess of the subsequent time steps is the same the standard one (Li et al., 2022b); for each time step E {1...., ,T, the diffusion injects the ran- dom noise using the distribution q (yelye-1) N (yts Vory. 1, II)
3.1 Diffusion-based Document Generation
For Diffu(x), we deploy SeqDiffuSeq (Yuan et al., 2022), a variant of the sqquence-t--sqqeeee diffu- sion model, for controlled generation at a given x As illustrated in Figure 2, the diffusion process consists of I) the forward diffusion process, which gradually adds random noise to fit into standard Gaussian distribution at the final time step, and 2) the reverse denoising process, which gradually denoises random noise to generate synthetic sam- ple.
Reverse process In the reverse process, the de- noising model generates a sample y conditioned on both y: and x, using the denoising distribution Py as follows:
Po (y--1ly,, no Xy,, A, ED] (y,, x,t) + B, y3)3)
N (y1-1;440 x,t) ,8I)
where EDI xy,,x,t) is the denoising function based on the encoder-decoder architecture, that
Suppose a document in to be diffused is provided as an array of one-hot vectors,
Forward process
*Here, d a version of one-hot vectors for d, E

---

takes x in the encoder and denoises the noisy out- put embeddings Yı in the decoder, and A, and B, are functions of and 3,, defined in Appendix B. Importantly, when t = 0, the rounding mecha- nism (Li et al., 2022b) is applied on the generated hidden vector Yo, which is mapped back to the near- est token in the embedding space by the rounding function Ro)
Now, let P(g]q) be the generative probability of g obtained by the decoder. The n-gram weight function is defined as follows (Bevilacqua et al., 2022):
(1 P(o)) w(g. q) = maa (0.10g P()) P(gla))
argmax,Poo( (dl50) where po yo) is the multiplications of the softmax dis- tributions over tokens after projecting Yo using the embedding matrix, and list of one-hot vec- tors.
For each document d E C. we obtain Q(d) ç Q set of the n-grams with the highest weights that do not overlap each other in the document, so that maximizes Egegre w(g. q).
Finally, the relevance score of d is the weighted sum of all n-grams in Q(d) as follows:
In summary, (if is the function that repeat- edly applies the denoising steps and rounding mech- anism at the final timestep, as follows:
Ro(d, q)
E w(g. (1- (set(g) n (()))) (6) 3EQ(4)
Yr ~ NO,,I)
[set(g)]
- ~ pe (yt,1
q = (yo)
(4)
where Y, R are hyperparameters, set(g) is the set of tokens in g, and C(g) is the coverage set, the union of all other n-grams in Q with a higher score, defined follows:
The self-conditioning of (Chen et al., 2022b; Yuan et al., 2022) is also incorporated into the de- noising function EDA (yr,x, x, L). in a manner that the former prediction Y, is further concatenated with the noisy sample yt. as presented in Appendix B.
C
U
set(g')
(7)
We train the denoising function (-) and em- bedding parameters using the training objective of (Yuan et al., 2022), as presented Appendix C.
Details of the contrained decoding for Genngram (q) using the FM-index are present in Appendix D. It is also worth noting that SEAL (Bevilacqua et al., 2022) uses uniformly sampled spans from documents in the collection as 'uussupervieed" samples to train, aiming to expose the model to more pieces of evidence. In our setting, the generated documents contain semantically-rich contextual information; thus, we do not require such additional data.
3.2 N-gram-based Generative Retrieval
The remaining part is to apply Genngram (.) and Re(-) for retrieving an original document from the sampled pseudo-document q (x). Fol- lowing (Bevilacqua et al., 2022), we first train a sequence-to siveence-t-sequueee -eeuueece model to generate n-grams on FM-indices of (Ferragina and Manzini, 2000), referred to as Genmgram (-). Suppose that Q is set of n-grams resulting from Genngram (q). where each n-gram occurs at least once in the given collec- tion C. Then, we compute Reld, q). the relevance score of a document E by checking how is matched well with the generated n-grams Q.
Experiments
4.1 Datasets
We conducted experiments using two datasets: MS MARCO (Nguyen et al., 2016) and Natural Ques- tions (Kwiatkowski al., 2019).
MS MARCO. MS MARCO dataset was collected from Bing search queries and the corresponding web page documents. For the passage retrieval task, the full collection contains approximately 500k query-document pairs. For a fair comparison with previous works, we sample the 100k subset fol- lowing (Zhuang et al., 2022; Ren et al., 2023) for
To formally describe Ro(d, q) as in (Bevilacqua et al., 2022), for each fixed-length n-gram E with [ø| k. we first compute its normalized frequency from the FM-index follows:
C)g,C)
P(g) - Edec [d]
(5)
'Here, Id is the number of the occurrence of all fixed length n-grams in d, which makes Fq. (5) to sum to over all n-grams.
where c(g,C) is the frequency of the n-gram g in the collection and [d] is the length of document

---

Query
(o)
(Emb(1)) ve-1)
(Emb(X))
(Emb(x); yr)
What is uniform distribution in biology?
Embedding (Cold)
Noising 4('ebr--1)
Uniform distribution a statistical distribution which every possible outcome equal chance, likelihood, occurring (1 out of the total number of outcomes).
Po (o) Rounding
p(rr-117,,)) Denoising
Document
Gaussian Noise
Figure 2: The overall processes involved in diffusion brastssooo-baeddddd document generation, utilizing sauuence-to-sqquenee approach: i) the forward process first applies the embedding function Emd(-) to convert a discrete sequence to continuous vectors and then follows the diffusion process using 4 (y-1): gradually add Gaussian noise during the remaining time steps; ii) The reverse process performs the denoising function EDj (ye, x, x, under Pe (Yt-1lYt, x) of Eq. (3) during time steps until = 0 and finally applies the rounding function Round(-), converting continuous vectors discrete sequence, as summarized in Eq. (4).
N (yr: ya,yt-1, 1, (1 a;)I)
training, referred to as MS MARCO 100k. For evaluation, we use the "full dev" set containing 6980 query-document pair samples. We also test model 300k subset to explore the zero-shot setting of the diffusion model.
related queries based on the document, con- catenates them to the end of the document, and then performs the BM25 retrieval. For dense retrieval methods:
Natural Questions. Natural Questions dataset was collected from natural language questions and Wikipedia pages using Google. We use the pro- cessed dataset following NCI (Wang et al., 2022) and SEAL (Bevilacqua et al.. 2022), which contain- ing approximately 307k query-document training pairs. For evaluation, we also use the "full dev" set for evaluation containing 7830 test samples, referred to NQ 320k, as in (Tay et al., 2022).
DPR (Karpukhin et al., 2020) is the most representative dual-encoder model based on FAISS index (Johnson et al., 2019).
ANCE (Xiong et al., 2021) is dual-encoder model using hard negative samples for train- ing, based on an approximate nearest neighbor search.
For generative retrieval methods:
DSI (Tay et al., 2022) a pioneering work on generative retrieval that jointly trains indexing and retrieval tasks based on the T5 (Raffel et al., 2020) model.
4.2 Evaluation Metrics
We report the Hits@1 and Hits @ 10 metrics, indi- cating the retrieval success of the top-1 and top-10 documents in the ranking list among all evaluation samples, respectively.
DSI-QG (Zhuang et al., 2022), in addition to DSI, generates multiple queries for docu- ments for data augmentation and trains index- ing models with those generated queries tackle the data distribution mismatch in DSI. SE-DSI (Tang et al., 2023a) another work based on DSI that uses learning strate- gies from Cognitive Psychology to induce a semantic-enhanced DSI model.
4.3 Baseline
We selected subset of prevalent sparse/dense re- trieval methods and a set the recently pro- posed generative retrieval methods as baselines for comparison.
For sparse retrieval methods:
BM25 (Yang et al., 2017) is a classical sparse retrieval model that evaluates the similarity between queries and documents based TF- IDF for retrieval.
SEAL (Bevilacqua et al., 2022) is a gener- ative retrieval model generating n-grams of documents from query to enable retrieval, and our second-stage retrieval model is highly based on this work.
docT5query (Nogueira et al., 2019) is another strong sparse retrieval method based on doc- ument expansion that first predicts possible
NCI (Wang et al., 2022), referring to Neural Corpus Indexer, adopts multiple query gen-

---

Model
Hits @1
Hits@ 10
Model
Hits@ 1
Hits@ 10
BM25
46.50
73.90
BM25
29.27
60.15
BM25 docers
68.50
92.70
BM25 + docrsS
39.13
69.72
DPR
65.00
91.90
DPR
50.20
DSI-base
2.10
7.10
ANCE
52.63
DSI-large
10.40
23.00
DSI-large
35.60
62.60
SE-DSI
53.47
75.28
DSI-XL
39.10
66.80
DSI-QG-base
59.20
81.10
DSI-XXI.
40.40
70.30
DSI-QG-large
58.40
80.90
DSI-QG-base
63.49
82.36
TOME
71.93
DSI-QG-large
65.13
82.50
DiffusionRet 85.29
96.62
SEAL
59.93
74.50
NCI
66.23
Table Retrieval performances (Hits@k) on test sets MS MARCO 100k, comparing DiffusionRet with other previous baselines.
TOME
66.64
GenRet
68.10
DiffusionRet
71.66
80.23
eration strategies and a prefix-aware weight- adaptive decoder improve the retrieval model.
Table 2: Retrieval performances (Hits on test sets of NQ 320k, comparing DiffusionRet with other previous baselines.
TOME (Ren al., 2023) a two-stage gener- ative retrieval method partially similar to our architecture, but it uses the T5 (Raffel et al., 2020) model to generate documents, and the generated documents are used in another T5 model generate possible web URLs, with the URL the document identifier.
Training Diffusion Model. To train the diffusion model for document generation in Section 3.1, we followed the work of SeqDiffuSeq (Yuan et al., 2022) using a 12 layers encoder-decoder Trans- former (Vaswani et al., 2017) model as the back- bone. We set the diffusion steps to 2,000, as in the most previous works (Li et al., 2022b; Gong et al., 2023; Yuan et al., 2022). Some generated examples using the diffusion model are presented in Appendix
GenRet (Sun et al., 2023) is generative re- trieval method that learns to tokenize docu- ments into short discrete representations to define document identifiers better using dis- crete auto-encoding approach.
Training Retrieval Model. To train the generative retrieval model in Section 3.2, SEAL (Bevilacqua et al., 2022) was adopted but under modified set- tings for our case. In our training dataset, we built set of pairs of poseudo-docunne generated pscudo-documne while target n-grams sampled from the original "ground truth" document, not from the generated one. Beyond the use of short n-grams, we gathered 10, 30-long n-grams from the original document using SEAL's sample strategy. For each pseudo- document, we sampled 10 target n-grams of se- quence length 30 from th e corresponding ground truth document using SEAL's sample strategy.
For comparison, we present the results reported in the original paper as baseline performance. When multiple same-method performances are mentioned in other studies, we report the best per- formance.
n-gram), where is source sequence,
4.4 Implementation Details
Data Augmentation. To train DiffusionRett we applied data augmentation by directly performing query generation or adopting existing generated queries. For the MS MARCO 100k dataset, we used the model in the doc' 15query (Nogueira et al., 2019) pre-trained the MS MARCO dataset, to generate 5 corresponding queries for each training document. For the NQ 320k dataset, used the same dataset as previous studies (Wang et al., 2022: Bevilacqua et al., 2022) where 10 queries were generated for each training document 6
For the MS MARCO 100k dataset, we per- formed sampling process of the diffusion model the original 100k queries, obtained generated pseudo-documents, and finetuned the BART-large model (Leewis et al., 2019). For the NQ 320k dataset, we sampled pseudo-documents from the original 307k queries for training data and
of 100k
"NCI (Wang et al., 2022) made their processing scripts and the processed datasct publicly available

---

variants
MS MARCO 100k
NQ 320k
query
gen-doc
Hits@ 1
Hits@ 10
Hits 1
Hits@ 10
40.47
71.15
39.11
61.95
80.69
93.35
66.07
73.40
85.29
96.62
71.66
80.23
Table Ablation study M on MS MARCO 100k and NQ 320k where "query" and "gen-doc" refer to the runs where final query includes original query and generated document q. respectively.
Hits@1
Hits@2
Hits@3
Hits@5
Hits @ 10
parameter size for document generation as the num- ber documents in the collection increases. this study, we use vanilla Transformer backbone with few parameters, which might be insufficient for generating longer documents from shorter query, particularly when the collection size is large, compared with other simple generation tasks (such as question generation and text simplification). A valuable future direction would to handle low- quality pseudo-documents by improving the cur- rent settings of DiffusionRet.
65.44
73.42
78.35
83.26
87.94
Table 4: Retrieval performances (Hits @k) on MS MARCO 300k (with 200k zero-shot data), under the diffusion model trained from the MS MARCO 100k as in Table 1.
finetuned the model from SEAL's NQ checkpoint, where other parameter settings remain the same as those of (Bevilacqua et al., 2022).
Inference. Given query, DiffusionRet first ap- plies the diffusion model to generate pseudo- document and then performs constrained decod- ing of the generative retrieval model generate a set of long n- n-grams, as in Eq. (1). The FM- index was built on the union of the training and dev sets to restrict the decoding search space, as in SEAL (Bevilacqua et al., 2022). During the con- strained decoding, we prepend an original query to pscudo-document to improve performance. More detailed settings are presented in Ap- pendix
5.2 Ablation Study
In this section, we conduct ablation studies to ana- lyze the DiffusionRet strategies.
Table 3 examines the effects of removing either an original query generated document when formalizing an extended query for the generated retrieval. "Query" indicates whether an original query x is prepended and "gen-doc" whether a generated document is provided for forming a final query for the n-gram generation.
generated document qi
As Table 3 shows, the runs using a generated document (i.e., the 2nd-3rd rows) lead to signif- icant improvements compared to that with only an original query (i.e., the 1st row). When using only original query, the performances poor on both datasets. The results confirm that the setting of DiffusionRet using both the generated document and original query is effective.
5 Results and Analysis
5.1 Main Results
Results on MS MARCO 100k. Table lists the comparison results on MS MARCO 100k. Diffu- sionRet significantly outperforms all other baseline methods for sparse/dense retrieval and generative retrieval.
largely
5.3 Expansion of Diffussin Model Unseen Documents on MS MARCO 300K
Results on NQ 320k. Table 2 shows the com- parison results on the NQ 320k dataset. Again, DiffusionRet method achieves superior results on Hits @ metric over all baseline methods in pre- vious works. For Hits @ 10, DiffusionRet substan- tially outperforms most generative retrieval meth- ods, including SEAL, but underperforms compared DSI-QG (Zhuang et al., 2022). The relatively weak performance on the NQ 320k dataset com- pared the case of MS MARCO 100k possibly because the diffusion model may require a large
We further evaluate whether DiffusionRet's diffu- sion model is robust for other unseen documents. To extend the document space, we take subset of 300k pairs, called MS MARCO 300K, i.c., the number of documents is 300k. We the current diffusion model trained using 100k examples and apply it to the remaining 200k unseen queries generate 200k pscudo-documents, We use these 300k data to train the retrieval model and build

---

Model
source
Hits@1 1
Hits@ 10
Params
query 40.47
71.15
0
T5-base
gen-doc 19.53
45.63
220M
T5-base
query + gen-doc
45.79
76.05
220M
Diffusion
gen-doc
80.69
93.35
103M
Diffusion
query + gen-doc
85.29
96.62
103M
Table Comparison of retrieval performances between diffusion models with SLMs on MS MARCO 100k, where the T5-base model used for SLM; "query", "gen-doc", and "query gen-doc" indicates the SEAL's runs whose input is an orginal query, generated pseudo-document, and their concatenated sequence, respectively.
Model
Params
5.5 Comparison of Model Parameter Size
DSI-large
800M
To further demonstrate the parameter efficiency of DiffusionRet, Table 6 summarizes the model parameter sizes, comparing with those in related works. DiffusionRet uses 12-layer Transformer model (Vaswani et al.,, 2017) with 103M parameters for the diffusion model backbone and a BART-large model (Lewis et al., 2019) with 406M parameters as the generative retrieval model. As listed in Ta- ble 6, DiffusionRet has the smallest parameter size and reduces the required size of TOME (Ren et al.. 2023) consisting of a document generation model based on T5-large and URL generation model based on T5-base.
DSI-XL
3B
DSI-QG-base
800M + 336M + 220M
DSI-QG-large
800M + 336M + 800M
TOME
800M 220M
DiffusionRet
103M 406M
Table 6: Comparison of model parameter sizes between DiffusionRet, TOME, DSI and DSI-QG.
the FM-index decoding space. Table 4 shows the results, indicating that DiffusionRet, even in an enlarged document search space, yields the best re- sults in terms of Hits @ 10 and outperforms the other baseline models Hits @1 except for TOME.
6 Conclusion
In this paper, we proposed DiffusionRet, a two- stage architecture consisting of diffusion-based document generation and n-gram-based genera- tive retrieval. Extensive experiments the MS MARCO and NQ datasets showed that Diffusion- Ret achieved state-of-the teffftheartt performances terms of Hits 1, outperforming most existing retrieval models. Further analysis confirmed that the use of generated documents as a final query was the key component to bring about those significant im- provements.
5.4 Comparison of Diffusion Models with Small Language Models
To further examine whether the diffusion model improves small language models (SLM) in generat- ing pscudo-documentt, we select bS-base (Raffel et al., 2020) model as a SLM, which has roughly twice the number of parameters (220M) compared to the diffusion model, and train it for the first document generation stage on MS MARCO 100k. Table 5 shows the comparison result in retrieval performances on MS MARCO 100k between meth- ods using the diffusion and T5-base models. It is shown that the diffusion model significantly out- performs the T5-base model, confirming that dif- fusion models are able to generate high-quality pseudo-documents in carameter-efficieee manner. It should be noted that The T5-base model is not effective in generating pseudo-documente being failed improve the baseline in the setting us- ing the generated documents aglone (i.e., gen-doc), whereas the diffusion model consistently achieves marked improvements irrespective of whether an original query is appended or not.
In future work, we would like invent full- fledged diffusion model jointly generate and re- trieve without requiring a subsequent retrieval com- ponent. Furthermore, we would also jointly per- form a document generation and retrieval in a sin- gle transformer model by viewing document gen- eration as the -ffinnhf-thought (Wei et al., 2022). We would simplify refining n n-gram- baaun-based genera- tive retrieval such that the generation and retrieval components are less separated. Comparing the full-fled full-fledged diffusion-based generate-then--etrieva model with popular LLMs would be interesting and inspire future research.

---

Limitations
substrings as document identifiers. In arXiv pre-pi pre-print 2204.10628.
Although the diffusion model has the advantage of few parameters, it performs thousands of dif- fusion steps to sample the data during inference, slowing the inference speed. Although the adopted work (Yuan al., 2022) considerably improved the inference speed, it is still non-trivial for us to perform extensive sampling of the original training set to construct the training data for our retrieval model. Incorporating some studies including the consistency model (Song et al., 2023) based on ordinary differential equations (ODEs) which can significantly accelerate the inference would be valuable future work.
Guanqun Bi, Lei Shen, Yaman Cao, Meng Chen, Yuqiang Xie, Zheng Lin, and Xiaodong Hc. 2023, DiffusEmp: A diffusion model-hased framework with multi-grained control for empathetic response generation. In Proceedings of the 61st Annual Meet- ing of the Association for Compulational Linguistics (Volume 1: Long Papers). pages 2812-2831, Toronto, Canada. Association for Computational Linguistics.
Michael Burrows. 1994. A block-sorting lossless data compression algorithm. SRC Research Report, 124.
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2023. A unified generative retriever for knowledge- intensive language tasks via prompt learning. arXiv preprimt arXiv:2304. 14856.
Another limitation is the two-stage style, per- forming generation and retrieval in a cascaded man- ner. While generative retrieval provides an elegant framework that integrates indexing and retrieval into the transformer's parameters, DiffusionRet uses two different types of models for generation and retrieval. Thus, arguably desirable approach is to unify the two stages single diffusion model or unified transformer without losing the princi- pled and elegant manner of generative retrieval.
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2022a. Corpusbrain: Pre-train a generative retrieval model for knowledge- intensive language tasks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, page 191-200, New York, NY, USA. Association for Computing Machinery.
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. 2022b. Analog bits: Generating discrete data us- ing diffusion models with self-conditioning. ArXiv, abs/2208.042 042002.
In addition, our current experiments show the major performances without analyzing the parts with more errors, helping us to move a step towards promising future directions. As we establish two stage components, it is possible to evaluate the generation and retrieval modules separately, which is worthy of future research. In particular, we need to explore the relationship between the parameter sizes of diffusion model and collection size. This is helpful in determining the situation required to increase the parameter size of the diffusion model.
P. Ferragina and G. Manzini. 2000. Opportunistic data structures with applications. In Proceedings s1st Annual Symposium on Foundations of Computer Sci- ence, pages 390-398.
Luyu Gao, Xueguang Ma, Jimmy J. Lin, and Jamie Callan. 2022a. Tevatron: An efficient and flexible toolkit for dense retrieval. ArXiv, 20.2220000565....
Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. 2022b. Difformer: Empowering diffusion model embedding space for text generation. xiviv preprint ar Xix: 129042..
Acknowledgements
This work was partly supported by the Institute of Information & communications Technology Plan- ning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2002-001882288 De- velopment of Large Korean Language Model Tech- nology for Efficient Pre-training). Shanbao Qiao and Xuebing Liu were also supported by China Scholarship Council (CSC).
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2023. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representa- tions, ICLR.
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuan- jing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029,
References
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Ad- vances Neural Information Processing Systems. volume 33, pages 6840-6851.
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen tau Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive search engines: Generating

---

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. JEEE Transactions on Big Data, 7(3):535-547.
Jianmo Ni. Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. editors. 2022. Sentence-TS: Scaling up Sentence En- coder from Pre-trained Text-to- Text Transfer Trans- former.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov. Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-67 1781, Online. Association for Computational Linguistics.
Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019. From doc2query to doetttitqqery. Online preprint, 6.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 21(140):1-67.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, et al. 2019, Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Ruiyang Ren, Wayne Xin Zhao, Jing Liu, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023. Tome: A two-stage approach for model-based retrieval. arXiv preprint arxiv:2305.1116..
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdeirahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising squence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910 1346l.
Julian Risch, Samuele Garda, and Ralf Krestel. 2020. Hierarchical document classification
sequence generation task. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL '20, page 147-155, New York, NY, USA. Association for Computing Machinery.
Junyi Li, Wayne Xin Zhao, Jianyun Nie, and Ji rong Wen. 2023a. Glyphdiffusion: Text generation as image generation.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. 2022. Palette: Image-to-image diffusion models. SIGGRAPH '22, New York, NY, USA. Association for Computing Machinery.
Wenbo Li, Xin Yu, Kun Zhou, Yibing Song, Zhe Lin, and Jiaya Jia. 2022a. Sdm: Spatial diffusion model for large hole image inpainting. arXiv preprint arXiv:2212.02963.
Jascha Sohl-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. In International Conference on Machine Learning, pages 2266-2265. PMLR.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022b. Diffusson- Im improves controllable text generation. ArXiv, abs/2205. 14217.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency models.
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wen- jie Li. 2023b. Multiview identifiers enhanced gen- erative retrieval. In Proceedings of the ó1 Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6636-6648, Toronto, Canada. Association for Computational Lin- guistics.
Yang Song and Stefano Ermon. 2019. Generative mod- eling by estimating gradients of the data distribution. In Advances in Neural Information Processing Sys- tems, volume 32.
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. 2022. Self-conditioned embed- ding diffusion for text generation.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. 2022. Repaint: Inpainting using denoising diffusion prob- abilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 1461-11471.
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjic Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023. Learning to tokenize for generative retrieval. arXiv preprint 2rX4v:2304.0411
Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. 2021. Rethinking search: making domain experts out of dilettantes. In ACM SIGIR Forum, volume 55, pages 1-27. ACM New York, NY, USA.
Yubao Tang, Ruqing Zhang, Jiafeng Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2023a. Semantic-enhanced differen- tiable search index inspired by learning strategies. arXiv preprint 2305::3055.11155
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. choice, 2640:660.

---

Zecheng Tang, Pinzheng Wang. Keyan Zhou, Juntao Li, Ziqiang Cao, and Min Zhang. 2023b. Can diffu- sion model achieve better performance in text gener- ation? bridging the gap between training and infer- ence! arXiv preprint (rXivv2305.04400
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. 2023. Dinoiser: Diffused condi- tional sequence learning by manipulating noises.
Hongyi Yuan, Zheng Yuan. Chuanqi Tan, Fei Huang. and Songfang Huang. 2022. Segdiffuseq: Text dif- fusion with encoder-decoder transformers. ArXiv, abs/2212.1032 10325.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao. Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer Memory a Differentiable Search Index. arXiv e-prints, page wrXiv.2202.06991.
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Yu Wu, and Ji rong Wen. 2022. Dynamicrctriever: A pre- training model-based ir system with neither sparse nor dense index. ArXiv, abs/2203.00537.
Harsh Trivedi, Niramjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions. In Proceedings of the 61 Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pages 10014-10037, Toronto, Canada. Association for Computational Linguistics.
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei. Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the gap between indexing and retrieval for differentiable search index with query generation. arXiv preprint arXiv:2206. 10128.
A Examples of Generated Documents
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information proormmaiinprcccsiinn systems, 30.
We show some examples of diffusion models generated documents by using different color fonts to mark different quality of the generated content. Green: correct and relevant content.
Yellow: relevant content in similar domain
Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678.
imprecise knowledge.
Red: irrelevant content or incorrect knowledge.
Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuging Xia, Chengmin Chi, Guoshuai Zhao, et al. 2022. A neural corpus indexer for document retrieval. arXiv preprint 201.:22206007743
For seen data (in training set):
Query: who plays mantis in guardians of the galaxy
Ground truth document:
pom klementieff pom klementieff ( bborn 3 may 1986 ) french actress. she trained cours florent drama school paris appeared films loup 2009 sleepless night 201 hacker's game ( 2015 ) she plays role mantis film... Generated document:
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In International Conference on Learning Representations.
pom klementieff pom klementieff ( boon c. 1986 / 1987 french actress. she trained coeg florent drama school paris appeared films loup 2009 mouthless night ( (011)) book 's game ( (015 she plays role man...
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. 2022. Dif- fusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796
For unseen data (in dev/eval set): Query: what are chia seeds benefits Ground truth document: health benefits of chia seeds. chia seeds impart many health benefits, some of which include controlling diabetes, lowering cholesterol levels, improving the digestive system, boosting heart health, controlling hypertension, and they even contain anti inflammatory properties. Generated document:
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserimi: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, SIGIR '17, page 1253-1256, New York, NY, USA. Association for Computing Machinery.

---

chia seeds are good source of protein. the seeds are rich in dietary plants, wheat, fiber, and dental fiber. fruits so eat and the digest digestiveive system, potassium, fruits, and vegetables products. all chia are rich in fruit and developing and have and to health and properties.
of the continent of Eastern Europe. An independent, independent nation or state in the Eurasia. The Union
B Denoising Process of Diffusion- Document Generation
There are also examples of failed generation, usually for long-tail data where the query is not very relevant the document, such the following example. And there would be future work to handle these luw-quality samples. Query: aarra definition government Ground truth document:
To describe A and B, in Eq. (3), we follow ad- ditional notation of ā of (Yuan et al., 2022), as follows:
á Os
8=1
81 0i
(8)
american recovery reinvestment act 2009 the american recovery reinvestment act 2009 arra ) pub. 111 - nicknamed recovery act stimulus package enacted 111th congress signed law president barack obama february 2009, developed response great recession arra's 's primary objective
A and B, are defined as follows:
Ag v 1
1 Ot
/a att. 1)
B.
(9)
āt
Generated document:
According to Eq. (3). is sampled by mixing the formed predicted sequence yú = ED (y x,,,,,). and ys, and random noise.
federal government - the government federal government government community government 1. M.L.ittiillliii 1.
The self-conditioning (Chen et al., 2022b) ad- ditionally takes the formed predicted sequence yó to predict the current sequence y0 using the ex- tended denoising function ED! (y,, x, x,t) The encoder part takes x, while the decoder part takes the concatenated sequence of y: and Yi with the dimensionality of X 2d.
We also provide some document samples generated by T5-base as comparison.
Query: visable light from the sun Ground truth document:
The incoming energy from the Sun to Earth is mainly visible sunlight, called the visible portion of the spectrum of electromagnetic radiation. We perceive visible sunlight as colors from violet (short-wave radiation) to red (long-wave radiation)
C Training objective of Diffusion model
We optimized the denoising parameter in ED (-) and the embedding parameters by minimizing the variational lower bound of data log-likelihood. Fol- lowing the derivation of (Yuan et al., 2022), when pair of query and document d "X is given, the training objective simplified to:
Generated document:
The visible sun's 's light the most visible light from the Earth, and that why it is the cheapest light
source. a
T
ED/ (y,,x,t) t) yo 1|2
Query: define eurasia
E Bgyy..xd)
Σ
E Eg(r
Ground truth document:
|
Define Eurasia Eurasia. Eurasia is of major geographic and geologic significance. It is also important to the history of human civilization. Answer and Explanation: Eurasia is a landmass that consists of the continents of Asia and Europe. It measures over 21 million square miles...
+ ED (y1, Emd(d)
+ aTTv | log log
D Constrained Decoding using FM-Index
Generated document:
To meet the requirements that all the generated se- quences appear in the corpus and that the location
European Union (Europe): An independent region

---

of the sequences can be identified for retrieval, the FM-index (Ferragina and Manzini, 2000) is suit- able efficient data structure that can obtain all the possible successor tokens for constrained decoding a desired time (O[[V] log [V]).. Where V the vocabulary of the model, hence the computational time complexity is independent of the corpus size. In addition, because the FM-index is self-index in the form of compressed suffix array, it does not require a large storage space and usually smaller than the corpus data under the vocabulary size of regular language models.
retrieved documents following SEAL's (Bevilaequs al., 2022) strategy.
E Datasets Information
Microsoft machine reading comprehension (MS MARCO) (Nguyen et al., 2016) collection of datasets focused on deep learning in search and contains queries of real Bing searches and corre- sponding document answers. We used the MS MARCO passage corpus provided by Tevatron (Gao et al., 2022a) containing approximately 500k query-document pairs for training. Following pre- vious work that used different scales of data (Tay et al., 2022; Zhuang al., 2022; Ren et al., 2023), downsampled a 100k subset for training and used the full dev set for evaluation containing 6980 query samples.
The FM-index is built based on the Burrows- Wheeler Transform (BWT, Burrows, 1994), where all the rotations of the string are listed in a matrix and sorted lexicographically, Given an example string "APPLE," where $ denotes the end token, the corresponding matrix is shown in Figure 3. The L column is the BWT of the string, and the FM- index only stores the F and L columns to achieve an efficient localization of an arbitrary string by performing a reverse lookup of the token's 's prefix in F the same location in L., and then iteratively extending any n-gram in the corpus.
Natural Questions (NQ) comprises a collection of natural language questions and Wikipedia page documents. The training set contains 307k query- document pairs, and the dev set has 7830 queries, called the NQ 320k in DSI (Tay et al., 2022). We used the full training and dev sets in the experi- ments.
F
L
Rouge-Score of Generated Documents
$
A
P
P
L
E
We calculated the Rouge- score between the gener- ated document and ground truth document MS MARCO, and the data in the training are seen the model, while those the dev set are unseen for the model. Table 7 shows the results.
L
E
E
A
P
P
L
L
$
A
P
P
Set Rouge-1 Rouge-2
Rouge-L
P
E
$
A
train 0.696
0.537
0.677
P
$
A
P
dev
0.322
0.123
0.257
Table 7: Rouge-score of Generated Documents.
Figure 3: BWT Matrix
Despite the low Rouge-score on dev set, the gen- erated documents still offer rich keywords and valu- able semantic information for retrieval.
When applied the autoregressive model, the FM-index used constrain the decoding space within a possible next token set, and the logit of all unconsidered tokens (out the set) is masked to oo. When iteratively generating the target sequence, the next new token is selected from the ranks in L in the same row as the previous token in column F. Thus, the beam search can return set of grams appearing in the corpus and the location of the n-grams can be easily obtained using the FM- index row number. After decoding, we also used the FM scoring function to better select n- r-grams and intersective scoring functions to better rank the
G Detailed Hits@k
Figure 4-5 presents detailed Hits results, with varying k, in comparison to the best performing baselines on MS MARCO 100k and NQ 320k, re- spectively.
H Detailed Settings
Settings for the diffusion model used in docu- ment generation: For training examples in the

---

100
Total training steps: 500k 1M for MS MARCO 100k / NQ 320k dataset
95.29
91.33
RM-1
96.62
23,07
DPR Hits@
Settings for the generative retrieval model:
The generative retricval model was based on SEAL's work (Bevilacqua et al., 2022), and the model was fine-tuned from the BART-large model (Lewis et al., 2019) on 4 Nvidia RTX 8000 GPUs. The training performed using Fairseq Toolkit. The detailed hyperparameter settings are follow:
Hits@!
05
60
11023 111222
1nt223 H8024 FRugs Huug Higgh riag litagg Hitag300
Figure 4: Hits@k on MS MARCO 100k, varying the number of retrieved documents (k). The results compared with TOME (Ren et al., 2023) for Hits and DPR (Karpukhin al., 2020) for Hits@ metric.
Optimizer: Adam
Learning rate: 3e-5
Base model: BART-large
DSI-OG 20
50
TEAT 75.22
Hits@110
Tokens per GPU: 4096
77.25
75.31
75
Warmup steps: 500
73.05
70
GenRet H
Hyperparameter ry: 2.0
Hyperparameter k: 0.8
80
18teg3 [1tag2
Bl Hhligs Ettgge Itteg7 Hayg
Higg 1Rtag30
Total training epochs: 96/ 18 for MS MARCO 100k / NQ 320k dataset
Figure 5: Hits@k NQ 320k varying the number of retrieved documents (k). The results are compared with GenRet Sun et al., 2023 for Hits@ and DSI-QG, Zhuang et al., 2022 for Hits @ 10.
Specifically, for MS MARCO, we ran the train- ing for 96 epochs with learning rate of 3e-5 because the loss could not be further optimized. For the NQ 320k dataset, NQ check theck-point during 18 epochs when the loss reached stability, while other parameter settings remained the same as (Bevilacqua et al., 2022). We did not use the uniformly sampled document spans as "unsupervised" samples to expose evidence dur- ing training, different from SEAL (Bevilacqua et al., 2022), since our generated documents already have semantically-rich contextual information.
fine-tuned SEAL's
MS MARCO and NQ datasets, used pairs of query- docymett where all queries were automati- cally generated for each document.
The maximum length of the input query- document pair was capped at 128 because the vast majority sequence lengths fell within this range, and the maximum length of the output was set to dif-
64. We implemented the fusion model using PyTorch, and trained the model on 4 Nvidia TITAN-RTX GPUs. The optimizer used was AdamW. The detailed hyperparameter settings are as follows:
Optimizer: AdamW
Learning rate: le-4
Maximum input length: 128
Maximum output length: 64
Diffusion steps: 2000
Batch size: 192
Warmup steps: 10000


Reviews:
[{"review_id": "nWXo8UaQpx", "reviewer": "Reviewer_B9MB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer B9MB\n\nThanks for taking you time to review this manuscript. We appreciate that the reviewer commenting on the effectiveness of our work. Thank you again for your comments and the issues raised. However, we also believe that there are misunderstandings about some certain aspects of our work, in this discussion we will further explain all your concerns and correct some misconceptions. \n\n### 1. Response to Reason to Reject 1: Advantages of intermediate reasoning step and Explanation of the usage of term hierarchical classification\n\nThank you for raising insightful issues.\n\n**For the intermediate reasoning step issue.**\nExplicitly putting _“intermediate reasoning step”_ is mainly motivated by the success of the chain-of-thought (**CoT**) on various NLP tasks and the CoT-based retrieval and QA application [1,2]. Particularly for our addressed retrieval task, the use of intermediate reasoning step is directly motivated by the recently published two-stage generative retrieval [3].\n\nAs in [3], the generative retrieval can be equipped well with the intermediate reasoning step, possibly even in an end-to-end manner. Overall, we strongly believe that the generative retrieval itself has much advantages and potential to be extended. But, our criticism made in this statement is mainly targeted the early works of the generative retrieval that maps directly a query to docids, which does not explicitly put the intermediate reasoning step (or intermediate content), not all generative retrieval methods, particularly not the up-coming generative retrieval methods that will be extended and improved. Here, we want to mention that when the generative retrieval does not use the intermediate reasoning step, its performance may be limited, given the existing successful results using CoT and other similar works, including [3]. \n\nPlease understand our result provides some evidence that the intermediate reasoning step is helpful in improving the performance, with the stronger improvements than [3]. We believe that our result is positively affects the extension of generative retrieval. While the intermediate reasoning step is currently based on the diffusion model in our current paper, we are also really interested in extending the generative retrieval from the currently observed evidence. \n\nBut, we agree with your concern that we should be more careful in making this the criticism, not attacking all generative retrievals. To clarify that the limitation only belongs to the specific type of generative retrieval (but currently widely used), we would revise the statement and paragraph in a very careful way without making miss-understanding our intention, in any extended version. Also, in any further chance, we would like to add more references for motivating our work, including [2] which was missed in the current version of the paper. \n\n**For the use of the term hierarchical classification issue.**\nAs in the response above, our statement here concerns mainly a specific type of the generative retrieval, which is the early work of DSI (but still widely used as the basic framework to our knowledge), not all possible generative retrieval methods. Under such a specific type of generative retrieval, we view it as the hierarchical classification, largely based on [4], which interprets the hierarchical document classification as a sequence-to-sequence task; In [5], a hierarchical clustering process is deployed to assign semantically structured identifiers.\n\nIn fact, our intention here is not to reveal the _“hierarchical classification”_ itself but to strongly emphasize that the intermediate reasoning step is not involved in (some) generative retrieval methods, in a way of that it directly goes into the classification process, not “generating” the intermediate contents. \nAgain, we also aware your concern that the statement is also needs to be carefully provided. To clarify our intended meaning, If given the chance, we would like to rewrite the statement in a much proper way (such as removing the problematic term _“… mapping a query directly to docids or index terms, …”_, or clearly mention that only specific type of generative retrieval can be seen as the hierarchical classification with the reference [4]).\n\n### 2. Response to Reason to Reject 2: Evidence for addressing pretrain-finetune discrepancy.\n\nFirst of all, we are really sorry that DSI-QG [6], the reference in the paragraph when mentioning the term “pretraining-finetuning discrepancy” in Section 1, is wrongly cited; its correct one is TOME [3]. In other words, we have the citation-incorrect sentence in the 2nd paragraph in Section 1: \n\n> _As noted in (Zhuang et al., 2022), however these docid-like symbols are only seen during the finetuning but unseen at the pretraining stage, thus causing the pretrain-finetune discrepancy, which ..._\n\nwhich should be revised to the sentence below:\n\n> _As noted in (Ren et al., 2023), however these docid-like symbols are only seen during the finetuning but unseen at the pretraining stage, thus causing the pretrain-finetune discrepancy, which …_\n\nWe far believe that the wrong citation makes the confusion in understanding the intended meaning of “pretraining-finetuning discrepancy”. We are deeply sorry about this error again. \n\nIn fact, the issue of the pretraining-finetuning discrepancy of the generative retrieval is NOT newly addressed in this paper, instead being raised in [3]. To clarity this, let us allow to copy the corresponding snippet of paragraph of TOME [3] which mentions the pretraining-finetuning discrepancy:\n\n>_…\nSuch docids are not adequately captured in the pretraining stage of the generative PLM, thus limiting PLM’s capabilities for generative prediction (e.g., unseen docids during pre-training). This creates a discrepancy between the pre-training and fine-tuning phases\n…_\n\nThus, our statement of the _“pretraining-finetuning discrepancy”_ is a re-expression of the above cited part [3], thus having the exactly same meaning as that of [3]. \n\nOn the other hand, the terminology “pretraining-finetuning discrepancy” has also been introduced in XLNet [7], when criticizing BERTs, meaning that [MASK] token is ONLY seen during the pretraining but is NOT seen in downstream tasks during finetuning. This pretraining-finetuning discrepancy of BERT is quite similar to that of the generative retrieval, as two cases are all caused by introducing artificially-designed tokens (i.e., [MASK] and docids). \n\nBecause generative retrieval relies on the use of _“pretrained”_ transformers (i.e., encoder-decoder models), we expect that the effect of using the transfer learning from the pretrained model to the finetuned one in the downstream retrieval task will be substantially enlarged and improved when the pretraining-finetuning discrepancy (currently caused by docids) is somehow reduced or relaxed. \n\nThis motivates us to make the second stage of the retrieval based on the N-gram generative retrieval, without using docid-like symbols, although we adopt the existing SEAL model [8]. \n\nWhile we don’t clearly provide the comparison results with details, the potential advantage of reducing the pretraining-finetuning discrepancy from using n-grams is the training efficiency of the retrieval model; in our preliminary experiments, the n-gram generative retrieval is much quickly trained only using single training example per each document in the collection, comparing to DSI-QG [6]. Given our limited time, we couldn’t prepare the comparison results on that. If given change, we would definitely provide the results of the comparing training costs between our N-gram retrieval model and DSI-QG. \n\nWe also want to clarify that the discrepancy addressed by DSI-QG [6] and NCI [9] using the query generation is NOT the aforementioned pretraining-finetuning discrepancy, but the data distribution mismatch [6]; the long text of a document is trained during the indexing time, but a short query text is provided during the inference time. In fact, this mismatch issue is somehow handled by putting “intermediate reasoning step” in our DiffusionRet, as we generate a long pseudo document, whose length is similar to that observed during indexing time. \nComparison with those approaches using query generation is also interesting issue. As reported in Table 1-2, our DiffusionRet shows better performances than DSI-QG [6] and NCI [9] . Of course, the experiment results only provide positive evidence that the putting _“intermediate reasoning step”_ is helpful in improving the performance, not logically determining a better one between the query generation and the intermediate reasoning step. \nWe believe that the generative retrieval equipped with query generation is also advantageous, as it provides the end-to-end retrieval mechanism. Integrating these two directions for the query generation and the intermediate reasoning step would be interesting to be explored. \n\nTo clearly resolve your concern, if given a chance, we would like to revise our paper in any extended version, as follows:\n\n1. Correcting the wrongly-cited reference to TOME [3]\n2. Clarifying that the pretraining-finetuning discrepancy is introduced in [3] (as reference, the terminology is used in [7] similarly)\n3. Clarifying that the pretraining-finetuning discrepancy is different from the distribution mismatch issue addressed by the generative retrievals trained with query generation like DSI-QG [6] and NCI [9] \n4. Stating that generative retrievals trained with the query generation are on the valuable direction, so being complementary with the direction of putting the intermediate reasoning step. \n5. Presenting the comparison of training costs between the N-gram generative retrievals and DSI-QS [6] (or NCI [9]), to show that reducing pretraining-finetuning discrepancy is helpful to improve the training efficiency.\n\n### 3. Response to Reason to Reject 3: Explanations for Experiment 5.3.\n\nWe are sorry for making this confusion. Here, we want to mainly check the robustness the diffusion model to other unseen 200k queries, rather than the retrieval model itself. In other words, the diffusion model is trained only using 100k examples, but being expanded to other 200k unseen queries, thereby making the enlarged search space, (i.e., the search space is the sets of 100k and 300k documents in Table 1 and Table 4, respectively.) That is why the section title is “expansion of diffusion model to unseen documents on MS MARCO 300k”, thereby focusing on the diffusion model. \n    \nBut, as you worried, the retrieval models are not fully unseen; our retrieval model has the subtle difference from the MS MARCO 100k setting; While the retrieval model is trained using 300k, 200k pseudo documents may be different from 100k ones in quality, because the diffusion model is not trained on those 200k queries. \n\nGenerally, we believe that your raised issue and experiment setup is also important and valuable. In given a chance, we would like to provide the retrieval performance of the fully unseen setting in MS MARCO 300k in any extended version, where n-gram generative retrieval only trained over 100k documents is extensively applied over 300k documents. \n\nWe deeply appreciate your efforts in reviewing our paper. Hope this discussion could address all your concerns. Wish you all the best.\n\n    [1] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in Neural Information Processing Systems 35 (2022): 24824-24837.\n    [2] Trivedi, Harsh, et al. \"Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.\n    [3] Ren, Ruiyang, et al. \"TOME: A Two-stage Approach for Model-based Retrieval.\". Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.\n    [4] Risch, Julian, Samuele Garda, and Ralf Krestel. \"Hierarchical document classification as a sequence generation task.\" Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, 2020.\n    [5] Tay, Yi, et al. \"Transformer memory as a differentiable search index.\" Advances in Neural Information Processing Systems 35 (2022): 21831-21843.\n    [6] Zhuang, Shengyao, et al. \"Bridging the gap between indexing and retrieval for differentiable search index with query generation.\" arXiv preprint arXiv:2206.10128 (2022).\n    [7] Yang, Zhilin, et al. \"Xlnet: Generalized autoregressive pretraining for language understanding.\" Advances in neural information processing systems 32 (2019). \n    [8] Bevilacqua, Michele, et al. \"Autoregressive search engines: Generating substrings as document identifiers.\" Advances in Neural Information Processing Systems 35 (2022): 31668-31683.\n    [9] Wang, Yujing, et al. \"A neural corpus indexer for document retrieval.\" Advances in Neural Information Processing Systems 35 (2022): 25600-25614.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors integrate the diffusion model into the generative retrieval. \n2. The authors conduct extensive experiments to demonstrate its effectiveness.", "reasons_to_reject": "1. The motivation is not reasonable. As claimed in abstract, i.e., \"Generative retrieval suffers from the lack of the intermediate reasoning step, caused by the manner of merely using a query to perform the hierarchical classification. \", I have the following questions: \n   - The lack of intermediate reasoning step is not necessarily a weakness, as it implies an end-to-end retrieval, which on the contrary should be an advantage. \n   - Additionally, generative retrieval is not a hierarchical classification but a query2docid generation task, where docid can be atomic or ngrams like SEAL and this paper. \n   - The authors should give some evidences to support this claim. \n2. The model design to address the pretrain-finetune discrepancy lacks justification. Existing studies such as DSI and NCI have adopted query generation to address this discrepancy, while this paper applies document generation instead. The reason, necessity, and advantage of doing so compared to query generation are not provided in the paper.  \n3. Experiment 5.3 is essentially evaluating the ability of the adopted diffusion model (i.e., SeqDiffuSeq) to handle unseen documents. But for retrieval task, these documents are still visible and used for training. As such, this experiment cannot measure the retrieval performance of the proposed model for unseen documents.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors propose to leverage a diffusion-based two-stage generative retriever. It first leverages a diffusion model to generate pseudo documents based on input queries to address the gap between indexing and retrieving tasks. Then it performs a doc-ngrams retrieval with the generated documents. Experiments on two public datasets verify the effectiveness of the proposed model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T3gLGQ9FQi", "reviewer": "Reviewer_PVJR", "summary": "", "questions": "Q1: How to compare the generation quality and inference speed of a diffusion model of the same size and T5? Lack of some important ablation studies.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer PVJR\n\nThank you for your comments and questions! We appreciate that the reviewer mentioned the our method effective and leads to the state-of-the-art performances. In this discussion we hope that we can further address all your concerns as well as answer the questions. \n\n### 1. Response to Reason to Reject 1: Motivations for using diffusion model and Comparison of using other model(T5) of similar parameter size.\n\nAs replied to the reviewer B9MB, we would like to put the _“intermediate reasoning step”_ by generative a hypothetical document, mainly motivated by the success of the chain-of-thought (CoT) [1] and its relevant IR applications [2-3]. \nThe motivation for using diffusion models to generate such a pseudo document Is that we regard this process as “semantic inpainting”, similar to the image inpainting, where other remaining document contents need to be infilled given a query content. In such inpainting task, the diffusion model has shown largely successful results and performances [4-8]. In addition, diffusion models are increasingly applied to text generation process recently, showing improved performances [9-11]. \n\nOf course, we could apply the generative models such as LLM such as GPT3,  LLaMA, and PaLM to perform such the semantic inpainting task. But, given the recently emergent interests in diffusion models, it would be quite valuable and interesting to apply the diffusion model and explore by checking whether the currently available model works well on this addressed semantic inpainting task. \nIn addition to this general motivation, there are some technical motivations in using diffusion models as follows:\n\n+ The diffusion model is parameter-efficient; in the work of Diffusion-LM[9], DiffuSeq[10], SeqDiffuSeq[11], the small-scale diffusion models outperform some GPT variants on different generative tasks.\n\n+ Our semantic inpainting task does not necessarily produce grammatically perfect sentences, which making that the use of LLMs is not mandatory: \nWhile LLMs can also perform generative tasks and even generate sentences that are of higher quality in terms of syntax and grammar, these capabilities may not be mandatory for our addressed semantic inpainting task. In our task, pseudo documents are not necessarily grammatically perfect but required to provide semantically enriched results, which are helpful to the next stage of the retrieval; In fact, our example pseudo documents generated by diffusion models are somehow grammatically flawed as shown in Appendix A. \n\nOf course, we are also quite interested in applying LLMs to the semantic inpainting task. In the future, we will compare the results of using LLMs and diffusion models in generating pseudo-documents, in any extended version of the paper. \n\nOn the other hand, given your additional question _\"How would DiffusionRet work with a generative language model (like T5) of the same parameter size?\"_, we trained a T5-base (220M, twice the parameters of the diffusion model) which are  small language models (SLM), for generating pseudo documents on MS MARCO 100k dataset and show the retrieval results. \n\nAs shown in the Table below, the results confirm that the use of SLM does not show good performance and diffusion models are quite parameter-efficient. Unlike SLM, we believe that the use of LLMs will gain much improvements with its strong generation quality, and thus the comparison of LLMs with diffusion models is interesting and valuable as mentioned above. \n\n| Gen-Model | source | Hits@1 | Hits@10 | Params | \n| ---- | ---- | :----: | :----: | :----: |\n| - | query | 40.47 | 71.15 | 0 |\n| T5-base | gen-doc | 19.53 | 45.63 | 220M |\n| T5-base | query+gen-doc | 45.79 | 76.05 | 220M |\n| Diffusion | gen-doc | 80.69 | 93.35 | 103M |\n| Diffusion | query+gen-doc | 85.29 | 96.62 | 103M |\n\nIf given a chance, we would like to strengthen our motivation in diffusion models based to the above responses, add detailed comparison results between SLM and diffusion models, and make further statement that the comparison with LLMs is interesting and valuable to invoke the future work.\n\n### 2. Response to Reason to Reject 2: Discussion of the difference between DiffusionRet's N-gram-based Generative Retrieval and SEAL.\n\nWe are sorry that our current presentation does not clearly reveal the difference between n-gram generative retrieval and SEAL [12]. As you mentioned, the length of n-gram is one of differences, but there are other distinguishing parts  from SEAL [12], while some of them are presented in our papers. \n\n+ Types of a source sequence is different between SEAL and our method; while SEAL uses short queries, our n-gram generative retrieval deploys _“pseudo documents”_ as for source sequence. The results of SEAL and our method are compared in Table 2, which provide evidence that the use of pseudo documents as source part is important in improving the retrieval performance. documents made a major contribution to the performance improvement. In fact, this training difference is mentioned in **Section 4.4-Training Retrieval Model**.\n\n+ SEAL extensively uses the query generation as data augmentation by uniformly sampling spans from the same documents, where the sampled spans are other queries of the same document. In contrast, our N-gram generative retrieval has no such data augmentation based on query generation, as our method only uses a single instance of pseudo document, without more examples per document. This difference is presented in Appendix H:  \n> _It is noted that SEAL uses uniformly sampled spans from documents in the collection as “unsupervised” samples to train, aiming to expose the model to more possible pieces of evidence. In our generation setting, our generated documents contain semantically-rich contextual information, we do not need such additional data._\n\nTo clarify this difference between SEAL and our N-gram generative retrieval, if given a chance, we would like to add the key difference clearly in the main text, and revise Appendix H such that data augmentation is not used in our method. \n\nTo further clarify the comparison with original version of SEAL, we would like to add further experiments by directly evaluating SEAL using short queries as source part, both MS MARCO 100k and NQ 320k, varying the length of n-grams, in any extended version. \n\n### 3. Response to Reason to Reject 3: Formatting issue\n\nThank you for pointing out the formatting issue. We use latex to control the format and have checked the tex file to ensure that there are no redundant blank lines.\n\n### 4. Response to Q1: Discussion about generation quality and inference speed of diffusion model.\n\nThank you for your insightful question. It would be valuable and interesting to make the detailed comparison between the diffusion model and SLM (i.e., T5) in terms of not only the retrieval performance but also other generation quality and inference speed. If given a chance, we would definitely add more ablation studies comparing between the diffusion model and T5, both in terms of the inference speed and generation quality, as you raised. Here, let us mention the responses considerable on the raised issues. \n\n+ Inference speed: Because diffusion models require thousands of diffusion steps to sample the data during the inference process, the number of sampling steps or the sampling algorithm has a decisive influence on the inference speed rather than the model size. As emphasized in **Limitations**, the inference time of diffusion models is relatively slow, and enhancing their inference speed has gained significant research attention. Some specialized works have achieved significant success in this field (e.g., Consistency Model[13], LatentOps[14]), making it a good direction for future work. The comparison in inference time between T5 and diffusion model would be more interesting, when those recently-advanced diffusion models [13-14] are incorporated, which will be worth to be explored in any extended work of this paper. \n\n+ Generation quality: While there are various metrics for evaluating the quality of model generated text, we need to concern the specific demand of our semantic inpainting task (i.e., generating pseudo documents). First, we do not need to generate sentences that are of very high quality in syntax or grammar, since our goal is to inject rich semantic information into the input of the retrieval model. Second, in a two-stage model design, the second-stage retrieval model has tolerance for the generation errors of the first stage (TOME[3] also has statements and experimental support for this theory, see the second contribution point in Introduction and Section 5.3.2). \n\nTherefore, our goal in the first stage is to improve the retrieval performance, the evaluation on the generated samples needs to be designed to carefully capture its impact on the second stage. Designing such evaluation metric would be quite important and interesting topic to be explored in the future work.\nIn the meanwhile, for reference, we have used he ROUGE score of the generated results of diffusion model in **Appendix F** comparing to the ground-truth one. Here, for convenience, let us copy the table results from Appendix F as follows.\n\n\n|Set|Rouge-1|Rouge-2|Rouge-L|\n|----|:----:|:----:|:----:|\n|train|0.696|0.537|0.677|\n|dev|0.322|0.123|0.257|\n\n\n    [1] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in Neural Information Processing Systems 35 (2022): 24824-24837.\n    [2] Trivedi, Harsh, et al. \"Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.\n    [3] Ren, Ruiyang, et al. \"TOME: A Two-stage Approach for Model-based Retrieval.\". Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.\n    [4] Saharia, Chitwan, et al. \"Palette: Image-to-image diffusion models.\" ACM SIGGRAPH 2022 Conference Proceedings. 2022.\n    [5] Lugmayr, Andreas, et al. \"Repaint: Inpainting using denoising diffusion probabilistic models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n    [6] Dhariwal, Prafulla, and Alexander Nichol. \"Diffusion models beat gans on image synthesis.\" Advances in neural information processing systems 34 (2021): 8780-8794.\n    [7] Li, Wenbo, et al. \"Sdm: Spatial diffusion model for large hole image inpainting.\" arXiv preprint arXiv:2212.02963 (2022).\n    [8] Parida, Sibam, et al. \"Survey on Diverse Image Inpainting using Diffusion Models.\" 2023 2nd International Conference on Paradigm Shifts in Communications Embedded Systems, Machine Learning and Signal Processing (PCEMS). IEEE, 2023.\n    [9] Li, Xiang, et al. \"Diffusion-lm improves controllable text generation.\" Advances in Neural Information Processing Systems, 2022.\n    [10] Gong, Shansan, et al. \"Diffuseq: Sequence to sequence text generation with diffusion models.\" arXiv preprint arXiv:2210.08933 (2022).\n    [11] Yuan, Hongyi, et al. \"Seqdiffuseq: Text diffusion with encoder-decoder transformers.\" arXiv preprint arXiv:2212.10325 (2022).\n    [12] Bevilacqua, Michele, et al. \"Autoregressive search engines: Generating substrings as document identifiers.\" Advances in Neural Information Processing Systems 35 (2022): 31668-31683.\n    [13] Song, Yang, et al. \"Consistency Models.\" arXiv preprint arXiv:2303.01469 (2023).\n    [14] Liu, Guangyi, et al. \"Composable text controls in latent space with odes.\" arXiv preprint arXiv:2208.00638 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper employs the diffusion model for performing query expansion to generate the hypothetical document as the input of the generative retrieval.\n\n2. The experimental results show that the proposed DiffusionRet performs better on two datasets than previous work. And the ablation study shows the effectiveness of using generated documents.\n", "reasons_to_reject": "1. The idea of using the diffusion model to generate the hypothetical document lacks motivation. How would DiffusionRet work with a generative language model (like T5) of the same parameter size?\n\n2. Missing discussion of the difference between the N-gram-based Generative Retrieval used in DiffusionRet and the SEAL model. Is the main difference being the length of ngrams?\n\n3. This paper is poorly formatted with some blank lines (e.g., Page 3 and 8).\n", "questions_to_authors": "Q1: How to compare the generation quality and inference speed of a diffusion model of the same size and T5? Lack of some important ablation studies.", "paper_topic_and_main_contributions": "This paper proposes a two-stage method called diffusion enhanced generative retrieval (DiffusionRet) for generative retrieval. The method consists of two processing steps: first adopting a diffusion model to generate the hypothetical document for the raw query, then preforming N-gram-based generative retrieval like SEAL [1] based on the generated hypothetical document. Experiment results show that DiffusionRet leads to the state-of-the-art performances.\n\n[1] Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, Fabio Petroni. Autoregressive Search Engines: Generating Substrings as Document Identifiers.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RExqRBz867", "reviewer": "Reviewer_qdE4", "summary": "", "questions": "Question A: \nThe authors claim, \"For MS MARCO 100k dataset, we apply the inference based on the diffusion model on the original 100k queries, obtain a set of 100k generated pseudo documents ... For NQ 320k dataset,  we sample pseudo documents from the original 307k queries\". I was wondering why the sampling is only for NQ 320k.\n\nQuestion B:\nThe caption of Table 1, \"Retrieval performances (Hits@k) on test sets of MS MARCO 100k, comparing DiffusionRet with other previous baselines\". What are the test sets of MS MARCO 100k? Is the full dev set (6980 queries) a test set? If the full dev set is regarded as a test set, there would be no development set. If so, I would like to know how the authors tune the hyperparameters without a development set.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer qdE4\nThank you for raising valuable discussion and questions! We appreciate that the reviewer mentioned the good performance and smaller number of parameters of our work compared to strong baselines. In this discussion we hope that we can further address all your concerns as well as answer the questions. \n\n### 1. Response to Reason to Reject 1: Discussion about the N-gram length and other difference between DiffusionRet and SEAL.\n\nWe are sorry for making this confusion. While it is true that the length of n-gram in our n-gram generative retrieval is different from SEAL [1], it is hard to say that this is our major claim. Please understand that we clarify the difference but we do not claim here that the just use of long lengths of n-gram lead to the improvement in the retrieval performance. More precisely speaking, by putting the first stage for generating a pseudo-document, because its content and length are similar to those of a document (rather than a short query), it is now possible to use large n-grams under the SEAL framework, less likely causing the “unmatched” situation with long n-grams. \n    \nAlthough we don’t fully explore the advantage of using long n-grams, we believe that it can improve the retrieval speed, as the number of required n-gram terms keep much small; at the extreme case, we can only keep single 30-length long ngram, which will likely significantly reduce the retrieval time of using multiple short terms. \n\nIn addition, as replied to the reviewer PVJR, the difference between SEAL [1] and our n-gram generative retrieval is not only the length of n-gram, but also the requirement of the “extensive data augmentation” like query expansion. In the original SEAL [1], spans from a document are uniformly extracted and they are used as additional queries, as much like the data augmentation using the query generation similar to DSI-QG [2] and NCI [3]. On the other hand, our n-gram generative retrieval does not require such data augmentation, where only a single pseudo document per document is used as the source part (not multiple ones), where n-grams are used as target part; detaching such data augmentation from SEAL framework becomes possible resulting from the two-stage setting by putting explicitly the pseudo document generation part. As a result, it is expected that our n-gram generative retrieval can be much efficiently trained comparing to the original SEAL, given the manner that does not rely on such extensive data augmentation. \n\nWe are sorry again for this confusion. If given a chance, we will definitely revise our paper such that the difference in length is not our major and key claim, in any revised version of the paper. \n\nAlso, to clarify the difference between our method and SEAL, (as similarly replied to the reviewer PVJR), if given a chance, we would like to add the single-stage SEAL equipped with the data augmentation in our own experiment settings on MS MARCO 100k and NQ 320k, and revise the main text and appendix such that the training difference and other important ones are clearly revealed, etc, in any extended version of the paper. \n\n\n### 2. Response to Reason to Reject 2: Discussion about the differences between DiffusionRet and query2doc.\n\nThank you for introducing the relevant work. We are sorry for missing this important work. As in query2doc [4], we also believe that LLMs can be applied to generate pseudo documents. But, as similarly replied to the reviewer PVJR, the reasons for using the diffusion model in our work are summarized to 1) there is the recently emerged interest and success in diffusion model and it is interesting and valuable to explore the diffusion model on our addressed semantic inpainting task, 2) the diffusion model is much more parameter-efficient than LLMs; under SLM, the generated results are not satisfactory comparing to that of smaller diffusion model, 3) the semantic inpainting task does not require to generate grammatically perfect sentences, which are one of the key powerful capability in LLMs, so the use of LLMs is not very mandatory, etc. \n    \nIn addition to the responses on the reviewer PVJR, nowadays LLMs still have some known limitations such as hallucination, and so it is yet still valuable to explore other directions like diffusion model in completing the generation task. Also, diffusion models may different pseudo different samples from those of LLMs, and these samples may be complementary each other; Thus, the use of ensembles or combining multiple evidences from different pseudo documents from diffusion models and LLMs will make promising and interesting direction. \n\nWe would like to mention the further difference about query2doc [4]. \n\n1. Parameter size: the most significant difference is the scale of the model. Query2doc [4] employs LLMs as the pseudo-document generator and achieves good results with a 175B LLM, while we only use a diffusion model with 103M parameter as the pseudo-document generator.\n2. Dependency on parameter size: It is also worth noting that Query2doc presented in the paper that for generative models, the number of parameters has a significant effect on the performance. As they mention in Introduction: \n> _Further analysis also reveals the importance of model scales: query2doc works best when combined with the most capable LLMs while small language models only provide marginal improvements over baselines._ \n\n   While our work happens to address the generator's dependence on a large number of parameters using a diffusion model, this was also inspired by the success of previous diffusion model works on controllable text generation and sequence-to-sequence tasks (e.g., Diffusion-LM[5], DiffuSeq[6], SeqDiffuSeq[7], etc., where small diffusion models outperform some of the GPT variants on several different tasks). \n\n3. Sample quality of diffusion models (in the language perspective): Pseudo-documents generated by the diffusion model may differ from those generated by LLMs. As we mentioned in Section 1, the generation of the diffusion model is viewed as \"semantic inpainting\" style; When compared with the results of LLMs, the diffusion-generated documents may not be grammatically or semantically perfect sentences (there may be errors in syntax or grammar, etc., as shown in Appendix A), but can provide rich semantic information which is helpful to the next retrieval stage. Relying the second retrieval stage allows us to generate grammatically less-plausible sentences by diffusion model. \n\nThank you again for letting us the valuable related work. If given a chance, we would definitely add detailed discussion as mentioned above, in any revised version of the paper. \n\n### 3. Response to Question A: Clarification of \"inference\" and \"sampling\"\n\nThank you for raising this question. We apologize for the ambiguity and would unify the statements in the paper to avoid confusion if given chance. These two statements are essentially the same operation. The inference process of the diffusion model is also known as _sampling_, i.e., sampling step by step from a Gaussian distribution to get the generated result. In our case the inference process is to get the pseudo-document by _sampling_.\n\n### 4. Response to Question B: About test/dev set and hyper-parameter fine-tune\n\nThank you for mentioning this issue. We used the full dev set (6980 queries) as test set. This was done on the one hand to maintain the consistency of the evaluation setup with previous related work (e.g., DSI-QG, TOME, etc.), and on the other hand our model achieved the satisfactory performance with the initial hyper-parameter settings (which were set empirically based on previous works), and thus we did not pay much attention to tuning the hyper-parameters.\n\n### 5. Response to Missing References.\n\nThank you for recommending the relevant reference. If given a chance, We would happy to include the discussion about this paper in Related Works section. Since this work used different evaluation metrics than ours, it is difficult to add this work as a baseline to the table, fortunately the Hits@5 metrics on NQ dataset was provided, which can be compared to our Hits@5 in Appendix G (MINDER: 65.8, ours: 77.25). \n\nWe deeply appreciate your precious comments and suggestions and hope this discussion could address all your concerns. Wish you all the best.\n\n    [1] Bevilacqua, Michele, et al. \"Autoregressive search engines: Generating substrings as document identifiers.\" Advances in Neural Information Processing Systems, 2022.\n    [2] Zhuang, Shengyao, et al. \"Bridging the gap between indexing and retrieval for differentiable search index with query generation.\" arXiv preprint arXiv:2206.10128 (2022).\n    [3] Wang, Yujing, et al. \"A neural corpus indexer for document retrieval.\" Advances in Neural Information Processing Systems 35 (2022): 25600-25614.\n    [4] Wang, Liang, Nan Yang, and Furu Wei. \"Query2doc: Query Expansion with Large Language Models.\" arXiv preprint arXiv:2303.07678 (2023).\n    [5] Li, Xiang, et al. \"Diffusion-lm improves controllable text generation.\" Advances in Neural Information Processing Systems, 2022.\n    [6] Gong, Shansan, et al. \"Diffuseq: Sequence to sequence text generation with diffusion models.\" arXiv preprint arXiv:2210.08933 (2022).\n    [7] Yuan, Hongyi, et al. \"Seqdiffuseq: Text diffusion with encoder-decoder transformers.\" arXiv preprint arXiv:2212.10325 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Conduct query expansion by a diffusion-based document generation method.\n2. Consider generative retrieval with long n-grams (previous studies used much shorter n-grams).\n3. The experiments show the effectiveness of DiffusionRet.", "reasons_to_reject": "1. One important claim in the introduction is not supported by experiments: the authors claim, \"Different from the setting in [1] where the length of n-grams is small (i.e., 3-to-5), we consider long n-grams with the length of 30, which are more likely unique to a specific document\". However, there is no corresponding analysis (e.g., a variant of DiffusionRet generating shorter n-grams) in the paper and so it's unclear if generating longer n-grams could really bring about improvements. The length of n-grams is the only difference between this paper and [1] in terms of the phase of retrieval.\n2. The idea of feeding the original query and pseudo document into a generative retrieval method is really similar to query expansion (e.g., [2]). However, the difference from query expansion is not discussed in the paper.\n\n[1] Bevilacqua, M. ,  Ottaviano, G. ,  Lewis, P. ,  Yih, W. T. ,  Riedel, S. , &  Petroni, F. . (2022). Autoregressive search engines: generating substrings as document identifiers.\n[2] Wang, L., Yang, N., & Wei, F. (2023). Query2doc: Query Expansion with Large Language Models. ArXiv, abs/2303.07678.", "questions_to_authors": "Question A: \nThe authors claim, \"For MS MARCO 100k dataset, we apply the inference based on the diffusion model on the original 100k queries, obtain a set of 100k generated pseudo documents ... For NQ 320k dataset,  we sample pseudo documents from the original 307k queries\". I was wondering why the sampling is only for NQ 320k.\n\nQuestion B:\nThe caption of Table 1, \"Retrieval performances (Hits@k) on test sets of MS MARCO 100k, comparing DiffusionRet with other previous baselines\". What are the test sets of MS MARCO 100k? Is the full dev set (6980 queries) a test set? If the full dev set is regarded as a test set, there would be no development set. If so, I would like to know how the authors tune the hyperparameters without a development set.\n", "paper_topic_and_main_contributions": "This paper aims to propose a new generative retrieval method, DiffusionRet. It first expands a given query with a pseudo document generated by a diffusion model. Then it generates long n-grams that would be mapped to an original document in the collection, given the original query and pseudo document. The experimental results show that DiffusionRet not only exhibits great performance but also has a smaller number of parameters compared to strong baselines.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The following paper considered generating synthetic identifiers, which are longer n-grams.\n\n[3] Li, Y., Yang, N., Wang, L., Wei, F., & Li, W. (2023). Multiview Identifiers Enhanced Generative Retrieval. ArXiv, abs/2305.16675."}}]

Author rebuttal:

Meta-review: All the reviewers have agreed that it is interesting to see the integration of diffusion models into generative retrieval and the improvement gained.

However, common concerns have also been raised that the motivation is somehow weak and some discussion on the relations and differences between the proposed method and its competitors are missing.

During the rebuttal period, the authors have only partly addressed these concerns, leaving room for further improvement.
Decision: Accept-Findings