Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback
Authors: Yujia Zhou
Zhicheng Dou
Ji-Rong Wen
Venue: EMNLP 2023 Main (2023)
Keywords: Generative retrieval; Reinforcement learning; Document retrieval;

Abstract: The recent advent of end-to-end generative retrieval marks a significant shift in document retrieval methods, leveraging differentiable search indexes to directly produce relevant document identifiers (docids) in response to a specific query. Nevertheless, this approach faces two fundamental challenges: (i) a discrepancy between the token-level probabilistic optimization and the broader document-level relevance estimation; (ii) an overemphasis on top-1 results at the expense of overall ranking quality. To tackle these challenges, we propose a generative retrieval model with reinforcement learning from relevance feedback, which aims to align token-level docid generation with document-level relevance estimation. The training process incorporates three stages: supervised fine-tuning, relevance reward model training, and reinforced learning-to-rank from relevance feedback. To train a high-quality reward model, we define "relevance" under three progressive scenarios, which collectively offer a comprehensive evaluation of the document relevance. Experiments conducted on two benchmark datasets demonstrate the effectiveness of our proposed approach.

Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback
Yujia Zhou', Zhicheng Dou?; Ji--ong Wen
School of Information, Renmin University of China "aaoling School of Artificial Intelligence, Remmin University of China [zhouyujia, jtreaad,,,,j,jeeeeeerr jrw.n)eruc.edde cn
Abstract
et al., 2022: Zhou et al., 2022a). These models supersede the conventional explicit index by imple- menting a large-scale model known as the differ- entiable search index (Tay et al., 2022). This in- novative approach facilitates end-to-end document retrieval using seq-to-seq generative model, such as T5 (Raffel et al., 2020). In this framework, the model directly generates the corresponding docid for a given query.
The recent advent of end-to-end generative retrieval marks significant shift in docu- ment retrieval methods, leveraging differen- tiable search indexes to directly produce rel- evant document identifiers (docids) in response to a specific query. Nevertheless, this approach faces two fundamental challenges: (i) dis- crepancy between the token-level probabilistic optimization and the broader document-level relevance estimation; (ii) an overemphasis on top-1 results at the expense of overall rank- ing quality. tackle these challenges, we propose generative retrieval model with re- inforcement learning from relevance feedback, which aims to align token-level docid genera- tion with document-level relevance estimation. The training process incorporates three stages: supervised fine-tuning, relevance reward model training, and reinforced learning-to-ran to-rank from relevance feedback. To train a high-quality re- ward model, define "relevance" under three progressive scenarios, which collectively of- fer comprehensive evaluation the docu- ment relevance. Experiments conducted two benchmark datasets demonstrate the effective- ness of our proposed approach.
Despite the notable progress made in generative retrieval models, they still face two major chal- lenges that limit their ability to retrieve relevant documents. First, these models primarily rely on the auto-regressive loss to maximize the token-level generation probability of the ground-trume docid, However, such optimization goal is not aligned with the primary objective of ranking tasks, which is to estimate document-level relevance. Second, existing approaches assume deterministic distri- bution in which all probability mass is allocated the "golden" docid. Such an assumption can lead the model to overemphasize the top-1 generation results, while ignoring the quality of the overall ranking list. In light of these challenges, establish- ing alignment between generation probabilities and document relevance is vital necessity in adapting generative models to ranking tasks.
1 Introduction
Information retrieval systems have undergone progressive advancements over the past few decades (Zhu et al., 2023). Historically, the re- trieval candidate documents has been heavily re- liant on the use of an inverted index (Robertson and Zaragoza, 2009), a conventional method that em- ploys term-based parsing. With the emergency of neural networks, dense retrieval (Karpukhin et al.. 2020; Xiong et al., 2021) is proposed encode both query and document into dense vectors, so as measure the similarity in semantic space.
In this paper, we present a Generative Retrieval model with Reinforcement Learning from rele- vance feedback (GenRRL). Our objective align the generative retrieval model, trained specifically on the task of docid generation, with the model responsible for assessing document relevance. The training process comprises three pivotal steps: (1) Supervised fine-tuning. This initial step involves training the model using labeled data to improve its performance in generating docid. (2) Relevance re- ward model training. The subsequent step focuses on training a relevance reward model, specifically designed to measure the degree of relevance be- tween the generated docids and their correspond-
Recently, there has been a growing focus on investigating end-to-end generative retrieval mod- els (Tay et al., 2022; Bevilacqua et al., 2022; Wang
*Corresponding author

---

ing queries. We expect the reward model to assign higher scores docids that are more relevant to the queries. (3) Reinforced learning-to-rank (L2R) from relevance feedback. The final step involves using the trained relevance reward model to guide the generative retrieval system towards generating more relevant docids. To align with the ranking task requirements, we incorporate the idea of L2R into the reinforcement learning procedure. This enables the provision of not only pointwise reward but also pairwise and listwise relevance feedback, thereby enhancing the overall ranking performance.
align the docid generation process with document relevance assessment. (2) We outline reinforeed training process specifically tailored for the ranking task, which includes relevance reward model train- ing and reinforced learning-to-rank, (3) We provide classification of "relevance" within the context of document retrieval. By considering three pro- gressive aspects, the trained reward model enables thorough assessment of the relevance between queries and documents.
Related Work
In order to train a high-quality relevance reward model, it is crucial to define "relevance" accurately within the context of document retrieval tasks. In this regard, focus three key scenarios that robustly reflect the relationship between queries and documents. Firstly, term-based overlap con- cerns the presence of identical or similar words within both the query and the document. Tech- niques like TF-IDF can effectively gauge this over- lap. Secondly, semantic similarity extends beyond mere term overlap, delving into the shared mean- ings concepts within the query and the docu- ment. Dual encoder models are particularly useful in modeling the semantic similarity by encoding the query and the document into dense vectors. Lastly, contextual dependency considers how ef- fectively the document satisfies the information needs expressed in the query, extending beyond the boundaries of specific terms or concepts within the query. Techniques for capturing contextual depen- dency could include language models, which have demonstrated proficiency in understanding the con- textual relationships between two sentences. By considering these three progressive scenarios, we can comprehensively assess the relevance between queries and documents, leading to a more effective reward model.
Traditionally, index-hased document retrieval meth- ods encompass both sparse and dense retrieval tech- niques. Sparse retrieval methods (Dai and Callan, 2019, 2020; Bai et al., 2020; Formal et al., 2021), such BM25 (Robertson and Zaragoza, 2009), utilize inverted indexes assess term importance and calculate matching scores between queries and documents. Dense retrieval methods have been employed to overcome the limitations of word mis- matching (Gao et al., 2021). These methods con- vert queries and documents into dense vectors and employ ANNS (Jégou et al., 2011) for efficient vector-based search. The dual encoder approach in- corporating pre-trained language models has shown improved performance in dense retrieval. Tech- niques like hard negative sampling (Xiong et al., 2021; Karpukhin et al., 2020; Zhan et al., 2021; Guu et al., 2020) further enhance effectiveness.
In recent times, generative retrieval has gained significant popularity in the field of information retrieval (IR) tasks. A model-based IR approach, where docids are embedded into the model, was first introduced in (Metzler et al., 2021; Tay et al., 2022). Since then, several other models (Wang et al., 2022; Zhuang et al., 2022; Zhou et al., 2022b) incorporating query augmentation module have been developed to enhance the performance. More- over, the model-based IR framework has been tended to knowledge-intensivee language tasks, lead- ing to notable improvements in performance (Chen et al., 2022a; Bevilacqua et al., 2022; Chen et al., 2022b). Additionally, a series of studies by Sun et al. (2023); Chen et al. (2023); Ren et al. (2023); Li et al. (2023) explored various methods for repre- senting docids to retain docid semantics.
To assess the effectiveness of our model, perform comprehensive experiments on widely adopted MS MARCO and NQ document retrieval datasets. Experimental results demonstrate the effi- cacy of the proposed method, including the diverse relevance annotations and the reinforcement Iearn- ing process. Our in-depth analysis on different model sizes also reveals the scaling laws of our proposed method.
In this paper, we focus on aligning the optimiz- ing objective of generative retrieval models with document-level relevance, so as to alleviate the bias against the objective of ranking tasks.
Our contributions are summarized as: (1) We em- ploy reinforcement learning from relevance feed- back to improve generative retrieval, aiming to

---

scores from the generative model, whereas the rank- ing score curve of the dense retrieval model appears smoother. This outcome is likely due to the discrep- ancy between the generation probability and the document relevance. To enhance the model ability on document relevance estimation, we propose a reinforcement learning framework for generative retrieval as follows.
cense retrievel
generative retrivval
Ranking Scores
#Docs
Rank position
Figure 1: Empirical study on ranking results of genera- tive retrieval and dense retrieval.
Methodology
In this section, we focus on the training pipeline of GenRRL, including supervised fine-tuning, rele- vance reward model training, and reinforced L,RR from relevance feedback.
3
Preliminary
In this section, we formulate the generative retrieval problem and investigate its limitations.
4.1 Supervised Fine-tuning
3.1 Task Definition
Supervised fine-tuning (SFT) entails training the model to generate relevant docids based token- level generation probabilities for given query. Specifically, we choose keyword- docids (e.g. URL, summary) as the training target, as these can more intuitively reflect the semantics of the doc- ument. In addition, we select standard T5 as the pre-trained model and apply pscudo query genera- tion for data augmentation, which has been proven to be effective in improving the performance of gen- erative retrieval (Zhuang al., 2022). Formally, the generative loss function can be formulated the maximization of the likelihood of the target sequence using teacher forcing, we have:
Generative retrieval aims to generate the relevant docids for a given query directly with seq-to- seq model. Assuming that d' is the identifier of the document the generative retriever takes the query input and generates the relevant docid with an auto-regressive score, written as:
score(d, g) = Go(d - II Go(d)den))) (1) E=
where G is the generative retriever with param- eters Φ. and d' is the deh token of the docid. The parameters & is optimized with the standard seq-to- seq objective. During inference, the model applies constrained beam search to guide the decoder in searching within a limited token space at each step, as to generate valid docids from candidate set.
Egen log (ol')'
Σ log GogG(dd)ees)))
(2)
3.2 Empirical Study
where Goldlld's is the generation probability of token d' based on the given input. After su- pervised fine-tuning, our next goal to train a reward model designed to incorporate relevance preferences into docid generation process.
In this section, we present an empirical study aiming to investigate the limitations of genera- tive retrieval models compared to advanced dense retrieval models in ranking tasks. We choose DPR (Karpukhin al., 2020) and Ultron (Zhou et al., 2022b) as representatives of the dense and generative retrieval model respectively, and investi- gate their ranking results on NQ dataset.
4.2 Relevance Reward Model Training
A high-quality relevance reward model is capa- ble of effectively assessing the relevance between queries and documents. To accomplish this, we conduct analysis of relevance judgments within the document retrieval. Subsequently, we employ three distinct methodologies to annotate document relevance, thereby facilitating the training of com- prehensive relevance feedback model. The details are introduced as follows.
As shown in Figure 1, it can observed that: Firstly, the generative retrieval model demonstrates higher likelihood of placing the correct document the top position, while the dense retrieval model exhibits a more balanced distribution across all rank positions. Secondly, the relevant documents positioned at the first rank receive remarkably high

---

Step 1: Supervised Fine-tuning
Stap 2: Relevance Reward Modil Training
Step 2: Reinforced Learning Rank
Sample query
Sample snother query
Corpus
what is cost of peoport
How to exiep fast
Human libeling
Query pereration
Generative Rutriover
Generative
Perguort Fies
United Stetis
panaport
Renaw paupert
Peopport Applying
Shepp
Auasiity Miluaai Sleep
Meslbatión
Lobeled
Pasulla queries
tways To
Aclepp
LLM
Generalive Day
Reward Modil
Rewerd Model
Reward Model
Decument ilentifer
Painwie
Rewerd Modell
better
than i
Polntwise
Pairwise
Lébase
Tralnable parameters
parameters
Gierr r0oc
Relevence feedback
Figure 2: The training pipeline of GenRRL with supervised fine-tuning, relevance reward model training, and reinforced tearning-to-rank from relevance feedback. Blue arrows indicate data used for model training.
Relevance Classification. We examine three sce- narios wherein queries and documents exhibit strong associations. They are:
tuned model, which corresponds to its own beam search results specific to that query. order to capture the relevance of the above three scenar- ios respectively, we use three different models to annotate the relevance of the sampled query-doc pairs: (1) To measure the term-based overlap, we leverage BM25 (Robertson and Zaragoza, 2009), well-established ranking function used by search engines. BM25's strength lies in its ability to weigh query terms based on their TF-IDF importance, thus providing a robust measure of term- over- lap. (2) To capture the semantic similarity, we employ the dual encoder model DPR (Karpukhin et al., 2020), which works by separately encod- ing the query and document into dense vectors in shared embedding space. It focuses deeper semantic connections between words and phrases in the query and the document. The dot product of the two vectors then gives a measure of the se- mantic similarity. (3) For assessing contextual dependency, we employ the large language model (LLM), LLaMA-13b (Touvron et al., 2023), to com- pute the generation likelihood of the document given specific query. We assume that the gen- eration probability can as indicative met- ric of how well the document fulfills the implicit demands enclosed in the query.
Term- based Overlap. This scenario refers to the presence of identical or similar terms in both the query and the document. a document contains many the same words that are the query, is likely to be relevant.
Semantic Similarity. This goes beyond term over- lap to look at the meaning of the words and phrases the query and document. Even a doc- ument does not contain the exact terms present in the query, it might still be relevant if it discusses the same concepts. For example, query for "cardiovascular fitness exercises" might retrieve document about "aerobic workouts" because the two phrases refer similar concepts.
Contextual Dependency. This scenario refers to the extent which document fulfills the infor- mation needs expressed by the query, surpassing the limitations posed by specific terms or con- cepts. For instance, a user has been searching for "how to save battery of laptops". A relevant document might say: "reducing screen brightness and turning off keyboard backlight".
Relevance Annotation. For each sampled query g from supervised fine-tuning dataset, we generate a set of candidate documents based on the fine-
To mitigate the discrepancies arising from varia- tions in scoring distributions across different meth-

---

ods, rankings are employed as a means of introduc- ing regularized supervision signals. Specifically, when provided with multiple rankings of the candi- date documents, we adopt a voting-based approach, Borda Rank, to consolidate them into a unified ranking. For example, given N items and M indi- vidual rankings, the Borda rank score B; for item i is calculated as B, 2j=1 (N rankk (t)) where ranky(1) represents the rank of item in the j-th individual ranking. Based on ranking results, we can construct document pairs including a posi- tive document d+ and a negative document d= for reward model training.
To align with the requirements of the ranking task, we introduce the concept of learning-to- into the RL procedure, employing pointwise, pair- wise, and listwise optimization strategies.
Pointwise Optimization. We begin by inde- pendently updating the model for each query- document pair. For given query q. the generative retriever functions as a policy network, generating a candidate docid corresponding to the document d. The RL loss can be formulated as:
CRL = - - b) Σ log Go(di
where b is the baseline incorporated to mitigate variance during RL training. It is computed by av- eraging the rewards of all samples.
Reward Model training. To train our reward mod- els, we start from the supervised fine-tuned model, and then add a randomly initialized linear head that outputs scalar reward. The reward model is trained in a pairwise manner to predict which doc- ument is more relevant. Given the query and a document pair (d+, dd) sampled from the ranking D, the loss function for the reward model is:
Pairwise Optimization. Instead of regarding each query-
pair independently, we incorpo- rate the assessment of pairwise document compar- isons as form of reward. Given query q. the docid generation model samples plenty of docu- ment pairs (d, d,). The generator model is trained with the pairwise loss, denoted as:
CRM Fl (f)= [og((a d+)) ro(a, d ))),
where re(g, d) is the scalar output of the reward model with parameters A.
CRI. -
Σ (R(g. d;) log (d..d))
d,) log
Now we have an initial model capable of docid generation and a preference model that assigns score to assess the document relevance. Subse- quently, reinforcement learning employed op- timize the initial model based on the reward model.
where Pty computed by [Go(d]lg) - Gd(()l))) with the normalization of logistic function.
Listwise Optimization. Listwise strategy provides an alternative approach by considering the entire list of generated documents. In this context, we define a listwise loss that aims to minimize the ranking discrepancy between the SFT model and the reward model. Given set of sampled candi- dates C for the query q, the loss function can be defined as:
4.3 Reinforced L2R from Relevance Feedback
To begin with, we cast the generative retrieval task as a reinforcement learning (RL) problem. Initially, the policy is defined as a language model that ac- cepts query as input and generates a sequence of tokens as output. The action space of the policy encompasses all tokens in the language model's vo- cabulary, while the observation space corresponds to the distribution of potential input queries.
R(g, d) log exp (Gold") (q))
LRI. = Σ
Σ,exp exp (Go(d")a))
dEC
By incorporating listwise optimization, the gener- ative retriever is promoted to generate docids that closely resemble the ordering preferred by the re- ward model. This approach takes into account the overall quality and relevance of the entire list of generated documents, rather than focusing solely individual document or comparisons.
Reward Function. To guide the RL training, the reward function is constructed by combining the re- wards re(g, d) generated by the reward model with KL divergence constraint, which ensures that the policy does not deviate significantly from its initial behavior, defined as:
R(g, d) = re(g, d) log (dlg)/
5 Experimental Setup
where TT is the learned RL policy, n SPT is the su- pervised fine-tuned model, 3 is The KL reward co-
5.1 Datasets and Evaluation Metrics
Our experiments are conducted on two widely used document retrieval benchmarks: MS MARCO Doc-
efficient controlling the strength of the KL penalty.

---

ument Ranking (Nguyen et al., 2016) and Natural Questions 320k (Kwiatkowski et al., 2019). MS MARCO Document Ranking dataset is a large- scale collection specifically designed for the doc- ument ranking task. We create a candidate docu- ment subset of approximately 320k articles based on the labeled documents. Natural Questions (NQ) dataset comprises actual questions along with cor- responding Wikipedia articles. During the dataset processing, we extract the title, abstract, and con- tent from each Wikipedia article while removing unnecessary HTML tags, We climinate duplicate documents based on their URL, which remains around 231k articles, with 307k training pairs and 7.8k test queries. To assess the ranking perfor- mance of document retrieval, we report MRR and Hits (at cutoff 1, 5, 10) the evaluation metrics.
bone and sets the learning rate and the batch size as 1e-3 and 128 during training. To enrich the training data, we employ the existing Ducer model (Nogueira et al., 2019) to generate 10 pseudo queries for each document.
RL. The reward model is initialized using the SFT model, with línear head that is randomly initial- ized. During the training of the reward model, we employ the SFT model to generate 8 candidate do- cids for relevance annotation. We set the batch size to 64 and utilize learning rate of 3e-4 for this training phase. In the reinforced L2R process, the policy model provides 1, 2, 4 candidate docids at each step corresponding to pointwise. pairwise, and listwise optimization, while the learning rate is set to 3e-5. All experiments are conducted on up to 8 NVIDIA A100 GPUs.
5.2 Baselines
Experimental Results and Analysis
To establish baselines for comparison, we con- sider three classes of models: (1) Sparse Re- trieval models. We select BM25 (Robertson and Zaragoza, 2009), which incorporates tf-idf weighting to determine term importance, and DocTsQuery (Nogueira et al., 2019), which ex- pands document with queries predicted by T5 model, as baselines.
In this section, we present the experimental find- ings of our proposed approach and perform an em- pirical analysis to provide comprehensive insights.
6.1 Overall Results
The overall results of models are listed in Table 1. Some findings are summarized as follows.
(2) Dense Retrieval mod- els. We consider two dual encoder models trained with different hard negatives sampling strategies DPR (Karpukhin et al., 2020) and ANCE (Xiong et al.. 2021). (3) Generative Retrieval models: DSI (Tay et al., 2022), the pioneering generative retrieval model that uses seq-to-seq model to map the query to the relevant docid. Building upon DSI, SEAL (Bevilacqua et al., 2022) uses n-grams within passages as potential identifiers. DSI-QG (Zhuang et al., 2022), NCI (Wang et al., 2022), and Ultron (Zhou et al., 2022b) incorporate query generation modules for data augmentation. To ensure a fair comparison, all baseline models employ the "base" version of pre-trained models.
Comparison with Generative Retrievers. Our ob- servations indicate consistent superior performance of GenRRL compared to baseline models and SFT models when evaluated on the MS MARCO and NQ datasets. This pattern of results serves com- pelling evidence for the effectiveness of our rein- forced training strategy. Notably, when compared to the leading generative retrieval model Ultron on the MS MARCO dataset, GenRRL. (URL) ex- hibits a noteworthy improvement of over 9.6% terms of MRR @ 10. Similarly, on the NQ dataset, ROGER (Sum) surpasses the competitive baseline NCI by over 8. % in MRR @ 10. These findings strongly support the notion that integrating rele- vance feedback into the initial generative retrieval model holds immense promise for enhancing per- formance in ranking tasks.
5.3 Implementation Details
Docid. We explore two types of linguistic docids: document URL and document summary. For the summary-based docids, we employ the LLaMA- 13b model to generate document summaries using the prompt "Summarize the following paragraphs with meaningful keywords: [document]".
Effect of Different Docids. crative retrieval, the selection of docids can be cat- egorized into two forms: linguistic docids and ID- based docids. Through a comparative analysis be- tween DSI-QG and Ultron, we have observed that linguistic docids are better suited for generative retrieval tasks. Building upon this insight, we have
In the context of gen-
SFT. Our SFT model utilizes the T5-base pre- trained model (Raffel et al., 2020) as the back-

---

Table Overall results. Document Rep, indicates the method to represent document, The best results are shown in bold. "f" indicates the result is significantly better than the corresponding SFT model with paired t-test at < 0.05.
MS MARCO
Natural Questions
Model
Document Rep,
Hits@1
Hits@s
Hits@ 10
MRR@10
Hits@1
Hits@5
Hits@
MRR@ 10
Index-based Retrieval
BM25
sparse terms
18.94
42.82
55.07
29.24
14.06
36,91
47.93
23.60
Docery
sparse terms
23.27
49.38
63.61
34.81
19.07
43.88
55.83
29.55
dense vector dense vector
62.75
73.13
43.41
22.78
35,92
ANCE
29.65
63.43
74.28
44.09
24.54
54.21
69.08
36.88
Generative Retrieval
DSI
semantic id
25.74
43.58
53.84
33.92
27.42
47.26
56.58
34.31
DSI-QG
semantic id
28.82
50.74
62.26
38.45
30.17
53.20
66.37
38.85
SEAL
n-grams
27.58
52.47
61.01
37.68
29.30
54.12
68.53
40.34
NCI
semantic id
29.54
$7.99
67.28
40.46
32.69
55.82
69.20
42.84
Ultron
title URL
29.82
60.39
68.31
42.53
33.78
54.20
67.05
42.51
SFT (URL)
doc URL
29.90
60.28
68.17
42.52
33.91
54.18
66.92
42.52
SFT (Sum)
dec summary
30.12
60.68
68.71
42.84
33.72
54.71
67.69
42.83
GenRRL (URL)
doc URL
33.01
63.62
74.91
45.93*
35.791
56.491
70.961
45.731
GenRRL (Sum)
doc summary
33.23
64.48*
75.80
46.62
36.32
57.421
71.49
46.31
Table 2: The results of different relevance annotators. and indicates annotating with and without the model.
Relevance annotator
MS MARCO
Natural Questions
Model
Sparse
Dense
LLM
Hits@10 10
MRR@ 10
Ilits@11 10
MRR @ 10
SFT Model (Sum)
68.71
42.84
67.69
42.83
GenRRL (Sum)
0
74.37
45.85
70.40
45.32
GenRRL (Sum)
73.19
45.18
69.78
44.67
GenRRL (Sum)
c
75,36
46.28
70.92
45.81
GenRRL (Sum)
75.80
46.62
71.49
46.31
explored two types of linguistic docids, namely doc URL and doc summary. The experimental results demonstrate that our model consistently outper- forms purely supervised fine-tuned models both types of linguistic docids. Furthermore, the per- formance is particularly enhanced when utilizing summary-based docids, indicating that docids gen- erated by large language models can better capture document relevance compared to the more simplis- tic approach of URL-based docids.
bution to the overall performance, with the dense model cxhibiting the most notable improvement in results. This finding suggests that the dense model possesses more pronounced advantage in assessing the relevance between queries and doc- uments. Additionally, the inclusion of the LLM annotator yields performance enhancements. in- dicating that leveraging the zero-shot capability of large language models can further augment the model's ability to capture contextual dependencies.
6.2 Ablation Study on Relevance Annotators
6.3 Comparison of Different RL Losses
In this section, we perform ablation study to assess the impact of relevance annotation in the GenRRL model. As described in Section 4.2, the relevance annotation process involves three types of annotators: the sparse model, the dense model, and LLM, each representing distinct relevance see- narios. We explore various combinations of these annotators to evaluate their effectiveness.
The incorporation of relevance feedback signals with reinforcement learning serves crucial step in model fine-tuning. To investigate the impact of different reinforced feedback strategies, we com- pare three reinforcement learning losses: pointwise, pairwise, and listwise, and examine their influence on ranking performance for both type of docids.
Table presents the results obtained for both URL-based docids and summary-basee docids. It is observed that as the loss transitions from point- wise to pairwise and listwise, there is a gradual im-
Table presents the results for variants of Gen- RRL. We can observe that each annotator in the relevance annotation process demonstrates a contri-

---

(Sum)
DPR Ultron
GenRRL (5um)
0/5
0.5
MAR310
MRR910
622
Small
Small
Base
Bese
Large
large
0.3
1260
training steps
training steps
0.1
lemm-bosed
(a) Scale of SET model.
(b) Scale of reward model.
Query
Figure 3: Performance on different query sets corre- sponding to three types relevance scenarios on MS.
Figure 4: The variation trends on MS with the increase of training step under different model scales.
Table 3: The MRR @ 10 of different RL losses.
6.5
Effect on Model Scale
Model
Loss
MS
NQ
To investigate the training characteristics of models with varying parameter scales, we try T5-small, T5-
GenRRI (URI)
Pointwise
43.42
43.01
(URL)
Pairwise
44.82
45.10
GeaRRL (URL)
Listwise
45.93
45.73
base and T5-large models for training SFT model and reward model respectively.
GeaRRL (Sum)
Pointwise
43.78
43.92
GenRRL (Sum)
Pairwise
45.37
45.46
GeaRRL (Sum)
Listwise
46.62
46.31
Scale of SFT Model. The learning curve depicted in Figure 4(a) illustrates the convergence behavior of the GenRRL model with varying parameter sizes as the number of training steps increases. It is ob- served that the small model exhibits comparatively lower MRR at convergence. Conversely, both the base and large models achieve similar performance levels after sufficient number of training steps, with the large model slightly surpassing the base model. These findings emphasize the critical influ- ence of model capacity on retrieval performance, indicating that larger models have the potential to yield improved results given adequate training.
provement on the search results. This progressive enhancement suggests that the listwise loss func- tion could better alleviates the issue of the model's excessive focus on top-1 results. By assuming non-deterministic distribution, the listwise loss as- signs probability mass to other candidate docids based on their relevance.
6.4 Performance on Different Query Sets
To investigate the distinctions between the three relevance scenarios, we partitioned the test queries into three categories based on the performance of sparse, dense, and generative retrieval. These cat- egories align with the dimensions of derm-based overlap, semantic similarity, and contextual depen- dency, respectively. We then conducted a compar- ative analysis of the model's performance before and after reinforcement learning.
Scale of Reward Model. Figure 4(b) shows the learning curve with different scale of the reward model. We observe that during the initial 4000 training steps, the differences between reward mod- els of different sizes are not distinct. However, as training progresses, the small reward model ex- hibits early convergence. In contrast, the base and large reward models continue improve their per- formance, with the large reward model outperform- ing the base model. This finding suggests that the larger reward model demonstrates stronger capabil- ity in capturing capture nuanced distinctions among sampled docids, leading to continued improvement.
From Figure 3, we find that the BM25 and DPR models have demonstrated noticeable advantages on their respective query sets. Although the SFT model exhibited certain gaps in comparison, the in- troduction of reinforcement learning allowed us to incorporate relevance signals from both the sparse and dense models. Furthermore, the SFT model augmented by incorporating the knowledge of the LLM. As a result, the retrieval quality across the three query sets experienced improvements. These enhancements signify the effectiveness of reinforce- ment lcarning process and diverse relevance signals, leading to enhanced retrieval performance.
7 Conclusion
In this paper. address the challenges faced by generative retrieval models by proposing an RL- based approach called GenRRL. Our objective is to align the generative retrieval with the task assessing document relevance, thereby improving

---

ranking performance. We introduce a three-stage reinforcement learning framework, where the rele- vance reward model captures different dimensions relevance by considering term-based overlap, semantic similarity, and contextual dependency. Through extensive experiments, we demonstrate the effectiveness of our proposed method in im- proving generative retrieval performance.
2022. Autoregressive search engines: Generat- ing substrings as document identifiers. CoRR. abs/2204. 106288
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2022a. GERE: generative evi- dence retrieval for fact verification. In SIGIR, pages 2184-2189. ACM.
Jiangui Chen, Ruqing Zhang. Jiafeng Guo, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2022b. Corpusbrain: Pre-train generative retrieval model for knowledge- intensive language tasks. In CIKM, pages 191-200. ACM.
Acknowledgement
This work was supported by the National Natural Science Foundation of China No. 62272467 and No. 61832017, Beijing Outstanding Young Scien- tist Program No. BJ/WZYJH012019100002000
Xiaoyang Chen, Yanjiang Liu, Ben He, Le Sun, and Yingfei Sun. 2023. Understanding differential search index for text retrieval. CoRR, abs/2305.0200
Public Computing Cloud, Renmin University of China, Engineering Research Center of Next- Generation Intelligent Search and Recommenda- tion, MOE, and Intelligent Social Governance Plat- form, Major Innovation Planning Interdisci- plinary Platform for the "Double-First Class" Ini- tiative, Renmin University China. The work was partially done at Beijing Key Laboratory of Big
Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. CoRR, abs/1910. 10687..
Zhuyun Dai and Jamie Callan. 2020. Context-aware document term weighting for ad-hoc search. WWw '20: The Web Conference 2020, Taipei, Tai- wan, April 20-24, 2020, pages 1897-1907. ACM IW3C2.
Data Management and Analysis Methods.
Thibault Formal, Carlos Lassance, Benjamin Pi-
Limitations
wowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse lexical and expansion model for informa- tion retrieval. CoRR, abs/2109.10086.
Despite the notable progress achieved within reinforced framework, GenRRL encounters several challenges that necessitate further exploration and advancement. Primarily, the extension of the model to web scale poses substantial requirements for the design of the differentiable search index, demand- ing enhanced capacity to handle vast amounts of data. Secondly, the incorporation of new incom- ing documents into the differentiable search index has yet been thoroughly investigated. This as- pect presents an unexplored avenue that requires careful exploration and analysis. Addressing this challenge entails devising strategies and techniques to seamlessly integrate new documents into the existing model-based framework, while ensuring that the indexer retains its efficiency, accuracy, and robustness.
Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Ben- jamin Van Durme, and Jamie Callan. 2021. Comple- ment lexical retrieval model with semantic residual embeddings. In Advances in Information Retrieval 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 April 1, 2021, Pro- ceedings. Part I, volume 12656 of Lecture Notes in Computer Science, pages 146-160. Springer.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming-Was Wei Chang. 2020. REALM: retrieval- augmented language model pre-training. CoRR, (5./2002.089099
Hervé Jégou, Matthijs Douze, and Cordelia Schmid. 201 Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell., 33(1):117-128.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. EMNLP (I), pages 6769-6781. Association for Computational Linguistics.
References
Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020. Sparterm: Learning term-based sparse representation for fast text retrieval. CoRR, abs/2010.00768.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova. Llion Jones. Matthew Kelcey, Ming-Was Wei Chang, Andrew M. Dai, Jakob
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen-tau Yih, Sebastian Riedel, and Fabio Petroni.

---

Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452- 466.
memory as a differentiable search index. CORR. abs/2202.06991.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. IJama: Open and efficient foundation language models. CoRR, abs/2302.
Yonggi Li. Nan Yang, Liang Wang. Furu Wei, and Wen- jie Li. 2023. Multiview identifiers enhanced genera- tive retrieval.
Donald Metzler, Yi Tay, Dara Bahri, and Mare Najork. 2021. Rethinking search: making domain experts out of dilettantes, SIGIR Forum, 55(1):13:1-13:211
Yujing Wang. Yingyan Hou, Haonan Wang. Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuging Xia, Chengmin Chi, Guoshuai Zhao. Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A neural corpus indexer for document retrieval. CoRR, abs/2206.02743.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng- 2016. MS MARCO: human generated machine reading comprehension dataset. In Proceedings of the Workshop Cognitive Computation: Integrat- ing neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Infor- mation Processing Systems (NIPS 2016). Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings. CEUR-WS,org WS..rg.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text trieval. In ICLR. OpenReview.net.
Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019. From doc2query doctuttqquery. Online preprint, 6.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. Si- GIR '21: The 44th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1503-1512. ACM.
Colin Raffel. Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:66
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, and Ji- Rong Wen. 2022a. Dynamicretriever: A pre-training model-based IR system with neither sparse nor dense index. CoRR, abs/2203.00537
Ruiyang Ren, Wayne Xin Zhao, Jing Liu, Hua Wu, Ji- Rong Wen, and Haifeng Wang. 2023. TOME: A two-stage approach for model-based retrieval. CoRR, Abs/2305.11166
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Peitian Zhang, and Ji-Rong Wen. 2022b. Ultron: An ulti- mate retriever on corpus with model-based indexer. CoRR, hass/2208.022555
Stephen E. Robertson and Hugo Zaragoza. 2009, The probabilistic relevance framework: BM25 and be- yond. Found. Trends Inf. Retr., 334)3333-3
Weiwei Sun, Lingyong Yan. Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023. Learning to tokenize for generative retrieval. CoRR, abs/2304.04171.
Yutao Zhu, Huaying Yuan, Shuting Wang. Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen, 2023, Large language models for infor- mation retrieval: A survey. CoRR, abs/2308.07 071107
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the gap between indexing and retrieval for differentiable search index with query generation. CoRR, abs/2206. 10128.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Prakash Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer


Reviews:
[{"review_id": "iC9ezeZyZn", "reviewer": "Reviewer_UusY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough and constructive review of our paper. We sincerely appreciate your recognition of the contributions of our work, particularly in identifying the novelty of our reinforcement-based optimization method, GenRRL, and its potential significance to the community.\n\nWe are heartened by your positive feedback, which motivates us to maintain the standards of our work and possibly even to enhance it further in subsequent versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-motivated, well-written. The preliminary experiments highlights the need for improving the optimization, and the method is clearly explained.\n- The reinforcement-based method is novel and can be of interest of the community working in this area.\n- The experiment results show the benefits of the method.", "reasons_to_reject": "Nothing specific.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work highlights the shortcomings of the recent generative retrieval models, namely overemphasis on the first document in the rank and lack of connection between the document-level relevance and token-level optimization, and suggest a novel reinforcement-based optimization to address these. The proposed GenRRL extends supervised training by defining a reinforcement loss function based an various reward definitions. The experiments on MS MARCO Document Ranking and Natural Questions datasets show that the method outperforms the recent generative retrieval models as well as some index-based dense retrieval ones.  ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IMFnrWIfQl", "reviewer": "Reviewer_pWtF", "summary": "", "questions": "Regarding the reward model development: \n\nQuestion A: Is directly using the ranking metric like MRR or NDCG as the reward a bad idea? If so, why? Perhaps the authors could provide some experiments to show that this approach does not work well.\n\nQuestion B: An alternative to Borda Rank is reciprocal rank fusion. Does this work as well as Borda Rank? The authors could provide some experiments to compare the two approaches.\n\nQuestion C: In addition to pairwise loss, does pointwise/listwise distillation loss also work in *reward model training*? The authors could provide some experiments to compare the performance of these different loss functions.\n\nI believe that providing this additional information would make the paper stronger and more convincing.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meticulous review and appreciation of our technical originality. We genuinely value your insights and are grateful for the opportunity to address your questions. We hope that the following clarifications will allay your concerns and further consolidate our paper's position.\n\n##  Explanations on the Reward Model:\n\n#### In Response to Quesion A:\nIn our early experiments, we did consider ranking metrics such as MRR as a direct reward, drawing inspiration from [1] where Rouge served as the reward to enhance document summarization. However, the result on the MS MARCO dataset, as shown in the table below, shows the limitations of this approach:\n\n| Reward Function | Hit@1     | Hit@5     | Hit@10    | MRR@10    |\n| --------------- | --------- | --------- | --------- | --------- |\n| MRR             | 26.21     | 54.03     | 60.28     | 37.43     |\n| RM              | **33.23** | **64.48** | **75.80** | **46.62** |\n\nWe've noticed that using MRR as the reward function resulted in a notable drop in performance. We deduced two primary reasons for this observed performance. Firstly, when the sampled docids does not include the ground-truth docid, the computation of MRR becomes unfeasible, thereby hindering effective model optimization. Secondly, employing MRR as the reward function presents limitations in that it only yields a single reward value for an entire list, failing to facilitate granular optimization of each token involved in the docid generation process.\n\nTherefore, based on these empirical observations and the identified limitations, we concluded that directly using ranking metrics like MRR may not be the most conducive to achieving optimal performance in generative retrieval tasks. To overcome these limitations, we decided to opt for a custom reward model (RM) that is capable of providing a more targeted and informative reward signal (relevance feedback). The robustness of this approach is substantiated by the improved performance metrics, as evidenced in our experiments.\n\nWe hope these explanations clarify why we didn't use ranking metrics like MRR as direct rewards and that this will address any concerns you may have had on this aspect.\n\n[1] Yixin Liu, Pengfei Liu, Dragomir R. Radev, Graham Neubig. BRIO: Bringing Order to Abstractive Summarization. In ACL 2022.\n\n#### In Response to Question B:\n\nReciprocal Rank Fusion and Borda Rank are both methods used in information retrieval and ranking scenarios. Upon your suggestion, we conducted experiments to compare these two algorithm. Here are the experimental results:\n\n| Rank Fusion            | Hit@1     | Hit@5     | Hit@10    | MRR@10    |\n| ---------------------- | --------- | --------- | --------- | --------- |\n| Reciprocal rank fusion | **33.36** | 64.45     | 75.61     | 46.54     |\n| Borda Rank             | 33.23     | **64.48** | **75.80** | **46.62** |\n\nThe findings reveal comparable efficacy of both methods. While Reciprocal Rank Fusion exhibits a slightly higher score for the Hit@1 metric, Borda Rank outperforms or matches Reciprocal Rank Fusion in the Hit@5, Hit@10, and MRR@10 metrics. The results suggest that both methods are quite effective, and the choice between them may depend on the specific performance metrics that are of interest for a particular application. For instance, if the focus is on retrieving the most relevant document (Hit@1), Reciprocal Rank Fusion might be the preferred choice. On the other hand, if a balanced performance across different rank positions is desirable, Borda Rank appears to be more effective.\n\nWe are genuinely grateful for the alternative suggestions you have presented. These suggestions offer valuable avenues for exploration and adaptation.\n\n#### For Question C:\n\nOur choice to initially utilize the pairwise loss was influenced by observations from InstructGPT and the approach delineated in \"Learning to Summarize from Human Feedback.\" In addition to the pairwise loss, we also explored the use of pointwise and listwise distillation losses in reward model training. The challenge is that it becomes less straightforward to extract score distributions for individual documents (pointwise) or the entire list (listwise). Thus, to make it feasible to apply pointwise and listwise training schemes, we made a delineation: the topmost 25% of documents in the list were categorized as relevant and the remaining 75% as irrelevant. The results from our experiments comparing the different RM training losses are tabulated below:\n\n| RM Loss   | Hit@1     | Hit@5     | Hit@10    | MRR@10    |\n| --------- | --------- | --------- | --------- | --------- |\n| pointwise | 29.01     | 59.92     | 66.83     | 41.70     |\n| pairwise  | **33.23** | **64.48** | **75.80** | **46.62** |\n| listwise  | 31.26     | 62.01     | 70.68     | 43.89     |\n\nUpon examination, the pairwise loss  outperforms both the pointwise and listwise losses across all evaluated metrics. This merit may be due to its ability to capture nuanced differentiations between document pairs. In contrast, the pointwise and listwise methods exhibited relatively weaker performance, possibly because of biases from the binary relevant/irrelevant annotations. In light of the results, the pairwise loss emerges as the most effective strategy among the three in the context of our reward model training. Therefore, we adopted the pairwise loss in our model. \n\nWe sincerely value your attention to these details and hope that our experiments and explanations provide clarity on our decision-making process and the outcomes achieved. \n\n### Explanations On the Document Representations:\n\nIn generative retrieval, mainstream document identifiers (docid) can be categorized into two major types: ID-based and keyword-based docids. We have chosen the keyword-based docid (comprising URL and summary) due to its capacity to leverage the strengths of pre-trained language models effectively and offer enhanced interpretability. \n\nRegarding your concerns on document representations, we further conducted experiments using semantic id as document representations. The table below shows our findings:\n\n| Model     | Document Rep. | Hit@1 | Hit@5 | Hit@10 | MRR@10 |\n| --------- | ------------- | ----- | ----- | ------ | ------ |\n| SFT (NCI) | semantic id   | 29.54 | 57.99 | 67.28  | 40.46  |\n| GenRRL    | semantic id   | 30.61 | 59.02 | 68.78  | 42.02  |\n\nThe results demonstrate that our method consistently enhances model performance on ID-based docid. However, the degree of improvement is comparatively less pronounced than that achieved with keyword-based docids. This discrepancy might be attributed to the fact that keyword-based docids align more closely with the semantic space of the reward model. As a result, the feedback provided by the reward model can exert a more refined influence on the docid generator.\n\nLastly, we'd like to acknowledge your suggestion concerning the inclusion of a cross-attention ranker in our model. This suggestion adds an intriguing layer to the conversation, and we plan to investigate the applicability and efficacy of a cross-attention ranker in future iterations of our project.\n\nWe are hopeful that these explanations will fortify the paper's standing. Thank you again for your invaluable feedback and for helping us enhance the quality of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Good observation on potential weakness of existing generative retrievers\n- The paper studies different RL strategies to optimize the generative retriever\n- Relatively thorough experiments on multiple data sets illustrating the effectiveness of RL\n\n", "reasons_to_reject": "\n- A more fair comparison between baselines like DSI-QG etc. using the same doc_id strategy might better justify the effectiveness of RL\n- Several steps in reward model training seem arbitrary and lack further explanations", "questions_to_authors": "Regarding the reward model development: \n\nQuestion A: Is directly using the ranking metric like MRR or NDCG as the reward a bad idea? If so, why? Perhaps the authors could provide some experiments to show that this approach does not work well.\n\nQuestion B: An alternative to Borda Rank is reciprocal rank fusion. Does this work as well as Borda Rank? The authors could provide some experiments to compare the two approaches.\n\nQuestion C: In addition to pairwise loss, does pointwise/listwise distillation loss also work in *reward model training*? The authors could provide some experiments to compare the performance of these different loss functions.\n\nI believe that providing this additional information would make the paper stronger and more convincing.", "paper_topic_and_main_contributions": "This paper proposes a reinforcement learning-based approach to training differentiable search index (DSI). The authors first develop a relevance reward model, which is distilled from an ensemble of different relevance models. This reward model is then used to train DSI using reinforcement learning with three optimization strategies: pointwise, pairwise, and listwise strategies. Experiments on two datasets show that the proposed approach, GenRRL, outperforms the baseline methods.\n\nThe paper is overall interesting as it explores a novel direction to combine ranking losses in learning-to-rank with generative retrievers using reinforcement learning. The authors select to create the reward model by distilling from an ensemble of multiple off-the-shelf relevance annotators. I do have questions regarding some technical choices in the development of the rewarding model (listed below). \n\nThe experiments were conducted on subsets of MS MARCO Document Ranking and NQ. I find the improvement from SFT to GenRRL encouraging. The only concern is that there seem to be some differences between document representations of SFT/GenRRL and other baselines. Perhaps a fairer comparison would be to use the same document representation. However, I do not think this would be a reason to reject the paper.\n\nA minor comment is to consider using a cross-attention ranker to replace DPR in the relevance annotators for reward model development, such as monoT5 [1] or RankT5 [2], which is shown to be effective in the RocketQA paper [3].\n\n[1] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document Ranking with a Pretrained Sequence-to-Sequence Model. In EMNLP.\n\n[2] Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses. In SIGIR.\n\n[3] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. In NAACL.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Minor comments:\n- Is there a missing negative sign in Line 340?\n- Report the results of the 3 relevance annotators and their ensemble.\n"}, "extra_text": {}}, {"review_id": "rLdSGFzNzc", "reviewer": "Reviewer_q2Hf", "summary": "", "questions": "1. Why the performance of DPR and ANCE are so low? \n2. Can you elaborate more on the advantages of generative retrieval? The reasons from line 42 to 45 are not convincing to me. For example, why differentiable search index is advantageous compared to conventional explicit index? I think explicit index is more explainable, easier to include new documents.  Why end-to-end document generative model is prefer to, e.g. Dense Retrieval?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our motivation and empirical conclusions, and we are grateful for your thoughtful comments and inquiries regarding our work. In response to your inquiries, we offer the following clarifications. \n\n**Performance of DPR and ANCE**:\n\nIn our study, both the reproduction of DPR and ANCE methods was based on their official codebase. The differences between our results and the original DPR paper primarily arise from two key factors:\n\n1. **Dataset Disparity**: We employed the NQ320k dataset following [1], a dataset tailored for generative retrieval based on NQ, comprising 307k training samples and 7.8k test samples. Conversely, the original DPR paper used the Natural Questions dataset where documents were segmented into multiple passages, including 58k training samples and 3.6k test samples. \n2. **Difference in Evaluation Methodology**: The DPR original paper considered retrieval results as correct if the retrieved documents contained the answer, allowing for multiple versions of wiki pages to potentially hold the correct answers. In contrast, our generative retrieval evaluation mandates the generated docid to be an exact match with the unique ground-truth document. \n\nThese two factors collectively contribute to the observed result disparities. We apologize for any misunderstanding that may have arisen. All our experiments were conducted under identical settings to ensure the fairness and comparability of the results. Thank you for pointing out this issue; if given the opportunity, we will elucidate it further in the camera-ready version.\n\n[1] Yi Tay, et al. Transformer Memory as a Differentiable Search Index. in NeurIPS 2022\n\n**Advantages of Generative Retrieval**:\n\nGenerative retrieval is an emerging approach to information retrieval that contrasts with traditional retrieval methods. While both aim to retrieve relevant information from a large corpus, they achieve this through fundamentally different mechanisms. Here are some advantages of generative retrieval over traditional retrieval methods:\n\n1. **Reduce storage overhead**: Tradition index-based retrieval needs a large document index to search over the given corpus, which requires considerable memory resources to store all the data. In contrast, generative retrieval efficiently compresses extensive bodies of knowledge into the model's parameters, eliminating the need for large-scale disk storage of indexes. This mirrors the compactness and efficiency observed in the human brain, which can store vast amounts of information without taking up too much space.\n\n2. **End-to-end optimization**: Traditional IR approaches are based on pipelined index-retrieval-rank framework, which is used in either dense and sparse retrieval models. Such a framework limits the optimization of the ranking results in an end-to-end way. End-to-end generative retrieval utilize a consolidated generative model, which is aware of the knowledge of all individual documents in the corpus, can be optimized end-to-end and increases the fitness of the model to the document retrieval task.\n\n3. **Seamless Incorporation with LLMs**: As large language models evolve, they can be smoothly integrated into the generative retrieval framework, serving as its core. The exploration of generative retrieval offers a promising step towards advancing LLM for Information Retrieval (LLM4IR).\n\n4. **Better Handling of Long-tail and Rare Queries**: Traditional dense retrieval methods might struggle with rare or long-tail queries since they rely on fixed vector representations. End-to-end generative retrieval models, due to their generative nature and powerful zero-shot capability of LM, can potentially generate relevant responses even for queries they haven't been explicitly trained on. This makes them more adaptive and capable of handling a wider range of search queries.\n\nIn sum, while generative retrieval does have challenges, its unique advantages position it as a promising direction for future research in information retrieval. This paper focuses on aligning training objectives of generative retrieval with document relevance estimation, which combines the advantages of both generative and dense/sparse retrieval. Recently, some studies have focused on challenges such as incorporating new documents [1] and applying to larger corpora [2]. These efforts underscore the evolving nature of generative retrieval techniques. Moreover, we anticipate that the continuous advancement of LLM will play a pivotal role in further enriching the landscape of generative retrieval.\n\n[1] Sanket Vaibhav Mehta, et al. DSI++: Updating Transformer Memory with New Documents. Arxiv\n\n[2] Ronak Pradeep, et al. How Does Generative Retrieval Scale to Millions of Passages? Arxiv\n\n**Necessity of Reward Model**:\n\nOur initial strategy was indeed direct usage of combined scores from BM25, DPR, and LLAMA as rewards for reinforcement learning. However, when we weighted and summed the scores from the three annotators, we observed a decrease in performance. We think this was due to discrepancies in the scoring ranges and distributions across different model types, resulting in less-than-ideal reinforcement learning results. We did attempt normalization to address this, which yielded some improvements, but the gains remained minimal:\n\n| Reward Function              | Hit@1     | Hit@5     | Hit@10    | MRR@10    |\n| ---------------------------- | --------- | --------- | --------- | --------- |\n| Combined scores              | 29.72     | 59.93     | 68.02     | 42.52     |\n| Combined scores (normalized) | 30.82     | 61.46     | 70.28     | 43.27     |\n| Reward Model                 | **33.23** | **64.48** | **75.80** | **46.62** |\n\nTo resolve this, we drew inspiration from RLHF's method of integrating multiple human annotator annotations. Instead of relying on raw scores, we asked each relevance annotator to produce a ranking. We then consolidated these rankings to generate pairwise labels to train our reward model. This approach effectively circumvented the issues from inconsistent scoring scales and biases across different models, providing a more comprehensive relevance feedback. More details regarding reward model can be referred to our response to the second reviewer (pWtF).\n\n**Model Complexity**:\n\nThe primary contribution of this work lies in the proposal of a reinforcement learning framework tailored for generative retrieval. Under this framework, the complexity of model training can be adjusted according to specific requirements. For instance, in the annotator selection aspect, it allows for the integration of various existing models as black-box annotators (such as MonoBERT, RankT5), eliminating the need for training from scratch. During the model inference stage, invoking the docid generator alone is adequate, which only requires 0.04s of decoding top-10 docids for a query (comparable with sparse retrieval).\n\n------\n\nWe hope these clarifications address your concerns and provide a clearer perspective on our work's intentions and methodologies. We are grateful for your insights and feedback,which have been instrumental in refining our understanding and presentation of the research. \n\nThank you once again for the time and effort you have invested in reviewing our manuscript and we are eager to address any additional concerns you may have. Your constructive feedback is invaluable in guiding the refinement of our research, and we look forward to possibly contributing a stronger and more refined piece to the conference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well motivated.\n2. The paper shows that the proposed method is better than other generative retrieval baselines.\n3. The writing is generally clear and easy to follow.", "reasons_to_reject": "1. More explanation is needed to justify the reason for training the reward model.  Unlike RLHF in which the reward model is trained to mimic human preference which is unknown, here, we can directly use the combined scores of BM25, DPR or LLAMA as reward for Reinforcement Learning. I am wondering why do we need to train the reward model instead?\n2. I have doubt on the experimental results as the performance of DPR and ANCE on Natural Questions are much lower than previously reported. For example, DPR can achieve Hit@1 of more than 40% (Table 5 in [1]), which is much higher than the reported result of ~23% reported in this paper. In addition, the author should cite the papers where the baseline results are reported if they are not reproduced by the authors.\n3. The model is complicated, which requires the training of a dense model (DPR). \n\n[1] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.", "questions_to_authors": "1. Why the performance of DPR and ANCE are so low? \n2. Can you elaborate more on the advantages of generative retrieval? The reasons from line 42 to 45 are not convincing to me. For example, why differentiable search index is advantageous compared to conventional explicit index? I think explicit index is more explainable, easier to include new documents.  Why end-to-end document generative model is prefer to, e.g. Dense Retrieval?", "paper_topic_and_main_contributions": "The paper considers the problem of end-to-end generative retrieval, and targets two issues: 1) the discrepancy between token-level probabilistic optimization and the broader document-level relevance estimation; 2) an overemphasis on top-1 results. The authors then propose to solve these problems using Reinforcement Learning with Relevance Feedbacks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Updated: I would like to thank the authors for addressing my concerns, and updated the scores correspondingly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper solves generative retrieval with a new reinforcement learning-based method, which is equipped with different reward models. Experiment results validate the effectiveness of the proposed method. The rebuttal has addressed most concerns of reviewers.
Decision: Accept-Main