Generative Retrieval Meets Multi-Graded Relevance

Authors: yubao  tang
ruqing  zhang
jiafeng  guo
maarten  de rijke
wei  chen
xueqi  cheng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 07:47:00

Abstract: Generative retrieval represents a novel approach to information retrieval, utilizing an encoder-decoder architecture to directly produce relevant document identifiers (docids) for queries. While this method offers benefits, current implementations are limited to scenarios with binary relevance data, overlooking the potential for documents to have multi-graded relevance. Extending generative retrieval to accommodate multi-graded relevance poses challenges, including the need to reconcile likelihood probabilities for docid pairs and the possibility of multiple relevant documents sharing the same identifier. To address these challenges, we introduce a new framework called GRaded Generative Retrieval (GR$^2$). Our approach focuses on two key components: ensuring relevant and distinct identifiers, and implementing multi-graded constrained contrastive training. Firstly, we aim to create identifiers that are both semantically relevant and sufficiently distinct to represent individual documents effectively. This is achieved by jointly optimizing the relevance and distinctness of docids through a combination of docid generation and autoencoder models. Secondly, we incorporate information about the relationship between relevance grades to guide the training process. Specifically, we leverage a constrained contrastive training strategy to bring the representations of queries and the identifiers of their relevant documents closer together, based on their respective relevance grades.Extensive experiments on datasets with both multi-graded and binary relevance demonstrate the effectiveness of our method.

Main paper: 
 introduction: 
 Generative retrieval (GR) [51] is a new paradigm for information retrieval (IR), where all information in a corpus is encoded into the model parameters and a ranked list is directly produced based on a single parametric model. In essence, a sequence-to-sequence (Seq2Seq) encoder-decoder architecture is used to directly predict identifiers (docids) of documents that are relevant to a given query. Recent studies have achieved impressive retrieval performance on many search tasks [15, 53, 67, 92] .Current work on GR mainly focuses on binary relevance scenarios, where a binary division into relevant and irrelevant categories is assumed [20, 52, 83] , and a query is usually labeled with a single relevant document [38, 57] or multiple relevant documents that have the same relevance grade [33] . The standard Seq2Seq objective, via maximizing likelihood estimation (MLE) of the output sequence with teacher forcing, has been used extensively in GR due to its simplicity. However, in real-world search scenarios, documents may have different degrees of relevance [18, 71, 72, 82, 91] as binary relevance may not be sufficiently represent fine-grained relevance. In traditional learning-to-rank (LTR), multi-graded relevance judgments [23, 69] are widely considered, with nDCG [35] and ERR [12] being particularly popular. In modeling multi-graded relevance in LTR, a popular approach is the pairwise method [10, 84] which involves weighting different documents based on their relevance grades and predicting the relative order of a document pair.Compared to common LTR algorithms, the learning objective currently being used in GR differs significantly: the standard Seq2Seq objective emphasizes one-to-one associations between queries and docids, aiming to generate a single most relevant docid. Inspired by pairwise methods in LTR, a straightforward approach to extending GR to multiple grades, involves having the GR model generate the likelihood of docids with higher relevance grades being greater than that of lower relevance grades. The docid likelihood is the product of the likelihoods of each token in the generated docid. Docids commonly exhibit distinct lengths, as a fixed length might not adequately encompass diverse document semantics. However, the variation in docid lengths within the corpus may lead to smaller likelihood scores for longer docids. Although some GR work [44, 78, 92, 93, 102] use a pairwise or listwise loss for optimization, they still only consider binary relevance or require complex multistage optimization. Besides, essential topics in multi-graded relevant documents may be similar, emphasizing the need for a one-to-one correspondence between document content and its identifier to ensure distinctness. Consequently, harnessing a GR model's capabilities for multi-graded relevance ranking in a relatively succinct manner remains an non-trivial challenge.To this end, we consider multi-graded generative retrieval and propose a novel GRaded Generative Retrieval (GR 2 ) framework, with three key features:(1) To enhance docid distinctness while ensuring its relevance to document semantics, we introduce a regularized fusion approach with two modules: (i) a docid generation module, that produces pseudo-queries based on the original documents as the docids; and (ii) an autoencoder module, that reconstructs the target docids from their corresponding representations. We train them jointly to ensure that the docid representation is close to its corresponding document representation while far from other docid representations. (2) For the mapping from a query to its relevant docids, we design a multi-graded constrained contrastive (MGCC) loss to capture the relationships between labels with different relevance grades. Considering the incomparability of likelihood probabilities associated with docids of varying lengths, we convert queries and docids into representations within the embedding space. The core idea is to pull the representation of a given query in the embedding space towards those of its relevant docids, while simultaneously pushing it away from representations of irrelevant docids in the mini-batch. To maintain the order between relevance grades in the embedding space, the strength of the pull is determined by the relevance grades of the docids. The distinction between MGCC and pairwise methods in LTR [9, 10, 84] lies in proposing more specific grade penalties and constraints to regulate the relative distances between query representations and docid representations of different grades. (3) We explore two learning scenarios, i.e., supervised learning and pre-training, to learn generative retrieval models using our GR 2 framework. Importantly, our method for obtaining docids is applicable to both multi-graded and binary relevance data, and it can reduce to the supervised contrastive approach in binary relevance scenarios.Our main contributions are: (i) We introduce a general GR 2 framework for both binary and multigraded relevance scenarios, by designing relevant and distinct docids and using the information about the relationship between labels. (ii) Through experiments on 5 representative document retrieval datasets, GR 2 achieves 14% relative significant improvements for P@20 on Gov 500K dataset over the SOTA GR baseline RIPOR [85] . (iii) Even in low-resource scenarios, our method performs well, surpassing BM25 on two datasets. On large-scale datasets, it achieves comparable results to RIPOR.for a query. Inspired by this blueprint, there have been several proposals [7, 14, 22, 40, 43, 79, 85] to learn a Seq2Seq model by simultaneously addressing the two key issues below.Key issue 1: Building associations between documents and docids. The widely-used docid designs are pre-defined and learnable docids [77] . Pre-defined docids are fixed during training, e.g., document titles [22] , semantically structured strings [56, 79, 85] , n-grams [7, 14, 44] , pseudo-queries [76] , URLs [68, 102, 104] , product quantization code [13, 104] . Learnable docids are tailored to retrieval tasks, e.g., discrete numbers [74] , important word sets [86, 96, 97] and residual quantization code [92, 93] . Though effective in some tasks, these docid designs have limitations. Titles and URLs rely on metadata, while n-grams demand storage of all n-grams. Quantization codes lack interpretability.Learning optimal learnable docids is challenging, involving a complex learning process. Considering performance, implementation complexity, and storage requirements, the pre-defined pseudo-query is a promising compromise choice. Unfortunately, semantically similar documents might have similar docids or even repetitions, making it challenging for the GR model to distinguish them in the both binary and multi-graded relevance scenarios. To generate diverse and relevant docids, we propose a docid fusion approach based on pseudo-queries.Key issue 2: Mapping queries to relevant docids. Given a query, a GR model takes as input a query and outputs its relevant docids by maximizing the output sequence likelihood, which is only suitable for binary relevance. If a query has only one relevant document, it is paired with its relevant docid. If a query has multiple relevant documents at the same grade, it is paired with multiple relevant docids.The relative order of relevant docids in the returned list is random. Such a learning objective cannot handle search tasks with multi-graded relevance, which limits its efficacy for general IR problems. While certain GR studies [44, 92, 93, 102] employ a pairwise or listwise loss for optimization, they remain limited to binary relevance or necessitate intricate multi-stage optimization processes. In this work, we use all available relevance labels to enable multi-graded GR. For more related work, please refer to Appendix C.3 Preliminaries Document retrieval. Assume that L q = [1, . . . , l, . . . , L] is the grade set representing different degrees of relevance. We assume that there exists a total order between the grades l > l -1> • • • > 1, ∀l ∈ L q .Let q be a query from the query set Q, and D q = {d 1 , d 2 , . . . , d N } be the set of N relevant documents for q, which are selected from the large document collection D. D Q is the relevant document set for Q. We write D l q for the documents with grade l for q. The document retrieval task is to find a retrieval model f to produce the ranked list of relevant documents for the given query, i.e., π f (q) := [arg max (1) d f (q, d), arg max (2) d f (q, d), . . . ], where arg max (i) d f (q, d) denotes the iranked document d for q over D given by f via matching the query and documents. The model f is optimized by minimizing its loss function over some labeled datasets, i.e., min f Q D L(f ; q, D q ). Multi-graded generative retrieval. In an end-to-end architecture, the GR process directly returns a ranked list for a given query, without a physical index component. Assume that the indexing mechanism to represent docids is I : D → I D , where I D is the corresponding docid set. For the observed relevant document set D q , q ∈ Q, the indexing mechanism I maps them to the docid set I Dq = {I D l q | l = 1, . . . , L}, in which the docid id l ∈ I D l q is at grade l. The GR model observes pairs of a query and docid with multi-graded relevance labels under the indexing mechanism I, i.e., {Q,I D Q }, where I D Q denotes {I D l q | l = 1, . . . , L, q ∈ Q}.Given Q, the model g : Q → I D Q autoregressively generates a ranked list of candidate docids in descending order of output likelihood conditioned on each query. Mathematically, g(q; θ) = P θ (id | q) = t∈ [1,|id|] p θ (w t | q, w <t ), where w t is the t-th token in the docid id ∈ I D and w <t represents all tokens before the t-th token in id. θ is the model parameters. During inference, the GR model produces the ranked docid list via π g (q) := [g (1) (q), g (2) (q), . . . ] = [arg max (1) id P θ (id | q), arg max (2) id P θ (id | q), . . . ], where arg max (i) id P θ (id | q) denotes id for q whose generation likelihood is ranked at position i.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Abstract and Section 1Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Section 6Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
methodology: 
 In this section, we develop a general GR framework called GR 2 to support multi-graded relevance learning. We address two main challenges: (i) how to generate relevant and distinct identifiers given the original documents (Section 4.1), and (ii) how to capture the interrelationship between docids in a ranking for a query (Section 4.2). Next, we introduce the optimization process (Section 4.3).A popular docid representation method is to employ a query generation (QG) technique [60] to generate a pseudo-query conditioned on the document as the docid [43, 76] . It is common for different documents to share identical or similar docids when they contain similar information [7, 14, 76] . While this similarity can aid the GR model in recognizing the likeness, it also poses a challenge when the GR model needs to differentiate among multiple documents with varying relevance grades to a query. Therefore, as shown in Figure 5 in Appendix A, we propose a regularized fusion approach to optimize the trade-off between relevance and distinctness in docids.The key idea is to jointly optimize the relevance and distinctness that fuses the latent space of a QG model, i.e., a docid generation model, and that of an autoencoder (AE) model. The AE model is used to reconstruct the target query, and both models are based on an encoder-decoder architecture. We share the same decoder for both QG and AE models as in [29, 46] . Specifically, we propose two simple yet effective auxiliary regularization terms, i.e., a relevance term and a distinctness term.Relevance regularization term. To improve the relevance [36, 90] , we encourage the representation of a document and that of the corresponding docid (i.e., pseudo-query) to be close to each other in the shared latent space. We also aim to increase the distance between the representation of a document and that of irrelevant docids associated with other documents. This term is formalized as:EQUATIONwhere ζ = d∈D Q ,q∈Q,q̸ =q exp(sim(e d QG , e q AE )). For each query-document pair, e d QG is the document representation obtained by the encoder of the QG model, and e q AE is the query representation obtained by the encoder of the AE model; q is one of the remaining queries except for q in the batch, and the batch size is |Q|; sim(•, •) is the dot-product function; and θ QG and θ AE are model parameters of the QG model and AE model, respectively.Distinctness regularization term. To enhance the distinctness between documents and between docids, we push away the representations of different documents in the document space and, simultaneously, push away the representations of different docids in the docid space. Additionally, to establish a connection between the two latent spaces of docids and documents, we ensure that the representation of a document and its corresponding docid are close in the same latent space. In a batch, the distinctness regularization term L Div (Q, D Q ; θ QG , θ AE ) is formalized as:EQUATIONwhere d is an irrelevant document with respect to q in the batch. We include a discussion on the difference between the two regularization terms in Appendix B.Jointly training the QG and AE model. Both models use MLE to optimize their targets based on their inputs. Therefore, the overall optimization objectiveEQUATIONwhere L QG M LE (•) = -q∈Q,d∈D Q log P θ QG (q|d), and L AE M LE (•) = -q∈Q log P θ AE (q|e q AE ). P θ QG (q|d) and P θ AE (q|e q AE ) denote the output query likelihood conditioned on the document, and e q AE . α and β are hyperparameters. Relevant and distinct docid generation. During inference, following [29, 100] , we sample different latent vectors to generate docids. Given a document, we obtain its representation e d QG by the QG model's encoder. We introduce a random vector r that is uniformly sampled from a hypersphere of radius |r| centered at e d QG , denoted as z d,r = e d QG + r. The value of |r| is tuned on the validation set to optimize the trade-off between relevance and distinctness. z d,r is then used as the initial state for the decoder of QG model. We subsequently generate a list of pseudo-queries using beam search decoding. Initially, we choose the top 1 pseudo-query as the docid. If there are still duplicate docids, we select subsequent pseudo-queries from the list to replace these duplicates until all docids in the corpus are unique. According to our experimental analysis, selecting up to the top 2 can ensure docid uniqueness in the corpus. In this way, we first generate diverse docids id relevant to the original text, serving as the basis for GR model learning. Considering learning costs, the models for docid design and the GR model are distinct. Though not end-to-end, These fixed docids can guide the GR model towards appropriate optimization, whereas joint optimization increases the learning difficulty.After obtaining docids, we introduce the multi-graded constrained contrastive (MGCC) loss for the GR model. Positive pairs and negative pairs are constructed by pairing each query with its relevant docids drawn from all grades, and with all docids relevant to the rest of queries in the mini-batch except it, respectively. As illustrated in Figure 1 , the key idea is to force positive pairs closer together in the representation space, but the magnitude of the force is dependent on the relevance grade. The MGCC loss includes a grade penalty and constraint. Grade penalty. To distinguish between multiple positive pairs, our key idea is to apply higher penalties to positive pairs constructed from higher grades, forcing them closer than negative pairs constructed from lower grades. We first define the loss L Pair (q, id l ; θ) between a query q and its relevant docid at grade l, as L Pair (q, id l ; θ) = log exp(sim(f q , f l id )/τ )EQUATIONwhere A q includes all positive query-docid pairs at different grades and other negative query-docid pairs for q. f q and f l id denote the representation of q and id l , respectively. They are computed based on the encoder and decoder hidden states, respectively, i.e.,EQUATIONEQUATIONEQUATION)where ξ is the composition of affine transformation with the ReLU [54] and average pooling.H l = [h l 1 , . . . , h l |id l |] is a concatenation of the decoder hidden states of id l . M q = [m q 1 , . . . , m q |q| ] is the concatenation of the hidden representations generated by the encoder of q. In this way, the loss for Q is: q∈Q1 L L l=1 -λ l |I D l q | id l ∈I D l qL Pair (q, id l ; θ), where id l ∈ I D l q is a docid at relevance grade l for q; λ l is a controlling parameter that applies a fixed penalty for each grade, contributing to preserving the relevance level explicitly. Grade constraint. Inspired by the hierarchical constraint in classification [30, 98] , where a class higher in the hierarchy cannot have a lower confidence score than a class lower in the ancestry sequence, for each q, we propose to enforce a grade constraint L M ax , i.e., the maximum loss from all positive pairs at grade l:EQUATIONThe loss between query-docid pairs constructed from a higher relevance grade will never be higher than that constructed from a lower relevance grade. The final MGCC lossEQUATIONFor binary relevance datasets, i.e., where there is only a single level of relevance labels (i.e., L = 1), the MGCC loss reduces to the supervised contrastive loss [16] , which helps force the representation of the query close to that of its relevant documents, while far away from other irrelevant documents.In this way, GR 2 can also tackle the GR problem for the scenarios with binary relevance data, and this notably reduces complexity compared to multi-graded relevance.Supervised learning. Based on the docids, we directly supervise the GR model with L MGCC , and we denote this version as GR 2S . To index all documents in a corpus, we adopt the MLE loss to learn document-docid pairs [79] . To guarantee the generation of each relevant docid to a query, we adopt the MLE for query-docid pairs at different grades. The final supervised learning loss is:EQUATIONEQUATIONandL d MLE (D, I D ; θ) = -d∈D log P θ (id | d),where γ is a hyperparameter, P θ (id l | q) and P θ (id | d) denote the output docid likelihood conditioned on the query and document, respectively.The learning objective currently being used in GR is usually defined asL q MLE (Q, I D Q ; θ) + L d MLE (D, I D ; θ), which does not capture the relationships between labels. Pre-training and fine-tuning. We also explore the use of GR 2 in a pre-training scenario. To construct pre-training data, we use the English Wikipedia [87] to build a set of pseudo-pairs of queries and docids. We use the unique titles of Wikipedia articles as the docids for pre-training and assume that a random sentence in the abstract can be viewed as a representative query of the article.Then, for each query, we construct its relevant documents with 4 relevance grades as follows, and leave other grades as future work: (i) grade 4: the Wikipedia article from which the query is sampled, is regarded as the most relevant document. (ii) grade 3: We use the See Also section of a Wikipedia article in which hyperlinks link to other articles with similar or comparable information, which is mainly written manually. If there exists no See Also section, we use a similar section, i.e., the Reference section. (iii) grade 2 and grade 1: Besides the See Also section, some hyperlinks link to pages that describe the concept of some entities in detail. We randomly sample several anchor texts from the first section and other sections, respectively, and regard the linked target pages as grade 2 and grade 1 relevant documents, respectively.In this way, a total of 1,180,131 query-docid pairs are obtained, and we pre-train an encoder-decoder architecture using L total as defined in Eq. (10) . The architecture can be fined-tuned for downstream retrieval tasks using L total , where docids are obtained via the fusion method. We denote this version as GR 2P . In the future, we could explore using large language models to automatically label data. 
experimental settings: 
 Datasets and evaluation metrics. We select three widely-used multi-graded relevance datasets: Gov2 [18] , ClueWeb09-B [19] and Robust04 [82] . And we use the classic normalized discounted cumulative gain (nDCG@{5, 20}), expected reciprocal rank (ERR@20) and precision (P@20) as metrics [12, 31, 49] . Furthermore, we consider two binary relevance datasets: MS MARCO Document Ranking [57] and Natural Questions (NQ 320K) [38] . We take mean reciprocal rank (MRR@{3, 20}) and hit ratio (Hits@{1, 10}) as metrics following [7, 79, 85, 104] . Following existing Table 1 : Experimental results on datasets with multi-graded relevance. Results denoted with ⋆ are from [34, 55] . And * , †, ‡ and ≀ indicate statistically significant improvements over the best performing SR baseline QLM, the DR baseline PseudoQ, the GR baseline RIPOR, and all the baselines, respectively (p ≤ 0.05).ClueWeb 500K Robust04 nDCG P ERR nDCG P ERR nDCG P ERR @5 @20 @20 @20 @5 @20 @20 @20 @5 @20 @20 @20 Baselines. We consider three types of baselines: sparse retrieval (SR), dense retrieval (DR), and GR models. The SR baselines include: BM25 [70] , DocT5Query [28] , Query Likelihood Model (QLM) [105] , and SPLADE [24, 25] . The DR baselines include: RepBERT [95] , DPR [36] , PseudoQ [75] , and ANCE [89] . The GR baselines are DSI-Num [79] , DSI-Sem [79] , DSI-QG [106] , NCI [85] , SEAL [7] , GENRE [22] , Ultron-PQ [104] , LTRGR [44] , GenRRL [102] , GenRet [74] , NOVO [86] , and RIPOR [92] . Additionally, we compare our method with a full-ranking method, monoBERT [61] .BM25For a detailed description of the baselines, please refer to Appendix E.Model variants. We consider two versions of GR 2 : GR 2S and GR 2P , for supervised learning and pretraining, respectively. Additional variants are: (i) GR 2S -RF and GR 2P -RF omit the regularized fusion approach, and directly use pseudo-queries generated by a single QG model as docids. (ii) GR 2S -λ and GR 2P-λ omit the grade penalty λ l in L MGCC (Eq. ( 9)); (iii) GR 2S -Max and GR 2P -Max omit L Max in the MGCC loss; (iv) GR 2S MLE and GR 2P MLE only use L q MLE (Eq. ( 11)) andL d MLE ; (v) GR 2S CE and GR 2PCE use the weighted cross-entropy loss, where the relevance grades are the weights; it can be viewed as an adaption of the loss from [8, 9] ; (vi) GR 2S LR and GR 2P LR directly use the LambdaRank loss [10] . Implementation details. For backbones, we choose the widely-used backbone in GR research, i.e., T5-base model [66] to implement the GR 2 and GR baselines. For docid generation, we use the docT5query model [59] as the QG model and a transformer autoencoder [80] . For T5-base, the hidden size is 768, the feed-forward layer size is 12, the number of self-attention heads is 12, and the number of transformer layers is 12. GR 2 and the reproduced baselines are implemented with PyTorch 1.9.0 and HuggingFace transformers 4.16.2; we re-implement DSI-Num and DSI-Sem, and utilize open-sourced code for other baselines.For hyperparameters, we use the Adam optimizer with a linear warm-up over the first 10% steps. The learning rate is 5e-5, label smoothing is 0.1, weight decay is 0.01, sequence length of documents is Table 2 : Experimental results on datasets with binary relevance. And * , †, ‡ and ≀ indicate statistically significant improvements over the best performing SR baseline DocT5query or SPLADE, the DR baseline PseudoQ, the GR baseline RIPOR and all the baselines, respectively (p ≤ 0.05).NQ 320K MRR Hits MRR Hits @3 @20 @1 @10 @3 @20 @1 @10 BM25Note, Appendix G contains additional experimental analyses, i.e., comparisons with full-ranking baselines (Appendix G.1) and results on large-scale datasets (Appendix G.3).Results on multi-graded relevance. Table 1 shows the performance of GR 2 and baselines on multi-graded relevance datasets. We find that: (i) QLM performs the best among sparse retrieval and dense retrieval baselines on Robust04 and Gov 500K, confirming prior work [45, 48, 94] ; these multi-graded datasets have limited labeled training pairs, which may not be sufficient for learning semantic relationships between queries and documents. (ii) Existing GR baselines perform worse than QLM on Gov 500K and Robust04, indicating that developing an effective GR method remains an open challenge. (iii) RIPOR outperforms other GR baselines; the multi-stage training strategy appears to aid the effectiveness. (iv) By capturing the relationship between multi-graded relevance labels, GR 2 achieves significant improvements over GR baselines that only consider binary relevance based on the standard Seq2Seq objective. For example, GR 2P and GR 2S outperform RIPOR by about 11% and 14% on the Gov 500K dataset in terms of P@20, respectively. (v) Between our two methods, GR 2P outperforms GR 2S , indicating that pre-training on large-scale elaborately constructed multi-graded relevance data, is better than training a single generation model from scratch.Results on binary relevance. The performance on binary relevance datasets is shown in Table 2 . We find that the relative order of different models on these datasets is almost consistent with that on the multi-graded relevance data. (i) PseudoQ outperforms QLM on these datasets. The number of labeled query-document pairs appears to be sufficient, contributing to the learning of semantic relationships.(ii) DocT5query outperforms some dense retrieval baselines on MS 500K, which may differ from their performance on the full MS MARCO dataset. Similar experimental findings can be found in [74] . (iii) GR 2P and GR 2S perform the best, the former outperforms RIPOR by 11.7% in terms of Hits@1 on the MS 500K dataset, while the latter surpasses RIPOR by 4.3% in terms of Hits@10. This indicates that GR 2 is a general framework for generative retrieval that can adapt to both binary relevance and multi-graded relevance scenarios. Note, Appendix G.1 contains a comparison between GR 2 and the full-ranking method.Gov 500K MS 500K Gov 500K MS 500K GR !" GR #$% !" G𝑅 &' !" GR #( !" GR #)*+ !" GR ),' !" G𝑅 ,$ !" GR !- GR #$% !- G𝑅 &' !- GR #( !- GR #)*+ !- GR ),' !- G𝑅 ,$ !-In Figure 2 , we visualize the outcomes of our ablation analysis of GR 2 on Gov 500K and MS 500K. Docid design: regularized fusion approach. On both datasets, (i) GR 2S M LE and GR 2P M LE outperform NCI in retrieval performance, confirming that using docids trained with the regularized fusion method is more beneficial for retrieval performance. (ii) The performance of GR 2S RF and GR 2P RF is much lower than that of GR 2S and GR 2P , suggesting that in complex relevance scenarios, docids need to possess both relevance to the original document and distinctness.Training: MGCC loss. For Gov 500K: (i) Without the grade penalty in the MGCC loss (GR 2S -λ-λ ), the query-docid pairs at different relevance grades share the same weights, which cannot fully use information about the relationships between labels. (ii) Without the grade constraint (GR 2S -Max and GR 2P -Max ), documents with higher relevance grades play a smaller role in optimization, weakening the discriminative ability to distinguish between grades. For MS 500K, i.e., in binary relevance scenarios, the MGCC loss in GR 2S -λ , GR 2S -Max and GR 2S is the supervised contrastive loss [16] , showing the same performance.CE and GR 2P CE underperform GR 2S and GR 2P , respectively. GR 2S LR and GR 2P LR show similar results. Both variants can be viewed as the pairwise approaches. They only distinguish docids at different grades, while the MGCC loss not only penalizes docids at different grades to be distinguished from each other, but also encourages docids with the same grade to be similar.To simulate the low-resource retrieval setting, we randomly sample different fixed limited numbers of queries from the training set. To compare GR 2 , NCI and RIPOR, we randomly sample 15, 30, 45 and 60 queries from multi-graded relevance datasets. For binary relevance datasets, we randomly sample 2K, 4K, 6K and 8K queries. Zero-resource retrieval is performed by only indexing without retrieval task, i.e., the ground-truth query-document pairs are not provided. See Figure 3 . We observe the following: (i) GR 2S and GR 2P perform better than NCI and RIPOR, indicating that GR 2 is able to use relevance signals from limited information. (ii) Under the zero-resource setting, all GR methods perform worse than BM25, due to the requirements of learning the mapping between queries and relevant docids. (iii) Under the low-resource setting, on ClueWeb 500K, GR 2P can outperform BM25 in terms of nDCG@20. GR 2P has the pre-training stage, which helps the model to acquire a discriminative ability for relevance. In general, GR leaves considerable room for improvement under such settings; pre-training GR 2 with diverse corpora is likely to improve its generalization ability.We visualize query and text representations using t-SNE [81] to better understand the MGCC loss. Specifically, we sample the query "Radio station call letters" (QID: 848) from Gov 500K. We then plot a t-SNE example using the representations of the sampled query and its top-100 candidate documents given by the encoder output of GR 2P and the representative GR baselines NCI and RIPOR. As shown in Figure 4 , for GR 2P , documents with higher relevance grades are closer to the query than those with lower grades. And documents at the same relevance grade gather together. For NCI and RIPOR, the distribution of relevant documents in the latent space is relatively random: the standard Seq2Seq objective only learns to generate a single most relevant docid, from which it is difficult to learn the discriminative ability of multi-graded relevance.We compare the efficiency of GR 2 and the dense retrieval model ANCE on the Gov 500K dataset. The memory footprint refers to the amount of disk space required for storage. Additionally, we assess the end-to-end inference time during the retrieval phase. (i) Regarding memory usage, GR 2 primarily consists of model parameters and a prefix tree for docids. ANCE necessitates dense representations for the entire corpus, with the memory requirement increasing as the corpus size grows. Notably, GR 2 consumes approximately 16.7 times less memory compared to ANCE. This distinction becomes even more significant when dealing with larger datasets. For instance, in comparison to DPR, a GR approach uses 34 times less memory on the entire Wikipedia [15, 22] . (ii) In terms of inference times, the heavy process on dense vectors in dense retrieval is replaced by a lightweight generative process in GR 2 . Consequently, GR 2 consumes roughly 1.59 times less inference time than ANCE. Similar efficiency gains are observed in other GR-related work [15, 74, 76] . 
conclusion: 
 We have proposed a MGCC loss for multi-graded GR that captures the relationships between multigraded documents in a ranking, and a regularized fusion method to generate distinct and relevant docids. They work together to ensure more accurate GR retrieval. Empirical results on binary and multi-graded relevance datasets have demonstrated the effectiveness of the proposed method. There are several directions that we wish to explore: (i) We adopt hard weights for each relevance grade; what is the effect of a soft assignment setting in the MGCC loss? (ii) The generated docids remain fixed after initialization; how to perform joint optimization of the docid generation and the retrieval task? Current GR research focuses on technological feasibility, but using large language models for IR has implications for transparency, provenance, and user interactions [73] . Investigating the impact of scaled GR technology on users and societies is crucial. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Appendix F Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Section 5 Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We didn't introduce new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As shown in Figure 5 , the regularized fusion approach includes a query generation model, i.e., docid generation model, and an autoencoder model with shared decoders, to generate relevant and distinct docids.The key difference between the two regularization terms, i.e., the relevance and distinctness regularization terms, lies in the following: (i) The relevance regularization term, from a global perspective, encourages relevance between a document and its own docid, sampling docids from other documents as negative examples to ensure their non-relevance. (ii) The distinctness regularization term focuses on making documents distinguishable in the QG space (the first term of Eq. ( 2)) and, at the same time, ensuring distinguishability between docids in the AE space (the second term of Eq. ( 2)). To bridge the two spaces, we maintain a correlation between corresponding documents and docids (the third term of Eq. ( 2)). Although these two terms are not jointly optimized with the GR model, they can help generate more relevant and diverse docids. These fixed docids can guide the subsequent GR model towards appropriate optimization during learning, whereas joint optimization increases the learning difficulty. Multi-graded relevance datasets. In IR, datasets featuring multi-graded relevance annotations contribute significantly to the evaluation and advancement of retrieval models. Datasets like Robust04 [82] and large-scale web document collections such as ClueWeb09 and ClueWeb12 [19] offer diverse relevance grades. Gov2 [18] is a TREC test collection, containing a large proportion of pages in the .GOV domain. Furthermore, certain NTCIR [71] evaluation tasks contribute to the landscape.Sparse retrieval. Sparse retrieval methods typically use sparse vectors to represent queries and documents and rely on exact matching to compute similarity scores, e.g., BM25 [70] and query likelihood model [39] . However, these approaches only consider statistical information and fail to incorporate semantic information. To address such limitation, some studies [3, 4, 5, 21, 26, 101] have used word embeddings to re-weight term importance. While sparse retrieval methods can reduce the computational complexity of the retrieval process, making retrieval faster and more efficient, they may fail to match at a higher semantic level, resulting in the loss of information.Dense retrieval. To address the vocabulary mismatch problem [27, 99] in sparse retrieval, many researchers have turned to dense retrieval methods [47, 95] . These methods usually use a dualencoder architecture to learn dense representations of both queries and documents, which are then processed through a matching layer to produce the final relevance score. Formally, given a query q ∈ Q and a document d ∈ D, the two-tower retrieval model consists of two encoder functions, ϕ : Q → R k1 and ψ : D → R k2 , which map a sequence of tokens in Q and D to their corresponding embeddings ϕ(q) and ψ(d), respectively. The scoring function f : R k1 × R k2 → R is then defined to calculate the matching score between the query embedding and the document embedding: score(q, d) = f (ϕ(q), ψ(d)). To improve efficiency, approximate nearest neighbor (ANN) search [2, 6] is used to accelerate retrieval. Additionally, many pre-trained models and techniques are utilized to further enhance dense retrieval performance [1, 11, 32, 37, 41, 58] .Although this pipeline paradigm has achieved promising performance, there are still some limitations: (i) Without fine-grained rerankers, such as [11, 49, 50] , the performance of the vanilla dense retrieval method is still far from industrial applications. Using multiple heterogeneous modules, each with a different optimization objective, leads to sub-optimal performance. (ii) During inference, the query needs to search for relevant documents from the entire corpus. Although there are strategies for improving efficiency now available, such as ANN, such methods loss some semantic information.End-to-end retrieval. Thanks to the significant success of large-scale generative models in various natural language processing tasks [42, 62, 64, 65] , generative retrieval [51] has been proposed as an alternative paradigm in IR. The traditional external index is converted into a training process which learns the mapping from the documents to its docids. Given a query, a single model directly generates a list of relevant document identifiers with its parameters. This generative paradigm has potential advantages: (i) It enables end-to-end optimization towards the global retrieval objective. (ii) During inference, given a query, the model generates docids based on a small-sized vocabulary, achieving higher retrieval efficiency and eliminating the need for a heavy traditional index.Following this blueprint, GENRE [22] was the first attempt to explore this paradigm. Using the unique titles of Wikipedia articles as document identifiers, GENRE directly generated a list of relevant article titles for a query with constrained beam search based on BART [42] model. This method outperformed some traditional pipelined methods on multiple tasks based on Wikipedia. And subsequent work [7, 15, 79, 85, 106] has continued to explore and improve upon it. However, existing work focus on binary relevance scenarios, ignoring the multi-graded relevance scenarios. In this work, we aim to further explore how generative retrieval paradigm support fine-grained relevance.Exploring the scalability of large data and model size in GR is an area with limited research. Specifically, (i) there is work on scaling up GR to corpora in the millions [63, 92, 93] , finding that that as the corpus size increases, learning difficulty rises [63] . (ii) increased model parameters enhances retrieval performance [79] , which also increases latency and decreases throughput [85] .However, current research is mainly confined to datasets ranging from hundreds of thousands to millions in scale, and exploration on an even larger scale has not been undertaken. Generalizing to ultra-large-scale data is a future direction worth exploring in the field of GR.Multi-graded relevance datasets we used are (i) Gov2 [18] [38] , where the queries are natural language questions and documents are gathered from Wikipedia pages. We follow the settings of existing GR work [7, 79, 85] . Table 3 shows the statistics of these datasets.In this section, we introduce the baselines in detail. The sparse retrieval baselines include: (i) BM25[70] is a classical probabilistic retrieval model. (ii) DocT5Query [60] generates queries conditioned on a document using T5 [65] . And these synthetic queries are then appended to the original documents. Additionally, we consider two learned sparse retrieval baselines: (iii) the Query Likelihood Model (QLM) [105] adopts the query log likelihood conditioned on a document for a query as the relevance score, based on T5. (iv) SPLADE [24, 25] uses a BERT-based encoder to transform a text sequence into a sparse lexical representation.The dense retrieval baselines include: (i) RepBERT [95] is a BERT-based two-tower model trained with in-batch negative sampling. (ii) DPR [36] uses dense embeddings for text segments with a BERT-based dual encoder. PseudoQ [75] generates pseudo-queries via K-means clustering based on the token embeddings of documents to enhance learning. It is also a BERT-based two-tower model.(iii) ANCE [89] employs an asynchronously updated approximate nearest neighbors (ANN) indexer for extracting hard negative examples to train a RoBERTa-based dual-encoder model.The GR baselines, covering the majority of representative GR related work: (i) DSI-Num [79] uses arbitrary unique numbers as docids, and a MLE loss based on query-docid pairs L q MLE and document-docid pairs L d MLE . (ii) DSI-Sem [79] builds docids by concatenating class numbers generated by hierarchical k-means clustering algorithm and adopts the same training objective as DSI-Num. (iii) DSI-QG [106] uses a query generation model [60] to augment the dataset, and arbitrary unique numbers as docids. (iv) NCI [85] uses semantic structured numbers like DSI-Sem as docids and pairs of pseudo-query and docid generated by query generation strategies for data augmentation. (v) SEAL [7] uses arbitrary n-grams in documents as docids and retrieves documents based on an FM-index. (vi) GENRE [22] retrieves a Wikipedia article by generating its title, and can only be directly used for NQ. (vii) Ultron [104] starts with pre-training using document piece-docid pairs, followed by supervised fine-tuning with labeled queries and pseudo-queries. We uses the product quantization code as docids. (viii) GenRRL [103] uses multiple optimization strategies, i.e., pointwise, pairwise, and listwise relevance signals to train the model, via reinforcement learning. We use document summaries as the docid. (ix) LTRGR [44] uses a pairwise relevance loss, e.g., margin-based rank loss to train the model. (x) GenRet [74] introduces an autoencoder model to generate discrete numbers as docids. This model is learned jointly with the retrieval task. (xi) NOVO [86] selects important word sets from the document as docids via labeled relevance signals. This method also uses pseudo-queries to augment the effectiveness.Additionally, we compare current GR methods with full-ranking methods, since these approach typically yields better performance and are widely used. They consist of an initial retriever and a finer-grained re-ranking module. Specifically, we use a cross-encoder baseline, monoBERT [61] . BM25 retrieves the top 1000 candidate documents, and monoBERT ranks them by concatenating the query and document as the input.Hyperparameters.We specify α and β in Eq. ( 3) as 1 and 30, respectively. And the inference radius |r| is set to 2 and 1.5 for the multi-graded and binary relevance datasets, respectively. Since documents in NQ have unique titles, we directly use their titles as docids. We set τ in L Pair to 0.1, and γ in L total to 1. We define λ l = 1 l 2 used in L MGCC following [98] . Training. Following [17, 74, 85, 106] , we select the leading 3 paragraphs, leading 3 sentences, and randomly sample 3 entities from each document as queries, and associate them with the docid as Table 4 : Comparison between GR methods and the full-ranking baseline. * indicates statistically significant improvements over GR 2P (p ≤ 0.05).Gov 500K MS 500K nDCG@5 MRR@20 RIPOR 0.4713 0.3626 GR 2P 0.5095 0.3835 BM25+monoBERT 0.6953 * 0.5862 * additional query-docid pairs for data augmentation. And inspired by [92] , to enhance the model's effectiveness, we also adopt self-negative fine-tuning strategy.Inference. During inference, we construct a prefix trie [22] for all docids, and adopt constrained beam search to decode docids with 20 beams.Safeguards for the responsible release of resources. For pretrained language models, rigorous evaluation and testing protocols are employed to assess potential risks and biases before release. Additionally, we plan to outline strict guidelines for access control and usage policies to mitigate misuse upon publication. Regarding data release, anonymization techniques are used to protect privacy, and sensitive information is redacted or excluded where necessary upon publication.From Table 4 , we observe that GR is still in its early stages, and the current GR methods have some distance to cover compared to full-ranking methods. Achieving the integration of index, retrieval, and reranking into a single model poses a significant challenge. Possible reasons include (i) the design of docids is still independent of the final retrieval optimization; (ii) there is a lack of explicit interaction between query and document; (iii) the current optimization methods do not fully use the data; and (iv) the pre-training tasks for backbone models are not specifically designed for GR. Current GR methods can only be compared with index-retrieval frameworks, and there is still some distance from the ideal model-based IR. We hope that future work will address these aspects.We compare the efficiency of GR 2 and the dense retrieval model ANCE on the Gov 500K dataset. The memory footprint refers to the amount of disk space required for storage. Additionally, we assess the end-to-end inference time during the retrieval phase. (i) Regarding memory usage, GR 2 primarily consists of model parameters and a prefix tree for docids. ANCE necessitates dense representations for the entire corpus, with the memory requirement increasing as the corpus size grows. Notably, GR 2 consumes approximately 16.7 times less memory compared to ANCE. This distinction becomes even more significant when dealing with larger datasets. For instance, in comparison to DPR, a GR approach uses 34 times less memory on the entire Wikipedia [15, 22] . (ii) In terms of inference times, the heavy process on dense vectors in dense retrieval is replaced by a lightweight generative process in GR 2 . Consequently, GR 2 consumes roughly 1.59 times less inference time than ANCE. Similar efficiency gains are observed in other GR-related work [15, 74, 76] .Generalization on large-scale datasets for GR remains a significant challenge [63, 77] , which requires dedicated design and research, with only a few works exploring it [63, 92, 93] . Although they can achieve comparable effectiveness to dense retrieval methods, their performance still lags behind mainstream full-ranking methods. Therefore, current efforts are still focused on moderate-scale datasets [43, 74, 79, 86, 104] . Although generalization is not the main focus of this work, we still conduct some preliminary experiments. In the future, we plan to investigate how to generalize to extremely large-scale datasets, such as the corpora in the industry with billions of documents.Table 5 : Results on the MS 1M dataset.MRR@20 RIPOR 0.3674 GR 2P 0.3659To assess the performance of our method on million-scale datasets, specifically, we constructed a corpus of size 1M based on the MS MARCO dataset, referred to as MS 1M. Note, it is a binary relevance dataset, with annotation data and validation sets directly derived from the original dataset. Initially, we pre-train GR 2P on the constructed 4-grade relevance Wikipedia dataset and then fine-tune and evaluate on MS 1M. We compare the performance of GR 2P with RIPOR on MS 1M.As shown in Table 5 , we observe that (i) RIPOR performs well on large-scale datasets and exhibits strong generalization, possibly attributed to its docids generated by its own GR model, along with multi-stage enhancement training. (ii) Additionally, our method achieves comparable results to RIPOR on such a large-scale corpus, indicating that our method considering multi-graded relevance information also contributes to more accurate relevance distinction on binary relevance datasets.NeurIPS Paper Checklist 


Decision: The reviewers agreed that multi-graded relevance in generative retrieval systems is an important and largely unexplored research direction, that the proposed approach was novel, the proposed solution reasonable, and the experimental results strong.

To further improve the paper, authors are encouraged to consider the more critical feedback, including improving the clarity of the writing and possibly including additional experimental results, such as additional baselines, more ablations, and detailed error analysis.

Please see the individual reviews for additional feedback and suggestions.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a new generative retrieval model, which utilizes multi-grade relevance labels instead of binary relevance. Using graded relevance labels is not well-discussed in previous works.

Pros:
- The problem itself is interesting and important. Generative retrieval is a hot research topic in the IR community.
- The proposed solution is reasonable.


Cons:
- In the Introduction, the authors introduced the reason why the simple generation likelihood of docids cannot work for graded relevance.  It is strange to claim that "Docids commonly exhibit distinct lengths" given the truth that some existing docid schemas (such as semantic id used in DSI, the PQ-based ID, etc.) Even for token-based docID, we can still add a fix-length constraint. It is hard to convince the readers by claiming that "as a fixed length might not adequately encompass diverse document semantics".
- Both DocID schema design and learning from multi-grade relevance are considered in the paper. I am quite interested in learning whether the multi-grade relevance part can also work with other DocId schemas. So an ablation study about this (e.g., using semantic string as docids) should be given (note that this is different from the experiments in Section 5.2.2) .
strength_weakness: strength_weakness: Cons:
- In the Introduction, the authors introduced the reason why the simple generation likelihood of docids cannot work for graded relevance.  It is strange to claim that "Docids commonly exhibit distinct lengths" given the truth that some existing docid schemas (such as semantic id used in DSI, the PQ-based ID, etc.) Even for token-based docID, we can still add a fix-length constraint. It is hard to convince the readers by claiming that "as a fixed length might not adequately encompass diverse document semantics".
- Both DocID schema design and learning from multi-grade relevance are considered in the paper. I am quite interested in learning whether the multi-grade relevance part can also work with other DocId schemas. So an ablation study about this (e.g., using semantic string as docids) should be given (note that this is different from the experiments in Section 5.2.2) .
questions: questions: na
limitations: limitations: na
paper_summary: paper_summary: The paper proposes a novel QG based docid for generative retrieval, optimising relevance and distinctness of the generated queries jointly. It introduces the MGCC loss with multi-graded labels. Experiments on subsets of Gov2, ClueWeb09-B, Robust04, MS Marco, and NQ with up to 500k documents validate the effectiveness of the methods.

Overall, the contributed methods are valuable and inspiring, and the results are convincing.
strength_weakness: strength_weakness: - Limited corpus scale remains a concern for various GR works, further discussion on its potential influence on the proposed method is needed. Nonetheless, a 500k corpus is substantial and applicable.
- The paper should discuss the existing works more about the use of QG in GR
questions: questions: - The term "pre-train" is somewhat misleading, as using semi-supervised Wikipedia data for further training differs from typical LLM pre-training.
- Some examples for generated queries, both with and without using the RF building block, to better understand the reasons behind its effectiveness.
limitations: limitations: Yes
paper_summary: paper_summary: The work deals with generative retrieval for cases where documents have multi-graded relevance. The authors propose GR2 a framework for generative retrieval in such cases by tackling two important challenges. First they optimize for relevance and distinctiveness of document IDs by a regularized fusion approach which comprises a pseudo query generation module followed by an auto-encoding module that reconstructs the document ids. Secondly, the authors introduce a graded, constrained contrastive learning approach which aims to bring together representations of queries with the representations of the relevant docids and push apart irrelevant docids in mini-batch. GR2 shows impressive performance improvements over other approaches and is also relatively efficient.
strength_weakness: strength_weakness: - Some important baselines are missing in the current version of the work. ColBERT (ColBERTV2) in PLAID setup is a powerful dense retriever and  would serve as a valuable approach for comparison. Additionally,  dense retrieval models like contriever and tas-b have also demonstrated impressive performance on a wide range of benchmarks and are strong dense retrievers due to their training objectives and should be included for comparison. Particularly since GR2 is also used in pre-training setup it is important to compare to retrievers like Contriever. pre-trained using contrastive learning for retrieval.


- While there is clear intuition for the distinctiveness, relevance regularization losses and the MGCC loss, further experiments could be carried out with respect to the contrastive learning parts of these approaches  to understand the approach better and  to help improve or strengthen the results. For instance, negative sampling is an important aspect of contrastive learning and a good mix of hard and soft negatives are crucial for learning useful representations. While in MGCC authors explore mini-beach based negatives, it is also important to explore global negatives similar to works like ANCE [1].

- The paper would benefit from some error analysis. This involves analysis of cases where wrong docids are considered relevant, and an analysis of whether it stems from the stochasticity of generative models or other external factors. The paper does not currently report any list of common errors or a qualitative analysis of results. It is critical to understand where the current approach fails as it would provide avenues for further research.


[1] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval LEE et. al.
[2] Learning to tokenize for generative retrieval.
questions: questions: - Equation 10 gives equal weights to the regularization losses. Have you considered weighting them differently ?
limitations: limitations: While authors briefly describe the limitations in conclusion a more detailed explanation would help contextualize the work and understand future directions better.
paper_summary: paper_summary: The paper targets the problem of Multi-Graded Relevance in a Generative Retrieval setup. 

Generative Retrieval uses a Seq2Seq model to decode relevant “docids” for a given query. Prior work on Generative Retrieval has focused on tasks where there is only one relevant document per query and the degree of relevance is binary : “relevant” or “not-relevant”. The paper talks about Multi-graded Retrieval tasks, where each query has multiple relevant documents, each having its own degree of relevance. 

The paper proposes a methodology to tackle the graded retrieval tasks in a Generative Retrieval Framework. It identifies the key challenges : 1. Having distinct but relevant “docids” and 2. Graded Relevance when matching query and documents. It proposes solutions to these challenges in “regularized fusion” and “MGCC” approaches respectively. 

Additionally the authors also develop and experiment with a pre-training setup for multi-graded relevance retrieval.
strength_weakness: strength_weakness: While not a strong weakness, the mathematical notation in the paper can use some work to make the paper an easier read. I found it tough to parse through “Section 3 : Preliminaries”. The paper would benefit from resolving this.
questions: questions: I just have a few clarifying questions : 
1. Are there numbers on Recall? Prior work I believe focuses on MRR and Recall metrics. 
2. Why include the last term in Eq2 (similarity in embedding space) when the same is taken care of by Eq1? Is it for training stability?
3. If I understood correctly, then docids are generated by decoding a random sample on the hypersphere  (z=e+r). Does that mean if we were to sample multiple times for each document, we would get distinct docids from the randomness in r itself? Do we need e of documents (through QG module) to be “distinct” (i.e. do we need regularized fusion if we have better sampling or more samples of r?)
limitations: limitations: Typo : In line 188, “These fixed” needs “T” to be lowercase

While not a limitation, I believe the paper would also benefit from a few Qualitative examples showing Multi-graded relevance in practice. An uninitiated reader might not be familiar with Generative Retrieval or Multi-graded relevance and a figure/qualitative example might be useful for them.

Comments: comment: Thank the authors for the detailed response.
Thank the authors for the detailed response.