Self-Retrieval: End-to-End Information Retrieval with One Large Language Model

Authors: qiaoyu  tang
jiawei  chen
zhuoqun  li
bowen  yu
yaojie  lu
cheng  fu
haiyang  yu
hongyu  lin
fei  huang
ben  he
xianpei  han
†  le sun
yongbin  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:19:22

Abstract: The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. 
However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them.
In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture.
Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process.
Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning,  transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking.
Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation.

Main paper: 
 introduction: 
 Recently, information retrieval (IR) systems and large language models (LLMs) have witnessed a growing synergy, with advancements in one field driving progress in the other [13, 56] . On one hand, IR systems have proven effective in augmenting LLMs and mitigating challenges such as hallucinations and outdated knowledge [22, 16] . By providing accurate, up-to-date external knowledge, IR systems significantly enhance the reliability and performance of LLMs. On the other hand, the powerful language understanding and generation capabilities of LLMs have been leveraged to enhance almost all components of traditional IR systems-indexing, retrieval [42, 9, 26] , and reranking [58, 27, 40] . Through the integration of LLMs into the IR pipeline, these systems achieve substantially improved retrieval accuracy [57, 1] . However, current IR systems typically adopt a pipeline architecture where different components operate in isolation, limiting LLMs' role to specific components rather than leveraging their full Figure 1 : The Self-Retrieval framework consists of three key components: (1) corpus indexing through self-supervised learning, (2) passage generation via constrained decoding, (3) passage ranking using self-assessment scoring. potential across the entire system. This fragmented approach creates several challenges: it hinders knowledge sharing between components, prevents deep integration of LLMs' diverse capabilities, and results in complex implementations with potentially sub-optimal performance. These limitations underscore the need for a more unified approach that fully integrates LLMs across all components of the IR system. Such an approach would not only maximize the utility of LLMs' capabilities but also simplify system implementation while potentially achieving better performance through enhanced component synergy.In this paper, we introduce Self-Retrieval, an end-to-end information retrieval architecture driven entirely by one large language model. This integration is not trivial due to the inherent mismatch between information retrieval tasks and text generation, particularly in ensuring accurate document generation using language models. As illustrated in Figure 1 , Self-Retrieval consolidates the separate components of an IR system -indexing, retrieval, and reranking -into the parameters of a single LLM. For indexing, the corpus is internalized into the LLM's parameters through self-supervised learning, enabling the model to encode and store corpus information within its internal representations. During retrieval, Self-Retrieval leverages its encoded knowledge of the corpus to semantically match the input query and directly generates the relevant documents as outputs. To ensure the generated documents exactly match those in the original corpus, we employ the constrained decoding algorithm [10, 8, 24] based on the trie of the corpus. For reranking, Self-Retrieval performs self-assessment on the retrieved documents to evaluate their relevance. The output score is used to rerank the retrieved passages. Moreover, for downstream tasks such as retrieval-augmented generation (RAG), Self-Retrieval integrates the reader component into the model, enabling direct answer generation following retrieval. Through this end-to-end approach, Self-Retrieval fully leverages LLMs' powerful capabilities in language understanding, matching, assessment, and generation to achieve unified information retrieval.We evaluate Self-Retrieval on three representative retrieval benchmarks: NQ, TriviaQA, and MS MARCO. Experimental results demonstrate that Self-Retrieval substantially outperforms existing sparse retrieval, dense retrieval, and generative retrieval methods on both document-level and passage-level retrieval tasks. Furthermore, our experiments on retrieval-augmented generation tasks reveal that Self-Retrieval considerably enhances downstream performance. Additionally, larger LLMs lead to progressively better performance in Self-Retrieval, showing clear scaling benefits. These results demonstrate the effectiveness of Self-Retrieval across different retrieval tasks and application scenarios.The potential impacts of this paper may include the following aspects. First, we introduce Self-Retrieval, an end-to-end architecture that consolidates the entire information retrieval system within a single large language model. This unified approach demonstrates substantial performance improvements over existing IR methods. Second, the corpus internalization and indexing mechanism of Self-Retrieval establishes a new paradigm to memorize, organize and retrieve the learned documents (at least part of them) during the pre-training phase, paving the way for more transparent and trustworthy text generation from LLMs. Third, as a LLM-driven retrieval system, Self-Retrieval offers inherent advantages in terms of compatibility, consistency, and interaction with LLMs' internal knowledge. Through experiments on RAG, we demonstrate how this natural compatibility leads to superior performance, suggesting broader potential for enhancing various LLM-based applications. 
related work: 
 LLM for IR Recent studies have explored leveraging LLMs to enhance various components of IR systems, including query rewriting, retrieval, and reranking. For query rewriting, LLMs have been employed to generate pseudo-documents for query expansion [46] and to rewrite queries based on conversational context [15] . In the retrieval stage, researchers have explored augmenting data by generating pseudo-queries [6, 17] or relevance labels [25] using LLMs, as well as employing LLMs directly as generative retrievers [42, 5] . Regarding reranking, LLMs have been utilized in two ways: serving as rerankers directly [27, 40] and augmenting the reranking dataset [12] . While these methods have advanced specific components within the IR pipeline, Self-Retrieval distinguishes itself by presenting an end-to-end architecture driven entirely by a single LLM, eliminating the need for external components.Dense retrieval Dense retrieval models retrieve information by matching dense vector representations of queries and documents [19] . In this paradigm, an encoder transforms both queries and documents into dense vectors, with relevance determined by their vector distance. Various strategies have been proposed to enhance dense retrievers, including designing loss functions [45] , multivector [38] , training with synthetic queries [33, 47] , and leveraging large-scale query-document pairs [30, 50] . Recent work has also explored using large language models to generate dense vectors for both queries and documents [29] . However, the fundamental limitation of dense retrieval lies in its limited interaction with LLMs, as the compression of natural language into dense vectors inherently constrains the utilization of LLMs' sophisticated language understanding and semantic inference capabilities.Generative retrieval Generative retrieval methods leverage sequence-to-sequence language models to generate document identifiers for a given query [8, 42] . This paradigm is pioneered by GENRE [7] , which introduces the concept of entity retrieval through constrained beam search generation of entity names. DSI [42] extends it to document retrieval by training T5 models to generate document-specific identifiers. The field has since evolved through various innovations, including query generation techniques [11, 59] , sophisticated identifier design [48, 51] , architectural improvements [5, 36] , and continual learning strategies [20, 14] .Most relevant to our work, Yu et al. [52] proposed a "generate-then-read" approach, advocating for the use of LLMs to directly generate documents instead of relying on a retriever. UniGen [23] proposed a unified framework that integrates generative retrieval and question answering through a dual-decoder architecture. Compared to them, Self-Retrieval ensures accurate document generation through constrained decoding and accomplishes both retrieval and answer generation in one turn.The main distinctions between Self-Retrieval and existing generative retrieval methods can be summarized as follows: (1) Self-Retrieval enables LLMs to directly generate document content rather than relying on other text or numeric identifiers. This approach aligns naturally with LLMs' pretraining objectives, preserves their inherent knowledge, and eliminates the need for complex identifier construction schemes. (2) Self-Retrieval further integrates components such as reranking and answer generation into the framework, further expanding its scope and enhancing the retrieval performance. These distinctions highlight that Self-Retrieval represents a more natural and effective approach for leveraging the capabilities of LLMs in information retrieval.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have specifically written a chapter to illustrate the limitations (Section Limitations) of our method and the factors that may affect its performance. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
self-retrieval: 
 In this section, we introduce our proposed Self-Retrieval. The overall architecture is illustrated in Figure 1 . Different from traditional information retrieval systems that separate indexing, retrieval, and reranking components, Self-Retrieval integrates these functionalities directly into the parameters of a single large language model:• Indexing: Self-Retrieval internalizes the entire corpus into its parameters through selfsupervised learning, enabling the model to process passages internally without relying on external indices. • Retrieval: Given an input query q, Self-Retrieval generates relevant passage p using the knowledge embedded within its parameters, which is different from dense retrieval or generative retrieval that rely on embedding or document identifiers as proxies of passage.• Reranking: After generating passage p, Self-Retrieval assesses its relevance to the query q through self-assessment. The output logits provide the basis for reranking candidate passages.Through this unified approach, Self-Retrieval enables a streamlined, end-to-end process that enhances the overall effectiveness of information retrieval. In the following sections, we detail each component of our method.Self-Retrieval integrates indexing into the LLM's parameters through self-supervised learning, enabling the model to internalize the entire corpus. Unlike generative retrieval methods that rely on complex document identifiers and identifier matching, Self-Retrieval employs a straightforward sentence-to-passage task to construct the index. Specifically, given a passage p = {s 1 , s 2 , ..., s L } consisting of L sentences, each sentence s i is provided as input to the LLM with parameters θ.The training objective is to generate the source passage p in an auto-regressive way, represented as P (p|s i , θ). This self-supervised indexing approach offers several advantages. First, it provides a simple yet effective method for corpus indexing. Second, it naturally frames the indexing process as a retrieval-like task, enabling the model to simultaneously internalize the corpus and develop retrieval capabilities using a consistent data format. Furthermore, this indexing technique closely aligns with the pre-training processes of language models, suggesting that our method could be considered as continued pre-training on the corpus. Through this process, the LLM learns to efficiently memorize and organize corpus information within its parameters.Retrieval serves as a first-pass filter to collect passages related to the input query. In Self-Retrieval, we train the LLM to directly generate relevant passages in response to queries, eliminating the need for intermediaries such as embedding in dense retrieval or document identifier in generative retrieval. Specifically, given the query q and corpus D, Self-Retrieval first generates a potential document title t as global information, formulated as P ( t|q; θ). The model then generates a relevant passage, denoted as P (p|q, t; θ).However, since LLMs are general-purpose pre-trained models rather than statistical frequency models, the generated passage p may not exactly match any passage in D, making it challenging to locate the corresponding passages in the corpus. To address this challenge, we employ a trie-based constrained decoding algorithm [10, 8, 24] . This approach restricts generated tokens to a dynamically constrained vocabulary. We construct a prefix tree T from corpus D, where each path from the root to a leaf node represents a unique passage in the corpus, and each node stores valid tokens for the next generation step. During inference, the vocabulary at each generation step is constrained by the valid continuations in the prefix tree. Due to the relatively short common prefixes among documents, the LLM terminates generation once it has produced sufficient tokens to uniquely identify the current document and concatenates the full document to the context. This results in document title and passage generation processes represented as P ( t|q; θ; T ) and P (p|q, t; θ; T ). This mechanism ensures that generated passages align with existing corpus content.Reranking serves as a second-pass filter to precisely sort the retrieved passages based on the relevance to the query. We implement a self-assessment mechanism that leverages the Self-Retrieval model itself to evaluate the relevance of generated passages. Specifically, Self-Retrieval assesses the passage relevance by generating responses such as "can answer the query" for relevant passages and "cannot answer the query" for irrelevant ones. This self-assessment mechanism allows the model to generate passages and evaluate their relevance within a single inference turn.During training, we utilize the gold passage from the supervision data as the positive instance, while sampling negative instances from both the same and different documents. This training strategy conditions the LLM to accurately discern and verify the relevance of its outputs, thereby enhancing its autonomous relevance assessment capabilities and improving the overall precision of the retrieval process.During inference, the overall relevance score S is composed of the document title score S T and the self-assessment score S P . Specifically, the document title score is derived from the title generation probability, while the self-assessment score is calculated based on the probability of the language model rejecting the passage. Formally, for a set of generated titles and passages {(t 1 , p 1 ), (t 2 , p 2 ), . . . , (t n , p n )}, the title score for each (t i , p i ) is given by:EQUATIONand the assessment score is:S P i = Softmax((1 -P (rejection response|q, t i , p i ; θ))/δ) (2)where τ and δ are temperature parameters used to scale the logits. Based on preliminary experiments on the development set, we simply set τ = δ = 0.4 for the main passage retrieval experiments.The final relevance score is computed as the product of these two components:S = S T • S P (3)This combined score is then used to rerank the passage set, producing a more refined ordering based on relevance.Training Self-Retrieval unifies the three distinct tasks of information retrieval -indexing, retrieval, and reranking -into text generation tasks, trained using cross-entropy loss in an auto-regressive manner. Specifically, Self-Retrieval first internalizes the corpus into its parameters through selfsupervised learning as introduced in Section 3.1. Subsequently, in addition to a portion of selfsupervised instances, it incorporates two different types of data to build retrieval and reranking abilities:• Retrieval data: Utilizes supervised query-passage pairs from the dataset, where the model learns to generate both document titles and passage content in response to input queries.• Reranking data: Employs positive and negative examples to train the model in relevance assessment between queries and passages.This auto-regressive training approach enables Self-Retrieval to integrate traditionally separate IR components into a unified language model, establishing an end-to-end IR system.Furthermore, leveraging the universal language generation capabilities of LLMs, we can seamlessly integrate downstream task components, such as readers in RAG, into Self-Retrieval. This integration can be achieved by simply appending the golden answer after the assessment in Self-Retrieval. Consequently, the LLM can function as a comprehensive RAG system, effectively reducing the knowledge gap between IR system and reader modules.Inference During inference, given an input query, Self-Retrieval aims to obtain the relevant passages that are sorted based on the relevance to query. Firstly, the model generates i document titles through constrained beam search. Secondly, for each title, it generates j passages using beam search. Finally, the resulting i×j passages are scored using the self-assessment mechanism and reranked to produce the final output.4 Experimental ResultsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experimental setup: 
 Datasets and metrics We conduct main experiments on Natural Questions (NQ) [21] and Trivi-aQA [18] datasets, both of which are widely used retrieval benchmarks based on Wikipedia. We use their versions from the KILT benchmark [34] , which consolidates these datasets into a single pre-processed Wikipedia dump, facilitating easier evaluation. Since the KILT test set is not publicly accessible, we use the development set for testing and randomly sample 2,000 instances from the training set as our development set. For our experiments, we sample approximately 40K documents NQ TriviaQA Model Params H@1 H@5 M@5 H@1 H@5 M@5 Sparse Retrieval BM25 [ from Wikipedia for each dataset. Each document is segmented into passages of maximum 200 words, yielding approximately 1 million passages in total. The detailed statistics of the datasets are presented in Appendix A. We use passage-level Hits@{1, 5} and Mean Reciprocal Rank (MRR)@5 as evaluation metrics.To comprehensively compare with other generative information retrieval methods, we also conduct experiments on document retrieval. Following NCI [49] , we conduct experiments on NQ320K and utilize Recall@{1, 10} and MRR@100 as the evaluation metrics. To evaluate the model's robustness in non-Wikipedia scenarios where high-quality text and titles are not available, we conduct experiments on a subset of MS MARCO [3] following the experimental setup of Ultron [55] . The performance was measured using Recall@{1,5} and MRR@10.Implementation details In this study, we employ StableLM-3B [44] and Llama2-7B [43] as passage retrieval backbones. For document retrieval, we employ StableLM-1.6B [4] for NQ320K and StableLM-3B for MS MARCO. We train the models using ZeRO stage-2 optimization on 8 NVIDIA A100 (80 GB) GPUs with the AdamW optimizer, a batch size of 16 per GPU, and BFloat16 precision. The models are trained for 3 epochs with a learning rate of 2e-5. During inference, we use beam search to generate 5 titles and 10 passages for each title, with hyperparameters τ and δ set to 0.4 across all models and datasets.Baselines We evaluate Self-Retrieval models for both passage retrieval and document retrieval, comparing them with sparse, dense, and generative retrieval baselines. The sparse retrieval baselines are:BM25 [37] and DocT5Query [28] . The dense retrieval baselines include: DPR [19] , Sentence-T5 [31] , GTR [32] , BGE [50] , text-embedding-ada-002 [30] , GritLM [29] , and their finetuned variants, DPR-FT and BGE-FT. The generative retrieval baselines comprise: DSI [42] , DSI-QG [59] , NCI [49] , Ultron [55] , DynamicRetriever [54] , GenRet [39] , and SEAL [5] . Additionally, to ensure a comprehensive comparison, we also evaluate combinations of strong retrieval baselines with various rerankers, including BGE-Reranker, BGE-Reranker-FT, and RankGPT [41] . In the passage retrieval task, we use the official pre-trained models for all non-fine-tuned dense retrieval baselines. For fine-tuned dense models and generative models, we use their official implementations to replicate the experiments on our dataset. In the document retrieval task, we report the baseline performances from their original paper. For comprehensive details about these baselines, please refer to Appendix B.Passage retrieval In Table 1 , we compare the performance of Self-Retrieval with various baselines on the NQ and TriviaQA datasets. Self-Retrieval 3B outperforms both strong pre-trained dense retrieval models, such as BGE and GritLM 7B, and other generative retrieval methods. Specifically, Self-Retrieval 3B achieves improvements of 5.46 and 5.07 in MRR@5 over the fine-tuned BGE on NQ and TriviaQA datasets, respectively.Our results indicate that other generative retrieval baselines exhibit suboptimal performance on passage retrieval. Even the largest DSI-XXL model only achieves an MRR@5 of 50.20 on NQ, significantly lagging behind dense retrieval methods such as GritLM, which achieves an MRR@5 of 57.03. In contrast, our Self-Retrieval model demonstrates strong performance in passage retrieval, achieving an MRR@5 of 69.45, significantly outperforming all other generative methods.We further compare Self-Retrieval with conventional 2-stage retriever-reranker pipeline. Representative results are shown in Table 1 , while the complete experimental results are provided in Appendix D. Notably, even strong retrieval baselines (BGE-FT, GTR-XL, GritLM, and DSI-XL) enhanced with powerful rerankers (such as BGE-Reranker-FT) still fall short of Self-Retrieval's performance, highlighting the advantages of unifying multiple retrieval processes into a single framework rather than treating them as separate components.These findings underscore the efficacy of Self-Retrieval in harnessing the memory, generation, and ranking capabilities of LLMs, thereby excelling in passage retrieval tasks where other generative baselines struggle.Method R@1 R@10 M@100 Sparse Retrieval BM25 [37] 29 We present the document retrieval results on NQ320K dataset in Table 2 .Self-Retrieval outperforms all other generative retrieval methods and dense retrieval baselines across all three metrics. Compared to GenRet, the previously strongest generative retrieval method, Self-Retrieval improves Hits@1 by 5.2, Hits@10 by 3.8, and MRR@100 by 4.8 points. Notably, while other methods commonly employ query generation to augment their training data, Self-Retrieval achieves these results using only the original training set.To evaluate the effectiveness of Self-Retrieval in non-Wikipedia scenarios, we extend our experiments to MS MARCO. To address the absence of document titles in MS MARCO, we employ Llama2 to automatically generate titles. As shown in Table 3 , Self-Retrieval achieves comparable performance to the SOTA model GenRet, while significantly outperforming other baselines. These results demonstrate its adaptability and robustness in non-Wikipedia and title-lacking contexts.Ablation study To study the effect of each component, we conduct ablation study on both NQ and TriviaQA. Results are presented in Table 4 . All components prove crucial for Self-Retrieval's Method NQ TriviaQA H@1 H@5 M@5 H@1 H@5 M@5 Self-Retrieval (base) 62.16 79.28 69. 45 performance, with each ablation resulting in substantial performance degradation. Specifically, removing the indexing mechanism restricts the model to internalizing only the documents encountered during training, leading to poor performance on unseen passages. Without titles, we directly generate passages with constrained decoding. The absence of document titles significantly degrades performance, as titles provide critical global information that guides the LLM in generating relevant content. Furthermore, removing the self-assessment mechanism leads to a significant decrease in both datasets. Without self-assessment, the model cannot effectively evaluate and refine its initial retrieved passages, leading to less accurate document rankings. This degradation directly impacts downstream applications such as RAG, where precise passage ranking is crucial for generating highquality responses. These ablation results show that each component of Self-Retrieval addresses a specific challenge in the retrieval process, contributing to its overall effectiveness. Table 5 : The performance on retrieval-augmented generation. For baseline, we use BGE-FT as the retriever and a fine-tuned LLM as reader. Results are reported using Exact Match (EM) scores.The end-to-end architecture of Self-Retrieval seamlessly integrates retrieval and answer generation into a single inference process. To evaluate its effectiveness in RAG, we compare Self-Retrieval models with a strong baseline that combines BGE-FT for retrieval and fine-tuned versions of StableLM-3B and LLaMA2-7B as readers. We conduct experiments on subsets of NQ and TriviaQA using 10K and 40K documents for each dataset. We utilize the top-1 retrieved passage as the context and measure performance using the Exact Match (EM) metric. As shown in Table 5 , Self-Retrieval significantly outperforms the baseline on both datasets across different model scales. Unlike other RAG pipelines that separate retrieval and generation, Self-Retrieval integrates the entire process within the LLM framework, enabling more accurate and coherent responses through end-to-end modeling.Scaling model capacity To explore the impact of model scale on retrieval performance, we evaluate Self-Retrieval with various backbone models of different sizes, including StableLM (1.6B, 3B) [4, 44] , Llama2 (7B, 13B) [43] , and Qwen-1.5 (4B, 7B, 14B) [2] . Figure 2 presents the results on NQ, showing that Self-Retrieval's retrieval performance benefits from the general capabilities of larger language models. For models within the same series, as the model size increases, we observe consistent improvements in both Hits@1 and Hits@5, indicating strong scaling properties of the Self-Retrieval architecture.Recent studies [35, 53] have demonstrated that generative retrieval methods such as DSI or NCI experience more significant performance degradation compared to dense retrieval methods when scaled to larger corpora. To explore the impact of corpus size on Self-Retrieval, we expand our experiments from 10K to 200K documents, scaling the number of passages from 290K to 3M. Figure 4 illustrates the performance trends of BGE-FT and our Self-Retrieval 3B model on the NQ and TriviaQA datasets with increasing corpus sizes. While both models show performance decrease with larger corpus sizes, Self-Retrieval maintains a degradation rate comparable to BGE-FT. As the number of documents continues to increase, the degradation rate gradually diminishes, demonstrating Self-Retrieval's potential scalability to larger document collections. This observation indicates that Self-Retrieval effectively addresses some of the inherent limitations of generative retrieval approaches in large-scale scenarios.In this part, we conduct an in-depth analysis of the reranking performance of Self-Retrieval reranker module in comparison with the fine-tuned BGE-Reranker. We employ DPR-FT, SEAL and GritLM to retrieve 100 passages on TriviaQA, followed by reranking the retrieved results using both approaches. We evaluate performance using MRR@5 as the metric. The experimental results are presented in Figure 3 . The results reveal two key findings: (1) Reranking plays a crucial role in information retrieval systems, significantly enhancing the ranking performance across all models. (2) The Self-Retrieval reranker consistently outperforms the fine-tuned BGE Reranker in most scenarios, demonstrating its robustness and effectiveness. These findings demonstrate that Self-Retrieval performs effectively both as a complete IR system and as a reranker component.In Appendix C, we conduct additional experiments with a chunk size of 100 words, demonstrating Self-Retrieval's adaptability to different text segmentation strategies. In Appendix E, we further discuss Self-Retrieval's computational efficiency. 
conclusion: 
 In this paper, we propose Self-Retrieval, an end-to-end LLM-driven information retrieval architecture that unifies indexing, retrieval, and reranking in a single LLM. This approach enables the LLM to internalize the corpus, generate relevant content, and perform self-assessment within a unified framework. Unlike previous works that incorporate LLMs into individual IR components, Self-Retrieval provides a unified framework for the entire IR procedure, facilitating knowledge sharing and deep collaboration among different components. Experimental results demonstrate that Self-Retrieval achieves strong performance across various retrieval benchmarks and application scenarios. In future work, we plan to extend our method to further enhance the reliability and trustworthiness of LLM generation.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have submitted the code and instructions in an open-source github repository, and can obtain the experimental results in the paper by running our code.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have marked the significance analysis with an asterisk in the main experimental table (Table 1 ).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We talk about positive impacts in introduction and method section, and possible negative impacts in limitation part.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our experiments use publicly available base models and standard training datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly cite all datasets and models used in our experiments.. 
 
 Appendix: While our experiments demonstrate the effectiveness of Self-Retrieval, several limitations need to be addressed in future work. Our current evaluation is limited to 200K Wikipedia documents and 3M passages, and testing on larger and noisier text collections is needed. As an LLM-driven system, Self-Retrieval has lower retrieval efficiency compared to sparse or dense retrieval methods, which may limit its applications to specialized knowledge systems. Furthermore, enabling incremental learning and dynamic corpus expansion remains an important direction for future research. The sparse retrieval baselines are as follows:• BM25 [37] is a classical sparse retrieval algorithm based on probabilistic relevance framework and term frequency statistics. • DocT5Query [28] expands documents by generating potential queries using a fine-tuned T5 model.The dense retrieval baselines are as follows:• DPR [19] is a dual-encoder model trained with in-batch negative sampling. We fine-tune DPR on our training datasets to obtain DPR-FT, following the official implementation and hyperparameter settings. • BGE [50] is a state-of-the-art universal embedding model trained on approximately 200 million text pairs using contrastive learning. We employ the bge-large-en-v1.5 variant and fine-tune it on our training datasets to obtain BGE-FT. The fine-tuning process uses a learning rate of 1e-5, batch size of 128, and runs for 10 epochs. • Sentence-T5 [31] employs a dual-encoder T5 architecture to generate semantic embeddings through contrastive learning for efficient retrieval. • GTR-XL [32] is a dense retrieval model based on Sentence-T5, pre-trained on billions of question-answer pairs. • Text-embedding-ada-002 is a powerful embedding model developed by OpenAI, accessible through their API service. • GritLM [29] is built upon the Mistral 7B language model and optimized using both embedding and generation objectives.The generative retrieval baselines are as follows:• DSI [42] is a sequence-to-sequence model that directly maps queries to document identifiers. • DSI-QG [59] enhances the DSI framework by incorporating a doc2query model for dataset augmentation.• SEAL [5] utilizes n-gram as the document identifiers and constrains the generation process using FM-index. • NCI+BGE-Reranker-FT. NCI [49] employs a sequence-to-sequence architecture with a prefix-aware weight-adaptive decoder. We train the model using T5-Large for documentlevel retrieval following the official implementation. To obtain passage-level results, we further incorporate a fine-tuned BGE reranker (bge-reranker-large).• Ultron [55] represents documents using three types of identifiers (URL, PQ, Atomic) and trains the model through a progressive three-stage pipeline.• DynamicRetriever [54] parameterizes traditional static indices by embedding both tokenlevel and document-level corpus information into a pre-trained model for dynamic document identifier generation.• GenRet [39] employs discrete auto-encoding with progressive training and clustering techniques to learn semantic document identifiers for generative retrieval.To investigate the potential impact of chunk size, we conduct additional experiments comparing Self-Retrieval with strong baselines on the NQ dataset using a chunk size of 100 words, complementing our main experiments where chunk size is set to 200. The experimental results are presented in Table 7 . It demonstrates that Self-Retrieval significantly outperforms the baselines with both chunk sizes settings, further validating the effectiveness of our proposed method.Model Params Hits@1 Hits@5 MRR@5 BGE-FT 335M NQ TriviaQA H@1 H@5 M@5 H@1 H@5 M@5 BGE-FT We comprehensively evaluate Self-Retrieval against various two-stage retriever-reranker pipelines. Specifically, we construct these pipelines using state-of-the-art retrievers (BGE, GTR, GritLM, and DSI-XL) combined with three different reranking approaches: BGE reranker, fine-tuned BGE reranker, and RankGPT. As shown in Table 8 , Self-Retrieval achieves superior performance compared to most retriever-reranker combinations, demonstrating the effectiveness of our end-to-end approach over traditional pipeline methods.We conduct efficiency analysis on NQ dataset using an NVIDIA A100-80G GPU. Results in Table 9 illustrate that, while Self-Retrieval requires slightly higher computational resources than DSI, it provides notable performance benefits. Notably, Self-Retrieval with a beam size of 10 achieves significantly higher H@5 scores compared to DSI-XL with a beam size of 100, enabling a flexible trade-off between retrieval quality and computational efficiency. When compared to SEAL, which also employs natural language decoding, Self-Retrieval demonstrates more efficient memory usage (30MB vs 444MB) by utilizing a lightweight trie structure instead of SEAL's resource-intensive FM-Index post-processing mechanism. Furthermore, the efficiency of Self-Retrieval stands to benefit from ongoing developments in optimization techniques (e.g., quantization and attention acceleration) and hardware advancements. 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We have clearly explained the shortcomings of previous works in the abstract and introduction (Section 1), as well as how we can make improvements. We have validated these claims in the experiment section (Section 4). Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 


Decision: This paper introduces Self-Retrieval, an approach that uses LLMs for end-to-end information retrieval tasks. The proposed approach integrates indexing, retrieval, and reranking in a single model and utilizes a novel strategy for passage generation and self-assessment.

The reviewers found the paper to be well-written, the proposed approach to be highly novel, and the experimental results to be convincing and strong.

The reviewers did identify a number of weaknesses, as well, including deviations from standard evaluation practices (e.g., passage-based setup vs. document based one, KILT setup, etc.), the desire to see more detailed ablations, and questions around the computational efficiency of the approach.

Overall, this is a very interesting and novel paper that will very likely open up opportunities for future research along similar directions. While the paper isn't perfect, its strengths outweigh its weaknesses. The authors are encouraged to carefully consider the reviewer feedback when preparing an updated version of the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes an approach of self-retrieval, which uses the probability of generation of the passage as the ranking criterion. To limit the generation to the existing passages, a trie-structure is used, forcing the generation to produce the existing passages.
The experiments compared the method with several existing approaches, including sparse retrieval, dense retrieval and generation-based retrieval. The proposed method outperforms the others.
strength_weakness: strength_weakness: A key idea is the use of trie for passage generation. This is described briefly. More details should be presented. If a whole passage should be generated using trie, then the depth of trie structure will be very large (equivalent to the length of the passage). What about the storage cost of trie? What is the time efficiency?
questions: questions: Have you evaluated the space and time complexity of the approach, and compare it to the existing methods?
How do you deal with long document (in particular for the trie structure)?
limitations: limitations: yes
paper_summary: paper_summary: This paper introduces Self-Retrieval, a new generative retrieval architecture.  Self-Retrieval first memorizes the corpus into LLM's parametric knowledge using self-supervised training. Given a query, it generates the target document with constrained decoding, then re-assess document by decoding if the document can answer the query.  On subset of NQ and TriviaQA, this approach significantly outperforms existing dual encoders and generative retrieval models.
strength_weakness: strength_weakness: - Experiment only used wikipedia-based datasets. However, wikipedia is heavily used in pretraining, so it is unclear if the proposed approach can let model sufficiently memorize other datasets. 
- More importantly, the method relies on generating the passage title. Unlike wikipedia, many retrieval datasets do not have high-quality, natural language passage titles. It is unclear how the method works on those datasets.  It would be nice to test retrieval benchmarks like MS MARCO or BEIR/MTEB. 
- Missing dense retrieval + cross-attention reranking baselines. Such 2-staged pipeline is standard in IR. Since the proposed method's reranking stage essentially uses cross attention to judge the query and the retrieved candidate passage, the computational cost of the reranking stage is similar to that of a separate cross-attention reranker. It is fair and necessary to compare it with commonly-used rerankers such as MonoT5, RankT5, or BGE reranker.  The ablation in Table 3 seems to show that the proposed method's retrieval-alone performance is stronger than most retrieval baselines, but the paper can be more convincing if having e2e comparison to other 2-stage retrieval pipelines like BGE + BGE reranker or GTR + RankT5.
- Lacking efficiency discussion.
questions: questions: - Can the method scale up to the full NQ/TriviaQA? If so, would be nice to see the performance on these more standard setups. If not, what is the main bottleneck for scaling up? 
- How many candidates were considered in the reranking stage?
limitations: limitations: NA
paper_summary: paper_summary: This paper proposes Self-Retrieval, an LM that retrieve, rerank passages, and generate answers using a single model. For the retrieval task, it adopts generative retrieval and directly generates passage text. For reranking, it utilizes the generation probability as the relevance score. For answer generation, it generates answers based on the top-1 passage. Evaluation shows that Self-Retrieval outperforms previous dense and generative retrievers in retrieval tasks and achieves better EM scores in answer generation tasks.
strength_weakness: strength_weakness: 1. Generating passage content is time-consuming. This paper could analyze the latency of Self-Retrieval and compare it to other alternatives, such as generating spans (ref SEAL) or keywords (ref Term-Set Generation).

2. The proposed model includes an in-domain fine-tuned reranker, while the baseline BGE-FT + reader does not have a reranking stage. This may make the comparison unfair since reranking can significantly improve RAG results.

3. The evaluation are all based on Wikipedia. It is unclear whether the model can perform well on a corpus that is not as well-structured as Wikipedia.

4. The different steps in Self-Retrieval are independently optimized, and the `knowledge sharing and deep collaboration` effects of this consolidated model have not been validated.
questions: questions: 1. Compare the efficiency and effectiveness of using shared models versus separate models for the three steps in Self-Retrieval?
limitations: limitations: The limitations are adequately discussed.
paper_summary: paper_summary: This paper introduces Self-Retrieval, an end-to-end IR system driven entirely by a single LLM. This model integrates all essential IR functions—indexing, retrieval, and reranking—into the LLM's architecture. By internalizing the retrieval corpus through self-supervised learning, the model transforms the retrieval process into a sequence of passage generation tasks and conducts self-assessment for reranking. The authors provide experimental evidence showing that Self-Retrieval outperforms traditional sparse, dense, and generative retrieval methods on benchmark datasets like NQ and TriviaQA.
strength_weakness: strength_weakness: 1. My major concern is that the experimental settings are inconsistent with existing work, making the results unconvincing. Specifically,
  - Most existing studies conducted experiments on NQ@320k datasets, but the main experiments of this paper are conducted on NQ@40k datasets. It is important to explain the reason of this setting.
  - According to the statistics in Table 2, each document in the dataset is split into 26~29 passages, which indicates that the passages are quite short. It is known that retrieving short passages are easier for generative retrieval methods. 
  - It is necessary to explain why different models are selected for NQ@40k, TQA@40k, and NQ@320k experiments.
  - It is better to include more experimental results on the full KILT datasets as many existing studies for a fair comparison.
2. In Section 3.4, the three tasks are learned in a "1+2" manner. Please add more explanations on this design and provide experimental evaluation on other possible strategies (e.g., training the three tasks jointly)
3. Ensuring consistent use of key terms throughout the paper would improve its readability and professionalism. For example, all "large language models" should be written in "LLMs".
questions: questions: Please see my concerns in weaknesses.
limitations: limitations: n/a

Comments: comment: We sincerely appreciate your time and valuable feedback. We are pleased that our additional experimental results have addressed one of your primary concerns.

Regarding the scaling capabilities of Self-Retrieval:

1. As shown in Figure 3 of the main paper, our current experiments encompass a substantial dataset of approximately 200k documents and 3 million passages, demonstrating Self-Retrieval's capability with large-scale collections.
2. It's worth noting that in generative information retrieval research, using partial document collections is common due to training efficiency constraints. For instance, Ultron and GenRet utilized subsets of 300k documents from MS MARCO, while UniGen employed a subset of approximately 100k passages from the NQ dataset for evaluation. In contrast, Self-Retrieval conducted main experiments on 1 million passages of NQ and further expanded it to 3 million on the scaling experiment.
3. We are actively conducting experiments on the full NQ dataset. However, due to time and computational resource limitations during the rebuttal period, we were unable to complete these extensive experiments for this response.

Concerning efficiency, as mentioned in our Global Response, Self-Retrieval achieves comparable efficiency to SEAL while maintaining high performance. Moreover, Self-Retrieval requires only a single LLM to manage both retrieval and downstream LLM tasks. This unified architecture enhances deployment flexibility and potentially reduces computational overhead.

We hope these points can partially alleviate your concern about the scaling and efficiency aspects of our method. We remain committed to further large-scale experiments and are open to addressing any additional questions or concerns you may have.
Dear Reviewer uLuX,

As the discussion deadline is nearing, we wanted to gently follow up on our recent submission. We've carefully addressed your feedback and incorporated it into our revised manuscript. We provided clarifications on our choice of NQ@40K for alignment with standard passage-level experiments in dense retrieval, and explained why the "1+2" strategy is preferable to joint training in our context. 
We hope our rebuttal have addressed your concerns regarding our experimental and training settings.

Your feedback is highly valuable to us. We welcome any additional feedback or questions you may have about our revisions.

Thank you for your time and expertise.
Thank you for the response and the additional experimental results! The results addressed one of my major concerns. I'm still not quite clear about the scaling / efficiency of the proposed method. I'll keep my score unchanged.
Thanks for your time and valuable comment.
### Efficiency Analysis Details:

|Model Name|Memory|Beam Size|Latency(s)|H@5|
|---|---|---|---|---|
|SEAL|444MB|10|1.18|61.91|
|||100|5.92|59.57|
|DSI-XL|0|10|0.23|60.21|
|||100|0.45|60.21|
|Self-Retrieval|30MB|10|1.44|76.17|
|||100|6.06|81.49|
|Self-Retrieval w/o re-ranking|30MB|10|1.36|74.04|
|||100|4.57|72.97|

The latency/H@5 results in Table 1 of Global Response include the re-ranking step. We have updated the table to include results without re-ranking as well. These results demonstrate that without re-ranking, Self-Retrieval further improves efficiency while maintaining competitive performance.
### Early Stopping Mechanism Details:
When the Early Stopping Mechanism is triggered, we have typically decoded only a few tokens. Nevertheless, these tokens are sufficient to identify a unique passage. In the prefix tree, each leaf node corresponds to a specific passage ID, allowing us to map the decoded prefix directly to a passage ID. Subsequently, we extract the full passage content from the corpus and append it to the previously decoded content for re-ranking.
Thank you for your detailed reply, which addresses many of my concerns.
﻿
Regarding the comparison with DSI and SEAL, does the latency / H@5 only account for the retrieval step of Self-Retrieval, or does it also include the re-ranking step? If it includes the re-ranking step, can we still perform early stopping during passage generation, as in Eq 2, the re-ranking score seems to rely on the full passage content.

Thanks.