Self-Retrieval: End-to-End Information Retrieval with One Large Language Model
Authors: Qiaoyu Tang
Jiawei Chen
Zhuoqun Li
Bowen Yu
Yaojie Lu
ChengFu
Haiyang Yu
Hongyu Lin
Fei Huang
Ben He
Xianpei Han
Le Sun
Yongbin Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Information Retrieval
Retrieval Augmented Generation
Primary area: natural_language_processing
TL;DR: In this paper, we propose Self-Retrieval, an end-to-end LLM-driven information retrieval architecture that unifies indexing, retrieval, and reranking in a single LLM.

Abstract: The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. 
However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them.
In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture.
Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process.
Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning,  transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking.
Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation.

Self-Retrieval: End-to-End Information Retrieval with One Large Language Model
Qiaoyu Tang : Jiawei Chen 12, Zhuoqun Li Bowen Yu Yaojie Lu', Cheng Fu?, Haiyang Yu², Hongyu Lin j Fei Huang?, Ben Hel Xianpei Han " Le Sun', Yongbin Li Chinese Information Processing Laboratory. Institute of Software, Chinese Academy of Sciences 2 UUniversity Chinese Academy of Sciences
3 Alibaba Group
(tanggiaoyu2020, , htt.10001111hhhuuuu20011eeaaaaa ac, cn
,umle an.
(yubowen. srrbowen.ybb,11. ybw,,,,,,,neeee fuccheng.fu,,, buuc,yife..ypoe shuiie,, 1yb iabaaiinccom benheducas.c ac.
Abstract
The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being LLMs. This separated architecture restricts knowl- edge sharing and deep collaboration between them. In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture. Self-Retrieva unifies all essential IR functions within single LLM, leverag- ing the inherent capabilities of LLMs throughout the IR process. Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning, transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking. Experimental results demonstrate that Self- Retrieval not only outperforms existing retrieval approaches by significant mar- gin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmmented generation. B
constructed independently
Introduction
Recently, information retrieval (IR) systems and large language models (LLMs) have witnessed growing synergy, with advancements in one field driving progress in the other [[3 56]. On one hand, IR systems have proven effective augmenting LLMs and mitigating challenges such as hallucinations and outdated knowledge [22] [6]. By providing accurate, up-to-date external knowl- edge, IR systems significantly enhance the reliability and performance of LLMs. On the other hand, the powerful language understanding and generation capabilities of LLMs have been leveraged to enhance almost all components of traditional IR systems-indexing, retrieval [42 B, 26i), and rerank- ing (58, 27, HQ]. Through the integration of LLMs into the IR pipeline, these systems achieve substantially improved retrieval accuracy [557 D].
However, current IR systems typically adopt pipeline architecture where different components operate in isolation, limiting LLMs' role to specific components rather than leveraging their full
Equally Contribution.
Corresponding authors.
'The code of this work is available at ottpss///ttttbboccc/////c///e//d
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Indexing
Retrieval
Reranking
symbulize in the bible
Query: does pearl
Retrieved passage
Passage assessment
Corpus
Title: parable pearl
cun answer the quety OR Answer Kingsom
Trie
Usigats
Content: Parable of and illustrates the great value of the Kingdom of Heaven....
Internalize
Self-Retrieva L.M
Approval
paosog
Constrained Becoding
cannot answer the
Sellospervited Instances
query
Rejection response
The Self-Retrieva framework consists of three key components: (1) corpus indexing through supervie learning, (2) passage generation via constrained decoding, (3) passage rank- ing using aetesensesmmnt scoring.
Figure/ 1:
potential across the entire system. This fragmented approach creates several challenges: it hinders knowledge sharing between components, prevents deep integration of LLMs' diverse capabilities, and results in complex implementations with potentially sub-optimal performance. These limitations underscore the need for a more unified approach that fully integrates LLMs across all components of the IR system. Such an approach would not only maximize the utility of LLMs' capabilities but also simplify system implementation while potentially achieving better performance through enhanced component synergy.
In this paper, we introduce Self- Retr-Reerieval, an end-to-end information retrieval architecture driven entirely by one large language model. This integration is not trivial due to the inherent mismatch between information retrieval tasks and text generation, particularly in ensuring accurate document generation using language models. As illustrated in Figure II Self-Retrieval consolidates the sepa- rate components of an IR system indexing, retrieval, and reranking into the parameters of a single LLM. For indexing, the corpus is internalized into the LLM's parameters through self-supervised learning, enabling the model to encode and store corpus information within its internal representa- tions. During retrieval, Self-Retrieval leverages its encoded knowledge of the corpus to semantically match the input query and directly generates the relevant documents as outputs. To ensure the generated documents exactly match those in the original corpus, employ the constrained decod- ing algorithm [100, 8, 24 based on the trie of the corpus. For reranking, Belf-Retrieva performs suff-assessmet on the retrieved documents evaluate their relevance. The output score is used to rerank the retrieved passages. Moreover, for downstream tasks such estrieval-augmenn genera- tion (RAG), Self-Retrieval integrates the reader component into the model, enabling direct answer generation following retrieval. Through this end-to-end approach, Self-Retrieval fully leverages LLMs' powerful capabilities in language understanding, matching, assessment, and generation to achieve unified information retrieval.
We evaluate Self-Retrieval on three representative retrieval benchmarks: NQ. TriviaQA, and MS MARCO. Experimental results demonstrate that Self-Retrieval substantially outperforms existing sparse retrieval, dense retrieval, and generative retrieval methods on both document-level and passage-level retrieval tasks. Furthermore, our experiments on retrieval-augmentte generation tasks reveal that Self-Retrieval considerably enhances downstream performance. Additionally, larger LLMs lead to progressively better performance in Self-Retrieval showing clear scaling benefits. These results demonstrate the effectiveness of Self-Retrieval across different retrieval tasks and ap- plication scenarios.
The potential impacts of this paper may include the following aspects. First, we introduce Self- Retrieval, an end-to-end architecture that consolidates the entire information retrieval system within single large language model. This unified approach demonstrates substantial performance im- provements over existing IR methods. Second, the corpus internalization and indexing mechanism of Retr-vetrieval establishes new paradigm to memorize, organize and retrieve the learned docu- ments (at least part of them) during the pre-training phase, paving the way for more transparent and trustworthy text generation from LLMs. Third, as a LLM-driven retrieval system, Self- Retrivell of- fers inherent advantages in terms of compatibility, consistency, and interaction with L.MMs internal knowledge. Through experiments on RAG, we demonstrate how this natural compatibility leads to superior performance, suggesting broader potential for enhancing various L.M M-bsed applications.

---

2
Related Work
LLM for IR Recent studies have explored leveraging LLMs to enhance various components of IR systems, including query rewriting, retrieval, and reranking. For query rewriting, LLMs have been employed to generate pseudo-documents for query expansion [[6]] and rewrite queries based on conversational context [[S]. In the retrieval stage, researchers have explored augmenting data by generating pseudo-queries [R. [7] or relevance labels [25]] using LLMs, as well as employing LLMs directly as generative retrievers 122 [JJ. Regarding reranking, LLMs have been utilized in two ways: serving as rerankers directly [2Z. B00 and augmenting the reranking dataset [[22]. While these methods have advanced specific components within the IR pipeline, Self-Retrieval distinguishes
itself by presenting an end-to-end architecture driven entirely by a single I.I.,, eliminating the need for external components.
Dense retrieval Dense retrieval models retrieve information by matching dense vector representa- tions of queries and documents [19]. In this paradigm, an encoder transforms both queries and documents into dense vectors, with relevance determined by their vector distance. Various strate- gies have been proposed to enhance dense retrievers, including designing loss functions [A5]] multi- vector []8] training with synthetic queries [33] | and leveraging large-scale query-document pairs [50, 50J. Recent work has also explored using large language models to generate dense vec- tors for both queries and documents [29]. However, the fundamental limitation of dense retrieval lies in its limited interaction with L.Mss, as the compression of natural language into dense vectors inherently constrains the utilization of LLMs' sophisticated language understanding and semantic inference capabilities.
Generative retrieval Generative retrieval methods leverage els to generate document identifiers for given query [8] [2]. This paradigm is pioneered by GENRE [7]] which introduces the concept of entity retrieval through constrained beam search gen- eration of entity names. DSI [[2] extends it to document retrieval by training T5 models to generate document-specifie identifiers. The field has since evolved through various innovations, including query generation techniques [M 55),, sophisticated identifier design [4X] 51]]. architectural improve-
to-sequenee language mod-
ments [5], (56], and continual learning strategies [211,
Most relevant to our work, Yu et al.[52] proposed a "generateethen-readd" approach, advocating for the use of LLMs directly generate documents instead of relying on a retriever. UniGen [2S] proposed a unified framework that integrates generative retrieval and question answering through dual-decoder architecture. Compared them, Self-Retrieva ensures accurate document generation through constrained decoding and accomplishes both retrieval and answer generation in one turn.
The main distinctions between Retrivel and existing generative retrieval methods be sum- marized as follows: (1) Self-Retrieval enables LLMS to directly generate document content rather than relying on other text numeric identifiers. This approach aligns naturally with LLMs' pre- training objectives, preserves their inherent knowledge, and eliminates the need for complex iden- tifier construction schemes. (2) Self-Retrieval further integrates components such as reranking and answer generation into the framework, further expanding its scope and enhancing the retrieval per- formance. These distinctions highlight that Self-Retries represents a more natural and effective approach for leveraging the capabilities of LLMs in information retrieval.
3 Self-Retrieval
In this section, we introduce our proposed Self-Retrieval. The overall architecture is illustrated in Figure M1 Different from traditional information retrieval systems that separate indexing, retrieval, and reranking components, Self-Retrieval integrates these functionalities directly into the parameters of single large language model:
Indexing:
Self-Retrieval internalizes the entire corpus into its parameters through self- supervised learning, enabling the model to process passages internally without relying on externa indices.
Retrieval: Given an input query g, Self-Retrieval generates relevant passage using the knowledge embedded within parameters, which different from dense retrieval or gen- erative retrieval that rely embedding or document identifiers proxies of passage.
3

---

Reranking: After generating passage p, Self-Retrieval assesses its relevance to the query q through self-assessment The output logits provide the basis for reranking candidate passages.
Through this unified approach, Self-Retrieval enables streamlined, end-to-end process that en- hances the overall effectiivnness of information retrieval. In the following sections, detail each component of our method.
3.1 Indexing: Internalize the Corpus
Self-Retrieval integrates indexing into the LLM's parameters through self-supervised learning, en- abling the model to internalize the entire corpus. Unlike generative retrieval methods that rely on complex document identifiers and identifier matching, Self-Retrieval employs a straightforward sentence-to-passae task to construct the index. Specifically, given a passage p = {81, 88....,,... $L) consisting of L sentences, each sentence s; is provided as input the LLM with parameters The training objective is to generate the source passage in an auto-regressive way, represented as P(p)s;,0) 0). This self-supervised indexing approach offers several advantages. First, it provides simple yet effective method for corpus indexing. Second, it naturally frames the indexing process as a retrieval-Iie like task, enabling the model to simultaneously internalize the corpus and develop retrieval capabilities using consistent data format. Furthermore, this indexing technique closely aligns with the pre-training processes language models, suggesting that our method could be con- sidered as continued pre-training on the corpus. Through this process, the LLM learns to efficiently memorize and organize corpus information within its parameters.
3.2 Retrieval: Generate Relevant Passage through Constrained Decoding
Retrieval serves as a first-pass filter to collect passages related to the input query. In Self-Retrieval, we train the LLM to directly generate relevant passages in response to queries, climinating the need for intermediaries such as embedding in dense retrieval or document identifier in generative retrieval. Specifically, given the query q and corpus D, Self-Retrieval first generates a potential document title f as global information, formulated as P(flg: 0). The model then generates relevant passage, denoted as P(p f)
However, since I.LMs are general-purpose pre-trained models rather than statistical frequency mod- els, the generated passage p may not exactly match any passage in D, making it challenging to locate the corresponding passages in the corpus. To address this challenge, we employ a brse-based con- strained decoding algorithm [IIC. g. [4]] This approach restricts generated tokens to a dynamically constrained vocabulary. We construct a prefix tree T from corpus D, where each path from the root leaf node represents a unique passage the corpus, and each node stores valid tokens for the next generation step. During inference, the vocabulary at each generation step is constrained by the valid continuations in the prefix tree. Due the relatively short common prefixes among docu- ments, the LLM terminates generation once it has produced sufficient tokens to uniquely identify the current document and concatenates the full document to the context. This results document title and passage generation processes represented as P(i]g: 0; T) and P(glg, f; 0; T). This mechanism ensures that generated passages align with existing corpus content.
3.3 Reranking: Assess the Relevance
Reranking serves as second-pass filter to precisely sort the retrieved passages based on the rele- vance to the query. We implement a self-assessment mechanism that leverages the Self-Retrieval model itself to evaluate the relevance of generated passages. Specifically, Self- assesses the passage relevance by generating responses such as "can answer the query" for relevant passages and "cannot answer the query" for irrelevant ones. This self-assessment mechanism allows the model to generate passages and evaluate their relevance within single inference turn.
During training, we utilize the gold passage from the supervision data as the positive instance, while sampling negative instances from both the same and different documents. This training strategy conditions the L.M to accurately discern and verify the relevance of its outputs, thereby enhancing its autonomous relevance assessment capabilities and improving the overall precision of the retrieval process.

---

During inference, the overall relevance score S is composed of the document title score S and the self-assessment score SP. Specifically,, the document title score derived from the title generation probability, while the self-assessment score is calculated based on the probability of the language model rejecting the passage. Formally, for a set of generated titles and passages ((11,P)). (12.22... (un.Pr)) the title score for each (1,, Pc) is given by:
S
and the assessment score is:
S Softmax ( (1 P(rejection response gg P:: 0))/6)
(2)
where r and are temperature parameters used to scale the logits. Based on preliminary experiments on the development set, simply set
5 = 0.4 for the main passage retrieval experiments.
The final relevance score computed as the product of these two components:
S S SP
This combined score is then used to rerank the passage set, producing more refined ordering based on relevance.
3.4 Training & Inference
Training Self-Retrieval unifies the three distinct tasks of information retrieval - innlliindd.,.... retrieval, and reranking into text generation tasks, trained using cross-entropy loss in an auto-regressive manner. Specifically, Self-Retrieva first internalizes the corpus into its parameters through self- supervised learning as introduced in Section Gill Subsequently, in addition to a portion of self- supervised instances, incorporates two different types of data to build retrieval and reranking abilities:
Utilizes supervised query-passage pairs from the dataset, where the model learns to generate both document titles and passage content in response to input queries.
Retrieval data:
Reranking data: Employs positive and negative examples to train the model in relevance assessment between queries and passages.
This auto-regressive training approach enables Self-Retrieval to integrate traditionally separate IR components into unified language model, establishing an end-to-end IR system.
Furthermore, leveraging the universal language generation capabilities of LLMs, we can seamlessly integrate downstream task components, such readers in RAG, into tion can be achieved by simply appending the golden answer after the assessment in self-Retrieval. Consequently, the LLM function comprehensive RAG system, effectively reducing the knowledge gap between IR system and reader modules.
This integra-
Inference During inference, given input query, Self-Retrieval aims obtain the relevant pas- sages that are sorted based on the relevance to query. Firstly, the model generates document titles through constrained beam search. Secondly, for each title, it generates j passages using beam search. Finally, the resulting ixj passages are scored using the self-assessment mechanism and reranked to produce the final output.
4 Experimental Results
4.1 Experimental Setup
Datasets and metrics We conduct main experiments on Natural Questions (NQ) [2I]] and Trivi- aQA [[]] datasets, both of which are widely used retrieval benchmarks based on Wikipedia. We use their versions from the KILT benchmark [[=]], which consolidates these datasets into a single pre-processed Wikipedia dump, facilitating easier evaluation. Since the KILTT test set is not publicly accessible, we use the development set for testing and randomly sample 2,000 instances from the
training set as our development set. For our experiments, sample approximately 40K documents
5

---

NQ
TriviaQA
Model
Params
H@l
H@5
M@5
H@1
H@5
M@5
Sparse Retrieval
BM25 [7]
14.54
32.71
21.13
20.09
42.73
28.35
Dense Retrieval
DPR [19]
110M
40.41
61.79
48.80
35.57
57.39
43.93
DPR-FT
110M
42.21
60.45
49.33
36.58
53.05
42.91
BGE [50]]
335M
36.30
66.95
48.05
46.97
70.14
55,95
BGE-FT [50]]
335M
53.42
80.15
63.99
52.70
75.22
61.65
BGE-FT L.GE-Reranke-FPPe
770M
52.15
76.15
61.37
44.87
67.39
53.39
GTR-XL. [52]]
1.24B
37.64
66.84
48.94
35.97
63.75
46.67
GTR-XI. BGF-Reranker-FT
1.57B
57.50
78.92
66.06
58.56
77.65
66.22
GTR-XXL [[22]
4.86B
39.21
69.72
50.88
35.97
64.15
46.83
(ex-eembedding-ata--222
34.28
62,28
44.64
35.09
62.00)
45.15
GritLM [29]]
7.24B
44.67
76.00
57.03
39.91
69.34
51.14
GritLM BGE-Reranker-FT
7.57B
57.57
81.35
66.98
58.60
80.54
67.21
Generative retrieval
DSI-XL 10
2.85B
43.03
60.26
49.47
29.64
46.74
36.12
DSI-XXL 22
11.3B
43.81
60.45
50.20
30.55
46.67
36.56
SEAL [
406M
36.79
61.35
45.88
36.88
61.66
46.29
DSI-QG
550
2.85B
34.88
56.60
43.33
29.15
45.53
35.20
NCI
1.07B
50.86
70.27
58.53
28.42
42.18
33.62
Self-Retrieval (StableLM)
2.8B
62.16"
79.28
69.45*
58.69"
78.39"
66.72"
Self-Retrieval (Llama 2)
6.74B
63.44"
79.29
70.00"
59.94"
81.06"
68.74"
Table 1: The experimental results of passage retrieval NQ and TriviaQA test set. statistically significant improvements (p < 0.01) over state-of-the-an retrieval baselines.
indicates
from Wikipedia for each dataset. Each document is segmented into passages of maximum 200 words, yielding approximately million passages in total. The detailed statistics of the datasets presented in Appendix A We use passage-level Hits (1, 5) and Mean Reciprocal Rank (MRR) @5 evaluation metries.
To comprehensively compare with other generative information retrieval methods, we also conduct experiments document retrieval. Following NCI [49], conduct experiments NQ320K and utilize Recall@ (1, 10) and MRR 100 as the evaluation metrics. evaluate the model's robust- ness in non- Wikipedia scenarios where high-quality text and titles not available, conduct experiments on a subset of MS MARCO [[]] following the experimental setup of Ultron [53]] The performance was measured using Recall (1,5) and MRR @ 10.
Implementation details In this study, we employ StableLM-38 3B [4 and Llama2-74 7B [ as pas- sage retrieval backbones. For document retrieval, we employ StableLM-1.66 [[]] for NQ320K and StableI MM33 for MS MARCO. We train the models using 7eRO stage-2 optimization per GPU, and BF preci-
8 NVIDIA
A100 (80 GB) GPUs with the AdamW optimizer, batch size sion. The models are trained for 3 epochs with learning rate of 2e-5. During inference, we use beam search to generate titles and passages for each title, with hyperparameters т and set to 0.4 across all models and datasets.
Baselines We evaluate Self-Retrieval models for both passage retrieval and document retrieval, comparing them with sparse, dense, and generative retrieval baselines. The sparse retrieval base- lines are: [Во] and D5crryuery [28]. The dense retrieval baselines include: DPR [[T]] Sentence-T5 [[T], GTR [32], BGE [SQ], lext-embedding-adae
[BO]] GritLM [29]], and their fine-
tuned variants, DPR-FT and BGE-FT. The generative retrieval baselines comprise: DSI [22]] DSI- QG [59]], NCI [ Ultron [|رك], DynamicRetriever [[53]], GenRet [39]] and SEAL [5]] Additionally, to ensure a comprehensive comparison, we also evaluate combinations of strong retrieval baselines with various rerankers, including BGE-Reranker. BGE-Reranker-FT, and RankGPT [АI]]. In passage retrieval task, we use the official pre-trained models for all non-fine-tuned dense retrieval baselines. For fine-tuned dense models and generative models, we use their official implementations to replicate the experiments our dataset. In the document retrieval task, report the baseline

---

performances from their original paper. For comprehensive details about these baselines, please refer Appendix B
4.2 Main Results
Passage retrieval In Table we compare the performance of Self-Retrieval with various baselines on the NQ and TriviaQA datasets. Self-Retrieval 3B outperforms both strong pre-trained dense retrieval models, such as BGE and GritLM 7B, and other generative retrieval methods. Specifically, Seff-Retrieval 3B achieves improvements of 5.46 and 5.07 in MRR MRR@5 over the fine-tuned BGE NQ and TriviaQA datasets, respectively.
Our results indicate that other generative retrieval baselines exhibit suboptimal performance on pas- sage retrieval. Even the largest DSI-XXI. model only achieves an MRR@5 of 50.20 on NQ, sig- nificantly lagging behind dense retrieval methods such as GritLM, which achieves an MRR @5 of 57.03. In contrast, our Self-Retriewa model demonstrates strong performance in passage retrieval, achieving an MRR @5 of 69,45, significantly outperforming all other generative methods.
We further compare Self-Retrieval with conventional 2-stage retriever-reranker pipeline. Represen- tative results are shown in Table MI while the complete experimental results are provided in Ap- pendix D Notably, even strong retrieval baselines (BGE-FT, GTR-XL, XL, GritLM, and DSI-XL) en- hanced with powerful rerankers (such as SEGE-Reranker-FT still fall short of Self-Retrieval's perfor- mance, highlighting the advantages of unifying multiple retrieval processes into single framework rather than treating them as separate components.
These findings underscore the efficacy of Relf-Retrieval in harnessing the memory, generation, and ranking capabilities of LLMs, thereby excelling in passage retrieval tasks where other generative baselines struggle.
Method
R@L
R@ 10
M@ 100
Method
R@1
R@5
M@
Sparse Retrieval
Sparse Retrieval
BM25 [5%]]
29.7
60.3
40.2
BM25 [EV]]
18.9
42.8
29.2
DocT5Query [X
38.0
69.3
48.9
DocT5Query [2]]
23.3
49.4
34.8
Dense Retrieval
Dense Retrieval
DPR [|]]]]
50.2
77.7
59.9
DPR
29.1
62.8
43.4
Sentence- TS [1
53.6
83.0
64.1
Sentence-
27.3
58.9
40.7
-IR-Base
[22
56.0
84.4
66.2
Generative Retrieval
Generative Retrieval
DSI-Atomic [+
32.5
63.0
44.3
DSI [22]]
55.2
67.4
59.6
DynamicRetriever [54]
29.0
64.2
42.5
SEAL
59.9
81.2
67.7
Ultron-URD [
29.6
56.4
40.0
DSI-QG [59]
63.1
80.7
69.5
Ultron-PQ [55]
31.6
64.0
45.3
NCI [+5]
66.4
85.7
73.6
Ultron-Atomic [5.]]
32.8
64.9
46,9
GenRet [5]
68.1
88.8
75.9
GenRet [00]
47.9
58.1
Self-Retrieval
73.3
92.6
80.7
Self Relf-Retrieval
47.8
69.9
57.2
Table 2: The experimental result of document retrieval on NQ320K.
Table 3: The experimental result of document retrieval on MS MARCO.
Document retrieval We present the document retrieval results NQ320K dataset in Table Self-Retrieval outperforms all other generative retrieval methods and dense retrieval baselines across all three metries. Compared GenRet, the previously strongest generative retrieval method, Self- Retrieval improves @its I by 5.2, Hits@ 10 by 3.8, and MRR@ @ 100 by 4.8 points. Notably, while other methods commonly employ query generation augment their training data, Self-Retrieval achieves these results using only the original training set.
To evaluate the effectiveness of Self-Retrieval in Wikiiiiied scenarios, extend exper- iments to MS MARCO. To address the absence of document titles in MS MARCO, we employ Llama2 automatically generate titles. As shown in Table EL Self-Retrieval achieves comparable performance to the SOTA model GenRet, while significantly outperforming other baselines. These results demonstrate its adaptability and robustness in -ikiiiiipe and title-lacking contexts.
Ablation study To study the effect each component, we conduct ablation study on both NQ and TriviaQA. Results presented in Table M. All components prove crucial for Self-Retrieval's
7

---

NQ
TriviaQA
Method
H@1
H@5
M@5
H@1
H@5
M@5
Self- ReffeRetrieval (base)
62.16
79.28
69.45
58.69
78.39
66.72
w/o indexing
53.05
67.16
58.95
54.45
70.64
60,98
w/o title
47.81
60.90
52.81
52.32
67.91
58.48
w/o self-assessment
54.80
75.21
62.77
46.67
70.79
55,92
Table 4: Ablation study NQ and TriviaQA.
performance, with each ablation resulting in substantial performance degradation. Specifically, re- moving the indexing mechanism restricts the model to internalizing only the documents encountered during training, leading poor performance erate passages with constrained decoding. The absence document titles significantly degrades performance, titles provide critical global information that guides the LLM in generating relevant content. Furthermore, removing the self-assessment mechanism leads to significant decrease in both datasets. Without self-assessment, the model cannot effectively evaluate and refine its initial retrieved passages, leading to less accurate document rankings. This degradation directly impacts downstream applications such as RAG, where precise passage ranking is crucial for generating high- quality responses. These ablation results show that each component of Self-Ro Retrievll addresses specific challenge the retrieval process, contributing its overall effectiveness.
unseen passages. Without titles, directly gen-
4.3 Performance on Retrieval-Augmeente Generation
NQ
TriviaQA
10K
40K
10K
40K
BGE-FT + MtableLM-FT
43.18
41.24
56.79
58.15
Relf-Retrieval 3B
44.62
46.11
64.03
62.69
BGE-FT++ Llama2-FT
49.10
49.24
61.79
61.72
Self-Retrieval 7B
53.26
52.98
72.14
70.40
Table 5: The performance retrieval-augmenter generation. For baseline, use BGE-FT as the retriever and a fine-tuned LLM as reader. Results are reported using Exact Match (EM) scores.
The end-to-end architecture of Self-Retrieva seamlessly integrates retrieval and answer generation into single inference process. To evaluate its effectiveness in RAG, compare Self- Retrieval mod- els with a strong baseline that combines BGE-FT for retrieval and fine-tuned versions of StableLM- readers. We conduct experiments
3B and LLaMA2-7B 10K and 40K documents for each dataset. We utilize the top-1 retrieved passage as the context and measure performance using the Exact Match (EM) metric. As shown in Table EL Self- Retr-veerieval sig- nificantly outperforms the baseline on both datasets across different model scales. Unlike other RAG pipelines that separate retrieval and generation, Self-Retrieval integrates the entire process within the LLM framework, enabling more accurate and coherent responses through end-to-end modeling.
subsets of NQ and TriviaQA using
4.4 Detailed Analysis
Scaling model capacity To explore the impact of model scale on retrieval performance, we eval- uate Self-Retrieval with various backbone models of different sizes, including StableLM (1.6B, 3B) [H) 84], Llama2 (7B, 13B) [H3], and Qwen-1.5 (4B, 7B, 14B) [2]. Figure 0 presents the results on NQ, showing that Self-Retrieval's retrieval performance benefits from the general capabilities of larger language models. For models within the same series, as the model size increases, we observe consistent improvements in both Hits and @5 indicating strong scaling properties of the Self-Retrieval architecture.
Scaling corpus size Recent studies 35. 5 have demonstrated that generative retrieval methods such as DSI or NCI experience more significant performance degradation compared dense re- trieval methods when scaled to larger corpora. To explore the impact of corpus size on Self-Retrieval,

---

MtableLM
Llama2
Qwen1.5
Original
BGB-Reranker
Self-
88
80
Hiss@
86
70
MRR@S
60
50
Hits@l
68
66
40
64
30
1,6B 3B
7B
14B
DPR-FT
SEAL Retriever
GritLM
# Params
Figure
Impact of model capacity on Self- Retrieval performance.
Figure 3:
Reranking performance comparison
when processing top-100 passages.
BGE-FT H60
BGE-FT Hog5
Self-Retrieval H@!
Selt- Self-Reetieval H6p5
90
90
70
70
60
60
10k
40k
100k
200k
10k
40k
70k
100k
200k
Num of Documents
Num of Documents
(a) Scaling results on NQ.
(b) Scaling results on TriviaQA.
Figure 4: Scalability analysis of retrieval performance for Self Retrivetiievl and BGE-FT across varying corpus sizes.
expand experiments from 10K to 200K documents, scaling the number of passages from 290K to 3M. Figure g illustrates the performance trends of BGE-FT and our Self-Retrieval 3B model the NQ and TriviaQA datasets with increasing corpus sizes. While both models show performance decrease with larger corpus sizes, Self-Retrieval maintains a degradation rate compara- ble BGE-FT. As the number of documents continues to increase, the degradation rate gradually diminishes, demonstrating Self-Retrieval's potential scalability to larger document collections. This observation indicates that Relf-Retrieval effectively addresses some of the inherent limitations of generative retrieval approaches in large-scale scenarios.
Analysis on reranking In this part, we conduct an in-depth analysis of the reranking performance of Self-Retrieval reranker module in comparison with the fine-tuned BGE-Reranker. We employ DPR-FT, SEAL and GritLM retrieve 100 passages on TriviaQA, followed by reranking the re- trieved results using both approaches. We evaluate performance using MRR@5 as the metric. The experimental results are presented in Figure The results reveal two key findings: (1) Reranking plays crucial role in information retrieval systems, significantly enhancing the ranking perfor- mance across all models. (2) The Relf-Retrieval reranker consistently outperforms the fine-tuned BGE Reranker in most scenarios, demonstrating its robustness and effectiveness. These findings demonstrate that Relf-Retrieval performs effectively both as a complete system and reranker component.

---

In Appendix C we conduct additional experiments with a chunk size of 100 words, demonstrating Self- &elf-Retrieval's adaptability to different text segmentation strategies. Appendix H. we further discuss Self- Retrivall' computational efficiency.
5 Conclusion
In this paper, we propose Self-Retrieval, an end-to-end LLM-driven information retrieval architec- ture that unifies indexing, retrieval, and reranking in single LLM. This approach enables the LLM to internalize the corpus, generate relevant content, and perform self-assessment within a unified framework. Unlike previous works that incorporate LLMs into individual IR components, Self- Retrieval provides a unified framework for the entire IR procedure, facilitating knowledge sharing and deep collaboration among different components. Experimental results demonstrate that Self- Retrieval achieves strong performance across various retrieval benchmarks and application scenarios. In future work, we plan to extend our method to further enhance the reliability and trustworthiness of LLM generation.
Limitations
While our experiments demonstrate the effectiveness of Self-Retrieval, several limitations need to be addressed in future work. Our current evaluation is limited to 200K Wikipedia documents and 3M passages, and testing on larger and noisier text collections needed. As an LLM-driven system, Relf-Retrieval has lower retrieval efficiency compared to sparse or dense retrieval methods, which may limit its applications to specialized knowledge systems. Furthermore, enabling incremental learning and dynamic corpus expansion remains an important direction for future research.
Acknowledge
We sincerely thank the reviewers for their insightful comments and valuable suggestions. We are grateful to Le Yu and Xinyu Lu for their helpful feedback on the paper writing. This work was supported by the Natural Science Foundation of China (No. 62 62122077, 62272439), Beijing Mu- nicipal Science and Technology Project (Nos. Z231 100010323002), the Basic Research Program of ISCAS (ISCAS-JCZD-2022) 202333,,,, and CAS Project for Young Scientists in Basic Research (Grant No.YSBR-0400
References
[1] Qingyao Ai, Ting Bai, Zhao Cao. Yi Chang, Jiawei Chen, Zhumin Chen, Zhiyong Cheng, Shoubin Dong, Zhicheng Dou, Fuli Feng, Shengling Gao, J. Guo, Xiangnan He, Yanyan Lan, Chenliang Li, Yiqun Liu, Ziyu Lyu, Weizhi Ma, Jun Ma, Zhaochun Ren, Pengjie Ren, Zhiqiang Wang, Min Wang, Jirong Wen, Lei Wu, Xin Xin, Jun Xu, Dawei Yin, Peng Zhang, Fan Zhang, Wei Zhang, M. Zhang, and Xiaofei Zhu. Information retrieval meets large language models: strategic report from chinese ir community. ArXiv, abs/2307.0977 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2 2023.
[3] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human gener- ated machine reading comprehension dataset. arXiv preprint arXiv: 101i::66110000000 1.09268, 2016.
[4] Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, Meng Lee,
10

---

Emad Mostaque, Michael Pieler, Nikhil Pinnaparju, Paulo Rocha, Harry Saini. Hannah Teufel, Niccolo Zanichelli, and Carlos Riquelme. Stable Im 2 1.6b technical report, 2024.
[5] Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. Autoregressive search engines: Generating substrings as document identifiers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 31668 31683, Curran Associates, Inc., 2022.
[6] Luiz Henrique Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. In- pars: Data augmentation for information retrieval using large language models. ArXiv, abs/2202.05144 2022.
[7] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. ArXiv, abs/2010.00904, 2020.
[8] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In International Conference on Learning Representations, 2021.
[9] Jiangui Chen, Ruqing Zhang, J. Guo, Y. Liu, Yixing Fan, and Xueqi Cheng. Corpusbrain: Pre- train a generative retrieval model for knowledge-nttensive language tasks. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022.
[10] Pinzhen Chen, Nikolay Bogoychev, Kenneth Heafield, and Faheem Kirefu. Parallel sentence mining by constrained decoding. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Compula- tional Linguistics, pages 1672-1678, Online, July 2020. Association for Computational Lin- guistics.
[11] David R. Cheriton. From doc2query docttttiquery. 2019.
[12] Fernando Ferraretto, Thiago Laitz, Roberto de Alencar Lotufo, and Rodrigo Nogueira. Exaranker: Synthetic explanations improve neural rankers. Proceedings of the 46th Interna- tional ACM SIGIR Conference Research and Development in Information Retrieval, 2023.
[13] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval -uggmette generation for large language models: A survey, 2024.
[14] Jiafeng Guo. Changjiang Zhou, Ruqing Zhang, Jiangui Chen, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. Corpusbrain+4: continual generative pre-training framework for Knowledge-intensive language tasks. ArXiv, abs/2402.16667, 2024.
[15] Chao-Wei Wei Huang, Chen- Yu Hsu, Tsung- Yuan Hsu, Chen-An Li, and Yun-Nung (Vivian) Chen. Converser: Few-shot conversational dense retrieval with synthetic data generation. ArXiv, abs/2309.06748,, 2023.
[16] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-43, 2023.
[17] Vitor Jeronymo, Luiz Henrique Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto de Alen- car Lotufo, Jakub Zavrel, and Rodrigo Nogueira. Inpars-v2: Large language models as efficient dataset generators for information retrieval. ArXiv, abs/2301.01822 2023.
[18] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 1601-1611. 1, Vancouver, Canada, 2017. Association for Computational Linguisties.
11

---

[19] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wan-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769- 6781, Online, November 2020. Association for Computational Linguistics.
[20] Varsha Kishore, Chao gang Wan, Justin Lovelace, Yoav Artzi, and Kilian Q. Weinberger. In- cdsi: Incrementally updatable document retrieval. ArXiv, abs/2307.10323, 2023.
[21] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transac- tions of the Association of Computational Linguistics, 2019.
[22] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmmnned generation for knowledge-intensing H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing Systems, volume 33, pages 9459-9474. Curran Associates, Inc., 2020.
nlp tasks. In
[23] Xiaoxi Li, Yujia Zhou, and Zhicheng Dou. Unigen: A unified generative framework for re- trieval and question answering with large language models. ArXiv, abs/2312. 10036, 2023.
[24] Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. Text2Event: Controllable sequence-fo-strucctre generation for end-to-end event extraction. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2795-2806, Online, August 2021. Association for Computational Linguisties.
[25] Guangyuan Ma. Xing Wu, Peng Wang, Zijia Lin, and Songlin Hu. Pre-training with large lan- guage model-based document expansion for dense passage retrieval. ArXiv, abs/2308.08228, 2023.
[26] Xueguang Ma, Liang Wang. Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. ArXiv, abs/2310.008399 2023.
[27] Xueguang Ma, Xinyu Crystina Zhang, Ronak Pradeep, and Jimmy J. Lin. Zero-shot listwise document reranking with large language model. ArXiv, abs/22305.02166. 2023.
[28] Antonio Mallia, O. Khattab, Nicola Tonellotto, and Torsten Suel. Learning passage impacts for inverted indexes. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
[29] Niklas Muennighoff, Hongjin Su, Liang Wang. Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kicla. Generative representational instruction tuning, 2024.
[30] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201. 10005, 2022.
[31] Jianmo Ni, Gustavo Hernández Abrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Matthew Cer, and Yinfei Yang. Sentence-15: Scalable sentence encoders from pre-trained text-to-text models, ArXiv, abs/2108. 001.100...8777. 2021.
[32] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming- Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844-9855. Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics.
12

---

[33] Rodrigo Nogueira, Jimmy Lin, and Al Epistemic. From doc2query to docttitiquery. Online preprint, 6:2, 2019.
[34] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jemnite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online, June 2021. Association for Computational Linguistics.
[35] Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Met- zler, and Vinh Tran. How does generative retrieval scale to millions of passages? In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, pages 1305-1321, Singapore, December 2023. Association for Computational Linguistics.
[36] Shanbao Qiao, Xuebing Liu, and Seung-Hoon Na. Diffusionret: Diffusion-enhancee gener- ative retriever using constrained decoding. In Conference on Empirical Methods in Natural Language Processing, 2023.
[37] Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389 2009.
[38] Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715-3734, Seattle, United States, July 2022. Association for Computational Linguistics.
[39] Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang. Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. Learning to tokenize for generative retrieval. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[40] Weiwei Sun, Lingyong Yan, Xinyu Ma. Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agent. ArXiv, abs/2304.09542, 2023.
[41] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14918-149377 2023.
[42] Yi Tay, Vinh Tran, Mostafa Dehghani. Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W Cohen, and Donald Metzler. Transformer memory as a differentiable search index. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 21831-21844 Curran Associates, Inc., 2022.
[43] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (.XXx:2307.09288 2023.
[44] Jonathan Tow, Marco Bellagente, Dakota Mahan, and Carlos Riquelme. Stablelm 3b 4elt.
[45]
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. SimLM: Pre-traie traiiiiiing with representation bottleneck for dense passage retrieval. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 2258,, Toronto, Canada, July 2023. Association for Computational Lin- guistics.
13

---

[46] Liang Wang. Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
[47] Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 9414-9422 Singapore, December 2023. Association for Computational Linguistics.
[48] Yujing Wang, Ying Hou, Hong Wang. Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. A neural corpus indexer for document retrieval. ArXiv, abs/2200.007443 2022.
[49] Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. A neural corpus indexer for document retrieval. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Pro- cessing Systems, volume 35, pages 25600-25614 Curran Associates, Inc., 2022.
[50] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[51] Tianchi Yang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, and Qi Zhang. Auto search indexer for and-to-end document retrieval. In Conference on Empirical Methods in Natural Language Processing, 2023.
[52] Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal. Chen- guang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. In International Conference for Learning Representa- tion (ICLR), 2023.
[53] Peiwen Yuan, Xinglin Wang, Shaoxiong Feng, Boyuan Pan, Yiwei Li, Heda Wang, Xupeng Miao, and Kan Li. Generative dense retrieval: Memory can be burden. arXiv preprint 24X1v:2401.10487, 2024.
[54] Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, and Ji- Ji-Rong Wen. Dynamicretriever: pre-training modell-ased ir system with neither sparse nor dense index. arXiv preprint 22X1v:2203.00537, 2022.
[55] Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji-Rong Wen. Ultron: An ultimate retriever corpus with model-based indexer. arXiv preprint 27X.072208.09257, 2022.
[56] Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey, 2024.
[57] Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji rong Wen. Large language models for information retrieval: A survey. ArXiv, abs/2308.071077 2023.
[58] Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. Rankt5: Fine-tuning 15 for text ranking with ranking losses. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.
[59] Zhungy,
Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. Bridging the gap between indexing and retrieval for differentiable search index with query generation. arXiv preprint arXiv:2206.10122 2022.
14

---

A Dataset Statistics
Table B presents the statistics of the NQ and TriviaQA datasets used in our experiments.
Natural Questions 40K
TriviaQA
Dataset
10K
10K
40K
# doc
10,000
37,202
10,000
38,399
psg
291,506
979,804
390,586
1,193,047
# train 32,163
72.716
29,038
51,166
dev 2,000
2,000
2,000
2,000
# test 2,837
2,837
5,355
5,355
Table 6: Statistics of the experimental datasets. #doc/#psg denotes number of documents/pussages: #train/#de//Hestt denotes size of training/develomeenttsss set. Training instances without query- document pairs are removed.
B
Baselines
The sparse retrieval baselines are as follows:
BM25 [Z7]] classical sparse retrieval algorithm based on probabilistic relevance frame- work and term frequency statistics.
DocT5Query [28] expands documents by generating potential queries using fine-tuned
TS model.
The dense retrieval baselines are as follows:
DPR [ is a dual-encoder model trained with in-batch negative sampling. We fine-tune DPR on our training datasets to obtain DPR-FT, following the official implementation and hyperparameter settings.
BGE [60]] a state-of-the-art universal embedding model trained on approximately 200 million text pairs using contrastive learning. We employ the bge-large-en-v1.5 variant and fine-tune on our training datasets to obtain BGE-FT. The fine-tuning process uses learning rate of le-5, batch size of 128, and runs for 10 epochs.
Sentence-T5 [+1] employs dual-encoder T5 architecture to generate semantic embed-
dings through contrastive learning for efficient retrieval.
h is dense retrieval model based Sentence-T5, pre-trained on billions of question-answer pairs.
GTR-XL
Text-embedding-ada-O00 is powerful embedding model developed by OpenAI, accessi- ble through their API service.
GritLM [2%]] built upon the Mistral 7B language model and optimized using both em- bedding and generation objectives.
The generative retrieval baselines are as follows:
DSI 42 is sequennceto--sequeece model that directly maps queries to document identi-
fiers.
DSI-QG [59] enhances the DSI framework by incorporating doc2query model for dataset
augmentation.
SEAL [5]] utilizes n-gram as the document identifiers and constrains the generation process using FM-index.
NCI [ employs a sequence-to-sequence architecture with prefix-aware weight-adaptive decoder. We train the model using T5-Large for document- level retrieval following the official implementation. To obtain passage-level results, we further incorporate finc-tuned BGE reranker Oogereraaneer-large)
OCI+BGE-Rerankeee
15

---

Ultron [[5] represents documents using three types of identifiers (URL, PQ, Atomic) and trains the model through a progressive three-stage pipeline.
DynamicRetriever [ parameterizes traditional static indices by embedding both token- level and document-level corpus information into pre-trained model for dynamic docu- ment identifier generation.
GenRet [69] employs discrete auto-encoding with progressive training and clustering tech- niques learn semantic document identifiers for generative retrieval.
C Ablation on Chunk Size
To investigate the potential impact of chunk size, we conduct additional experiments comparing Self- Retrieval with strong baselines the NQ dataset using chunk size of 100 words, complementing our main experiments where chunk size set to 200. The experimental results are presented in Table @ It demonstrates that Self-Retrieval significantly outperforms the basclines with both chunk sizes settings, further validating the effectiveness our proposed method.
Model
Params
Hits@1
Hits
MRR@5
BGE-FT
335M
40.79
58.92
47.76
GritLM
7B
30.95
51.36
38.77
Self Retrieval (StableLM)
3B
58.43
77.76
66.18
Table 7: Retrieval performance with chunk length of 100 words.
D Full Comparison with Retriever-Reranker Pipeline
NQ
TriviaQA
H@T
H@5
M@5
H@1
H@5
M@5
BGE-FT
53,42
80.15
63.99
52.70
75.22
61.65
BGE-FT BGE-Reranker
21.91
54.58
33.33
45.36
72.16
55.78
BGE-FT + AGEERerannerr-F
52.15
76.15
61.37
44.87
67.39
53.39
BGE-FT RankGPT
44.21
73.68
55.51
48.00
72.00
57.33
GTR-XL.
37.64
66.84
48.94
35.97
63.75
46.67
GTR-XL + GGE-RReannerr
26.39
59.96
38.50
42.41
68.42
52.51
GTR-XL PGE-Reranker-FF
57.50
78.92
66.06
58.56
77.65
66.22
GTR-XL + RankGPT
42.11
68.42
52.30
47.00
66.00
54.95
GritLM
44.67
76.00
57.03
39.91
69.34
51.14
GritLM + BGE-Reranker
30.06
65.87
43.20
43.64
70.87
54.23
GritLM FFra-eeran
57.57
81.35
66.98
58.60
80.54
67.21
GritLM RankGPT
37.89
70.53
51.19
44.00
66.00
52.70
DSI-XI.
43.03
60.26
49.47
29.64
46.74
36.12
DSI-XI. + BGE-Reranker
34.39
64.26
45.74
37.85
52.57
43.49
DSI-XL + -CEE-Reranker-FT
50.02
68.60
57.43
36.49
52.40
42.36
DSI-XL + RankGPT
49.47
73.68
59.25
39.00
52.00
44.75
Relrivetrieval (StableLM)
62.16
79.28
69.45
58.69
78.39
66.72
Self-Retrieval (Llama 2)
63.44
79.29
70.00
59.94
81.06
68.74
Table 8: Comparison between Self-Retrieval and traditional two-stage retriever-reranker pipelines.
We comprehensively evaluate Self-Retrieval against various two-stage retriever-reranker pipelines. Specifically, we construct these pipelines using state-of-the-art retrievers (BGE, GTR, GritLM, and DSI-XL) combined with three different reranking approaches: BGE reranker, fine-tuned BGE reranker, and RankGPT. As shown in Table 8, Self-Retrieval achieves superior performance com- pared to most retriever-reranker combinations, demonstrating the effectiveness of our end-to-end approach over traditional pipeline methods.
16

---

E
Efficiency Analysis
We conduct efficiency analysis on NQ dataset using an NVIDIA A100-80G GPU. Results in Table illustrate that, while delf-Retrieval requires slightly higher computational resources than DSI, it pro- vides notable performance benefits. Notably, Self-Retrieval with beam size of 10 achieves signifi- cantly higher H@5 scores compared to DSI-XL with a beam size of 100, enabling flexible trade-off between retrieval quality and computational efficiency. When compared to SEAL, which also em- ploys natural language decoding. Self-Retrieval demonstrates more efficient memory usage (30MB vs 444MB) by utilizing lightweight trie structure instead of SEAL's resource-intensive FM-Index post-pr processing mechanism. Furthermore, the efficiency of Self-Retrieval stands to benefit from ongoing developments in optimization techniques (c.g., quantization and attention acceleration) and hardware advancements.
Model Name
Memory
Beam Size
Latency (s)
Hits@5
1.18
61.9T
SEAL
444MB
5.92
59.57
10
0.23
60.21
DSI-XL
100
0.45
60.21
10
1.44
76.17
Self-Retrieval
30MB
100
6.06
81.49
Table 9; Efficiency analysis.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly explained the shortcomings of previous works in the abstract and introduction (Section 1), well as how we make improvements. We have validated these claims in the experiment section (Section 4).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have specifically written a chapter to illustrate the limitations (Section Limitations) of our method and the factors that may affect its performance.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification: Our work mainly focuses on the application of large language models and does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
We provide a comprehensive description of the training and inference details for our proposed method in Section 3.4. In the experimental setup section (Section 4.1), we elaborate on the datasets used and the model training process. Additionally, we have made the complete training code available in an open-source repository, ensuring that other researchers can replicate our results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility.
19

---

reproducible, they In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have submitted the code and instructions in an open-source github reposi- tory, and can obtain the experimental results in the paper by running code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis..c/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:) www.oo/uwwiieiiioooooooiooee
//nips.cc/publl httc://bliccgguueee/eecee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide detailed description of the training and testing details in Section 4. including data partitioning, training hyperparameters. optimizer, and testing methodol- ogy.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have marked the significance analysis with an asterisk in the main experi- mental table (Table 1).
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mentioned in Section 4.1 that the experiments were conducted using 8 Nvidia A100 80GB GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///eearrggeccoooou/EEEEta
Answer: [Yes]
Justification:
We obey relevant rules in NeurlPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
21

---

Justification: We talk about positive impacts in introduction and method section, and possi- ble negative impacts in limitation part.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our experiments use publicly available base models and standard training datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cite all datasets and models used in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
22

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, haperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: All datasets are from existing public resources, and no new data assets are created.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can cither create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowd-sourcing or human annotation required in our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human participants or human subjects research, therefore no IRB approval or risk assessment required.
23

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "r1WHgXU4Re", "reviewer": "Reviewer_xB99", "summary": "The paper proposes an approach of self-retrieval, which uses the probability of generation of the passage as the ranking criterion. To limit the generation to the existing passages, a trie-structure is used, forcing the generation to produce the existing passages.\nThe experiments compared the method with several existing approaches, including sparse retrieval, dense retrieval and generation-based retrieval. The proposed method outperforms the others.", "questions": "Have you evaluated the space and time complexity of the approach, and compare it to the existing methods?\nHow do you deal with long document (in particular for the trie structure)?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of relying on the generation of an existing passage for ranking is interesting. The use of trie structure to constrain the generation to the existing documents is also nice. The proposed method thus has some novelty compared to the literature.\nThe experimental results are very good, showing improved performance on document retrieval and QA.", "weaknesses": "A key idea is the use of trie for passage generation. This is described briefly. More details should be presented. If a whole passage should be generated using trie, then the depth of trie structure will be very large (equivalent to the length of the passage). What about the storage cost of trie? What is the time efficiency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments! We are very glad to address your concerns one by one.\n\n### Details of Trie\nThe trie is pre-built based on the corpus before retrieval. Most documents/passages only share a small common prefix. The LLM stops generating once it has produced enough tokens to determine the current document. As shown in Figure 1 of the attachment file, the average decision length of the trie is around 13 tokens. For most documents, the retrieval process can decide on the current document after generating less than 20 tokens, at which point we \"early stop\" and manually append the document to the context.\n\nIn this case, the depth of the trie depends more on the number of documents/passages rather than the length of each document/passage. Therefore, even when dealing with long documents, the common prefix length between documents may not necessarily be long, allowing Self-Retrieval to handle long documents effectively.\nWe will include these details in the next version.\n### Space and Time Complexity\nPlease refer to the General Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xeR8rdcAPK", "reviewer": "Reviewer_g3kT", "summary": "This paper introduces Self-Retrieval, a new generative retrieval architecture.  Self-Retrieval first memorizes the corpus into LLM's parametric knowledge using self-supervised training. Given a query, it generates the target document with constrained decoding, then re-assess document by decoding if the document can answer the query.  On subset of NQ and TriviaQA, this approach significantly outperforms existing dual encoders and generative retrieval models.", "questions": "- Can the method scale up to the full NQ/TriviaQA? If so, would be nice to see the performance on these more standard setups. If not, what is the main bottleneck for scaling up? \n- How many candidates were considered in the reranking stage?", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Intuitive architecture for using LLM for retrieval. The paper presents a self-supervise object to help model memorize the corpus. Then it integrate retrieval and reranking into a single constrained decoding process. The \"reranking\" process can be viewed as self-critique / chain-of-thoughts, and may potentially unlock deeper integration between retrieval and reasoning.\n- Strong quality improvements. The paper reports substantial improvements over previous dual-encoder approaches and generative retrieval models.\n- Ablations shows that the model scales well with model size. Previous dense retrievers often plateau due to the bottleneck layer; the scaling curve of this new architecture is promising.", "weaknesses": "- Experiment only used wikipedia-based datasets. However, wikipedia is heavily used in pretraining, so it is unclear if the proposed approach can let model sufficiently memorize other datasets. \n- More importantly, the method relies on generating the passage title. Unlike wikipedia, many retrieval datasets do not have high-quality, natural language passage titles. It is unclear how the method works on those datasets.  It would be nice to test retrieval benchmarks like MS MARCO or BEIR/MTEB. \n- Missing dense retrieval + cross-attention reranking baselines. Such 2-staged pipeline is standard in IR. Since the proposed method's reranking stage essentially uses cross attention to judge the query and the retrieved candidate passage, the computational cost of the reranking stage is similar to that of a separate cross-attention reranker. It is fair and necessary to compare it with commonly-used rerankers such as MonoT5, RankT5, or BGE reranker.  The ablation in Table 3 seems to show that the proposed method's retrieval-alone performance is stronger than most retrieval baselines, but the paper can be more convincing if having e2e comparison to other 2-stage retrieval pipelines like BGE + BGE reranker or GTR + RankT5.\n- Lacking efficiency discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We appreciate your feedback and will address each of your points in turn.\n\n### Experiments on non-wikipedia datasets\nPlease refer to the General Response.\n### Experiments on untitled documents\nWith untitled documents or titles of poor quality, Self-Retrieval can use the generated model from other models. We conducted experiments on MS MARCO. Specifically, we prompted LLaMA3 8B in a zero-shot manner to generate appropriate titles for the documents missing the title. We tested the settings following GenRet on MS MARCO. The experimental results, shown in the General Response Table 2, indicate that titles generated by the LLM can effectively meet Self-Retrieval needs.\n\n### About retrieval+reranker baselines\nThank you for your feedback. We conducted experiments using a 2-staged retriever+reranker approach. Specifically, we chose strong retrieval baselines such as BGE, GTR, GritLM, and DSI-XL as the foundation, and then applied three different rerankers: BGE reranker, BGE reranker FT, and RankGPT. (We did not use RankT5 because the relevant code or model has not been open-sourced).\nAs shown in Attachment Table 1, our Self-Retrieval approach outperforms most retriever+reranker combinations, demonstrating the effectiveness of our method.\n\n### Efficiency discussion\nPlease refer to the General Response.\n\n### Scaling up to full NQ/TriviaQA\nIn Figure 3 of our paper, we demonstrate the performance of Self-Retrieval and BGE as the corpus size increases. Our experiments included a maximum of approximately 200k documents and 3M passages. We are currently conducting larger-scale NQ/TriviaQA experiments to further validate the scaling effects of our model. However, due to the full NQ/TriviaQA datasets involving the entire Wikipedia corpus (approximately 21M passages for the DPR version and 5.9M documents for the KILT version), it is challenging to complete training within the effective rebuttal period. We appreciate your insightful comment and plan to include more scaling experiments in the future.\n\n### Number of candidates in the reranking stage\nIn the reranking stage, we use 50 passages as candidates.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7EaZEFRp1Y", "reviewer": "Reviewer_iXLD", "summary": "This paper proposes Self-Retrieval, an LM that retrieve, rerank passages, and generate answers using a single model. For the retrieval task, it adopts generative retrieval and directly generates passage text. For reranking, it utilizes the generation probability as the relevance score. For answer generation, it generates answers based on the top-1 passage. Evaluation shows that Self-Retrieval outperforms previous dense and generative retrievers in retrieval tasks and achieves better EM scores in answer generation tasks.", "questions": "1. Compare the efficiency and effectiveness of using shared models versus separate models for the three steps in Self-Retrieval?", "limitations": "The limitations are adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Self-Retrieval consolidates the multi-step RAG pipeline with a single model. The concept is novel.\n\n2. Self-Retrieval achieves the best performance for both passage retrieval and answer generation tasks.\n\n3. Self-Retrieval shows promising results when the corpus size scales to 3 million.", "weaknesses": "1. Generating passage content is time-consuming. This paper could analyze the latency of Self-Retrieval and compare it to other alternatives, such as generating spans (ref SEAL) or keywords (ref Term-Set Generation).\n\n2. The proposed model includes an in-domain fine-tuned reranker, while the baseline BGE-FT + reader does not have a reranking stage. This may make the comparison unfair since reranking can significantly improve RAG results.\n\n3. The evaluation are all based on Wikipedia. It is unclear whether the model can perform well on a corpus that is not as well-structured as Wikipedia.\n\n4. The different steps in Self-Retrieval are independently optimized, and the `knowledge sharing and deep collaboration` effects of this consolidated model have not been validated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions on our work. Here is our response to your concerns.\n### Analyze the latency \nWe compared the efficiency of Self-Retrieval with SEAL in the General Response Table 1. Our generation process has the following characteristics:  \n1. Early Stop Mechanism: Our method includes an 'early stop' mechanism where the Self-Retrieval model ceases generation once the LLM has produced sufficient content to determine the current passage. As shown in Attachment Figure 1, the average decision length of the trie is approximately 13 tokens, which is similar to the decision length in Term-Set Generation. Consequently, the total number of tokens generated remains relatively small.\n2. No Additional Post-Processing: Unlike SEAL, our Self-Retrieval method does not require extra time for post-processing or retrieving passages. This means that even when using a larger LLM, the overall efficiency remains comparable to that of SEAL.\n\n### About reranker in RAG\nWe have augmented our experiments with the incorporation of a BGE reranker in the  RAG process (BGE-FT + BGE reranker-FT + StableLM-FT). EM scores on the NQ and TriviaQA datasets were recorded as 41.98 and 50.16, respectively. As can be seen from the results, there was no observed improvement over the BGE baseline that does not employ a reranker. This outcome is likely due to the BGE's substantial proficiency developed during the training phase. Consequently, as shown in Attachment Table 1, additional training of the reranker does not bring extra benefit.\n### Experiments on non-wikipedia datasets\nPlease refer to the General Response.\n\n### Shared Model v.s. Separated Models\nWe separately trained individual retrieval and ranking models, achieving Hit@1 and Hit@5 scores of 60.56 (compared to 62.16 with joint training) and 78.21 (compared to 79.28 with joint training) on NQ@40K. These results show a slight decline compared to joint training in Self-retrieval. This indicates that Self-Retrieval can synergize the retrieval and ranking tasks during training to achieve better performance. Additionally, using shared models can reduce deployment costs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WoHIlsOUVC", "reviewer": "Reviewer_uLuX", "summary": "This paper introduces Self-Retrieval, an end-to-end IR system driven entirely by a single LLM. This model integrates all essential IR functions—indexing, retrieval, and reranking—into the LLM's architecture. By internalizing the retrieval corpus through self-supervised learning, the model transforms the retrieval process into a sequence of passage generation tasks and conducts self-assessment for reranking. The authors provide experimental evidence showing that Self-Retrieval outperforms traditional sparse, dense, and generative retrieval methods on benchmark datasets like NQ and TriviaQA.", "questions": "Please see my concerns in weaknesses.", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of all IR functions into a single LLM is a novel contribution that leverages the inherent capabilities of LLMs across the full spectrum of IR tasks, offering a streamlined and potentially more effective approach.\n2. The concept of Self-Retrieval is introduced clearly, making it accessible to readers. The detailed explanation of how the LLM handles indexing, retrieval, and reranking provides a good understanding of the system's operation.\n3. The paper presents good experimental results that demonstrate significant improvements over existing retrieval methods.", "weaknesses": "1. My major concern is that the experimental settings are inconsistent with existing work, making the results unconvincing. Specifically,\n  - Most existing studies conducted experiments on NQ@320k datasets, but the main experiments of this paper are conducted on NQ@40k datasets. It is important to explain the reason of this setting.\n  - According to the statistics in Table 2, each document in the dataset is split into 26~29 passages, which indicates that the passages are quite short. It is known that retrieving short passages are easier for generative retrieval methods. \n  - It is necessary to explain why different models are selected for NQ@40k, TQA@40k, and NQ@320k experiments.\n  - It is better to include more experimental results on the full KILT datasets as many existing studies for a fair comparison.\n2. In Section 3.4, the three tasks are learned in a \"1+2\" manner. Please add more explanations on this design and provide experimental evaluation on other possible strategies (e.g., training the three tasks jointly)\n3. Ensuring consistent use of key terms throughout the paper would improve its readability and professionalism. For example, all \"large language models\" should be written in \"LLMs\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful comment. We would like to address your questions in turn.\n### About NQ@40k in our experiments\nCurrently, the most common and widely used retrieval method is dense retrieval, which primarily focuses on passage-level retrieval ([2, 3]). Consequently, we constructed NQ@40k to specifically evaluate performance at the passage level. Additionally, we observed that most other generative retrieval methods struggle with passage-level retrieval, whereas self-retrieval successfully overcomes this limitation.\n### The length of passages\nIn common settings for passage-level retrieval, documents are often segmented into chunks of 100 or 200 tokens ([1, 2]). Therefore, we believe that using 200 tokens as the passage length for our main experiment is reasonable. Additionally, we have reported the performance for the more commonly used 100-token segments in the Appendix. It should be noted that shorter passages may not necessarily lead to better performance, according to [1].\n\n[1] Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering.\n\n[2] Dense Passage Retrieval for Open-Domain Question Answering.\n\n[3] C-pack: Packaged resources 535 to advance general chinese embedding.\n\n### The differences in baselins of NQ@40k and NQ@320k\n1. NQ@40k is a passage-level retrieval setting,  following the setting of previous dense retrieval works. We reproduced the latest dense retrieval methods like BGE and GirtLM. Since some previous generative retrieval works have not been open-sourced, we have only reproduced and compared some key generative retrieval baselines.\n2. NQ@320K is a document-level retrieval setting, following the setting of previous generative retrieval works to ensure comparability with previous works.\nFurthermore, thank you for your reminder. In the subsequent versions, we will add the performance of the latest dense retrieval methods, such as BGE and GritLM, on NQ@320k for comparison.\n\n### About  full KILT datasets\nDue to time and computational resource limitations, we were unable to complete training and testing on the entire Wikipedia corpus from KILT during the rebuttal period. Our analysis of scaling corpus size in Figure 3 of the main paper shows that Self-Retrieval maintains a relatively parallel performance relationship with BGE-FT as the document size increases. This trend indicates that Self-Retrieval has substantial scaling potential. Furthermore, due to the scaling characteristics inherent to LLMs, we also believe that their capacity to handle documents has a very high upper limit. However, due to the entire Wikipedia corpus comprsing around 5.9M documents, it is challenging to complete training within the effective rebuttal period. We appreciate your insightful comment and will add more scaling experiments in the future.\n### Experiments on different training strategies\nDifferent training strategies indeed have a certain impact on Self-Retrieval's performance. Our choice of the \"1+2\" training strategy was confirmed through early preliminary experiments.  To compare different strategies, we conducted experiments on NQ40k. We found that jointly training on all three datasets yields overall performance (79.26 Hits@5) comparable to the \"1+2\" strategy(79.28 Hits@5). However, the \"1+2\" training method allows multiple different training sets with the same corpus to share a single indexing process. This significantly reduces the overall training time.\n### About wrting\nThank you for your suggestion. We will refine it in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their insightful comments and valuable suggestions. In our responses, we provided details of the Trie structure to Reviewer xB99, additional experiments on untitled documents and retrieval + reranker baselines to Reviewer g3kT, along with clarifications regarding scaling up to full NQ/TriviaQA and the number of candidates in the reranking stage. We also addressed the latency analysis and the reranker in RAG for Reviewer iXLD and clarified our dataset selection and training strategies for Reviewer uLuX. Here, we would like to provide a unified response to the common concerns raised by the reviewers regarding space and time complexity, as well as experiments on non-Wikipedia datasets as follows:

### Space and Time Complexity

Table 1
|Model Name|Memory|Beam Size|Latency(s)|H@5|
|---|---|---|---|---|
|SEAL|444MB|10|1.18|61.91|
|||100|5.92|59.57|
|DSI-XL|0|10|0.23|60.21|
|||100|0.45|60.21|
|Self-Retrieval|30MB|10|1.44|76.17|
|||100|6.06|81.49|

The table compares the time and space efficiency of Self-Retrieval with other typical generative retrieval methods on NQ40K.
1. Compared to DSI:
   - Although Self-Retrieval has some disadvantages in terms of time and space efficiency, it offers a significant performance advantage. Even with a beam size of 10, its performance far exceeds that of DSI with a beam size of 100. This allows for a trade-off between performance and efficiency when using Self-Retrieval.
   - Additionally, with the advent of software and hardware accelerations for large models, the performance of Self-Retrieval is expected to improve further.

2. Compared to SEAL:
   - Both Self-Retrieval and SEAL use natural language decoding, but SEAL employs an FM-Index, which results in extensive post-processing after generating the natural language, severely impacting efficiency.
   - Self-Retrieval uses a smaller additional storage structure compared to SEAL. The trie, with early stopping, has an average length of 13, resulting in significantly less additional storage than the FM-Index used by SEAL.

### Experiments on non-wikipedia datasets

Table 2
|Method|R@1|R@5|MRR@10|
|---|---|---|---|
|BM25|18.9|42.8|29.2|
|DocT5Query|23.3|49.4|34.8|
|Sentence-T5|27.3|58.9|40.7|
|DPR|29.1|62.8|43.4|
|DSI-Atomic|32.5|63.0|44.3|
|DynamicRetriever|29.0|64.2|42.5|
|Ultron-Atomic|32.8|64.9|46.9|
|GenRet|47.9|-|58.1|
|Self-Retrieval|47.8|69.9|57.2|

This is an excellent suggestion. We conducted experiments on the MS MARCO dataset to verify the effectiveness of Self-Retrieval on non-Wikipedia documents. The MS MARCO dataset consists of crawled web pages that differ from the structured content and titles found in Wikipedia. It contains various complex, repetitive, and lower-quality information and titles. Following Ultron and GenRet, we trained and tested on a subset of MS MARCO and used DocT5Query to generate additional training data. We utilized StableLM-3B as the backbone model and followed the training and inference methods described in Section 3.4.

The experimental results, as shown in the Table, indicate that on the MS MARCO dataset, Self-Retrieval outperforms most generative baselines such as Ultron and DSI, as well as dense baselines like DPR. This demonstrates that Self-Retrieval can still perform well on non-Wikipedia documents.

Meta-review: This paper introduces Self-Retrieval, an approach that uses LLMs for end-to-end information retrieval tasks. The proposed approach integrates indexing, retrieval, and reranking in a single model and utilizes a novel strategy for passage generation and self-assessment.

The reviewers found the paper to be well-written, the proposed approach to be highly novel, and the experimental results to be convincing and strong.

The reviewers did identify a number of weaknesses, as well, including deviations from standard evaluation practices (e.g., passage-based setup vs. document based one, KILT setup, etc.), the desire to see more detailed ablations, and questions around the computational efficiency of the approach.

Overall, this is a very interesting and novel paper that will very likely open up opportunities for future research along similar directions. While the paper isn't perfect, its strengths outweigh its weaknesses. The authors are encouraged to carefully consider the reviewer feedback when preparing an updated version of the paper.
Decision: Accept (poster)