Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
Authors: Minghan Li
Xilun Chen
Ari Holtzman
Beidi Chen
Jimmy Lin
Wen-tau Yih
Xi Victoria Lin
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model
nearest neigbour language model
speculative decoding
attribution
factuality
Primary area: generative_models

Abstract: Large language models (LLMs) often hallucinate and lack the ability to provide attribution for their generations. Semi-parametric LMs, such as kNN-LM, approach these limitations by refining the output of an LM for a given prompt using its nearest neighbor matches in a non-parametric data store. However, these models often exhibit slow inference speeds and produce non-fluent texts. In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), a novel semi-parametric language modeling approach that is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources. NEST performs token-level retrieval at each inference step to compute a semi-parametric mixture distribution and identify promising span continuations in a corpus. It then uses an approximate speculative decoding procedure that accepts a prefix of the retrieved span or generates a new token. NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks, surpassing the conventional kNN-LM method and performing competitively with in-context retrieval augmentation. In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B. Code will be released at https://github.com/facebookresearch/NEST/tree/main.

Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
Minghan Li Xilun Chen?, Ari Holtzman', Beidi Chen Jimmy Lin Wen-tau Yih Xi Victoria Lin
Cohere 2 Meta FAIR 3 University of Chicago
Carnegic Mellon University University of Waterloo
minghan@cohere com, ahol molltmmnnuchicage edu, beidic@andrew. cmu edu jimmylin@uaterrlo. ca, fxilun, scottyih, victorialin)Ometa com
Abstract
Large language models (LLMs) often hallucinate and lack the ability provide attribution for their generations. Semi-parametric LMs, such as kNN-LM, approach these limitations by refining the output of LM for given prompt using its nearest neighbor matches in a non-parametric data store. However, these models often exhibit slow inference speeds and produce non-fluent texts. In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), novel semi-parametric language modeling approach that is capable of incorporating real- world text spans of arbitrary length into the LM generations and providing attribution to their sources. NEST performs token-level retrieval each inference step to compute a semi-parametre mixture distribution and identify promising span continuations in corpus. It then uses an approximate speculative decoding procedure that accepts a prefix of the retrieved span or generates new token. NEST significantly enhances the generation quality and attribution rate of the base LM across variety of showledge-intensive tasks, surpassing the conventional kNN-LM method and performing competitively with in-context retrieval augmentation. In addition, NEST substantially improves the generation speed, achieving 1.8x speedup in inference time when applied to Llama-2-Chat 70B. Code will be released at maain.
https: github. com htmp:ffccoookkreeeerr///////////
Introduction
Large language models (LLMs) have demonstrated strong potential multi-task solvers, excelling in wide range applications (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a; Anil et al., 2024). Despite their advanced capabilities, LLMs frequently encounter the problem of hallucination, particularly when dealing with long-tail knowledge that is less represented in their training data (Kandpal et al., 2023; Asai al., 2023a). To address this limitation, retrieval augmenta- tion incorporates information retrieval and nearest neighbour search from a non-parametric data store enhance evidence-bare based and situated reasoning with LLMs. The resulting cemi-parametric LMs exhibit reduced tendency to generate unsupported content (Khandelwal et al., 2020; Borgeaud et al., 2022; Shi et al., 2024a.b; b; Asai al., 2023a).
However, the effectiveness of retrieval -uggeetee language models (RALMs) in ensuring accurate and reliable content generation varies. The widely used in-context retrieval- Sttrival-augmennttioon (RA) regime (Ram et al., 2023: Shi et al., 2024a,b) softly biases the LM output distribution by prepending retrieved content the input, which does not reliably guarantee faithful attribution of information. Approaches such as &NN-LM (Khandelwal et al., 2020) modify the LM output with a non non- oon-paramttric
* Work done during internship at Meta.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

token distribution derived from nearest-neighte matches in corpus, which provide more direct attribution but has also been shown to degrade the quality of text generation (Wang et al., 2023a). Additionally, retrieval augmentation can significantly increase the generation latency due to the time required for the retrieval processes to complete and the subsequent expansion of the LM's context.
In this work, we propose Nearest Neighbor Speculative Decoding (NEST), This new semi- parametrrieicccc language modeling approach capable of incorporating real-world text spans of arbitrary length into the generations of an off-the-shelf LM, leading to improved quality and latency. NEST extends the standard &NN-LM approach, which interpolates the output distribution of an LM using the distribution of possible next tokens retrieved from corpus (K (Khandelwal et al., 2020). It conducts an additional passage retrieval step at the beginning limit the need to store and search over all tokens in the corpus, offering a balanced trade-off between search accuracy and efficiency. At each inference step, NEST performs content generation with three sub-steps:
1) Confidence-based interpolation. We use a novel Relative Retrieval Confidence (RRC) score to measure the uncertainty of the token retriever and use it as the interpolation coefficient of the output probability mixture. This enables flexible adaptation of the LM's output to different downstream tasks through dynamic interpolation with the token retrieval results.
2) Dynamic span selection. Inspired by the Copy Generator (CoG) (Lan et al., 2023), NEST selects not only the best token predicted by the mixture probability but also extends to the span continuing from that token in the corpus when the token retrieval confidence exceeds a predefined threshold.
3) Relaxed speculative decoding. If a span of more than one token is selected, it undergoes evaluation based on the mixture probability. Through a rejection procedure similar to that in speculative decoding (Leviathan et al., 2023), only prefix deemed highly likely by the mixture probability accepted.
Evaluated on various free-form generation tasks-including question answering, text completion, and factuality-aware generation- using Llama-2-Chas models (Touvron et al., 2023b) of different sizes, NEST demonstrates superior performance compared to both the base LM and the standard kNN-LM under zero-shot setting. For example, combined with NEST, the Llama-2-Che 70B model demonstrates 42.3% improvement of ROUGE-I on WikiText- 103 and 21.6% improvement of FACTSCORE on Biography. Furthermore, NEST performs competitively with respect to in-context verrieval-augmentation on MMLU, Pile-of-Law, and TruthfulQA. We further demonstrate that the two approaches can be combined to enhance generation quality and attribution. Additionally, by generating multiple tokens at each time step, NEST significantly improves the efficiency of long- form generation. For Llama-2-Chat 70B, it achieves a 1.8x speedup in inference time without compromising attribution or fluency.
Background
2.1 Problem Definition
Given an input x, mixture model M predicts the output consisting of segments {81, U2...., ... Br }. In our setting, M may produce multiple tokens time step f, and therefore yt indicates the t-th
segment consisting of at most n tokens where < 1yd < n. Let (w(") _0!²) ...., (..,"("")) (((")) be the tokens in segment yr, we use P.ve PM((w]z, y<t) to denote the distribution of the next token, and use (w = wt x, Merr to denote the probability of w.!! of the next segment yr.
2.2 Nearest Neighbor Language Models (ANN-LM)
The mixture model M involves pre-trained LM and key-value datastore (K, V) that enables approximate nearest neighbors search without further training or fine-tuning.
Key-value datastore. To create the datastore (A., V)) using the LM for a corpus D, let f(-) be the mapping from input sequence to the hidden states d of the LM at some fixed layer. Let w be the next word of e in the corpus D. For sample (ci, wo;) in D after segmentation, we define the i-th key-value pair (ks, v.) in (KC, V) as (h4, wa), where h2 fc:). The whole datastore is thus defined

---

Span Selection 8
LM
LM hiddon
states
#D008
singles
Love
Passage Retrieval
both were released singlee. 'Does He You', that McEnttre...
hidden states
Academy
Country
May
LM hidden
Who sings "Does he Love You" with Reba?
#1338
duet
Reba McEnttie and Linda Davis
Peart (1 20
1) Interpolated Prediction
#(t.
3) Speculative
Decoding with
You
Q: Who sings...
Reba? A: The Does He Love You duet between Reba McEntire and Linda Davis
Generation Latency (Amortized)
Figure I: The NEST approach first locates the tokens in the corpus using the LM hidden states. The retrieval distribution Ps-NN is dynamically interpolated with PLM based on the retriever's uncertainty A₁ The token and its n-gram continuation are then selected from the mixture distribution PM, while the final span length is determined by speculative decoding to remove undesired tokens. The spans incorporated in the final generation provide direct attribution and amortize the generation latency.
as the set of all possible key-value pairs D:
(K,, V) = {(h,, (c))((; ws) E D).
The size of the datastore (Ic, V) is proportional to the total number of tokens in corpus D. This brings difficulty in scaling the size of the corpus and the model, which may require massive storage space and computational resources.
Probability interpolation. During inference, the language model outputs the token distribution (LM(w]e (z, Wecr). together with the hidden state gr- The model uses şt as a query search the datastore (IC, V) and retrieve the r-nearest neighbors π according to the similarity s(g, k) between a query and a key The final conpamamtriic distribution (l-NN(w|zz, V<t) is computed using a softmax function over the similarity of all retrieved neighbors:
Pu-NN(w|z, Vet)
Σ (ks,,)))n
Iw=n, exp(μ s(g.. k4)),
where yı is the inverse temperature. We use 1/ dim((:)) for in practice where dim(q:) ) is the hidden state dimension. This is similar to computing attention in the Transformer model (Vaswani et al., 2017). For similarity sqq, k), we follow Khandelwal et al. (2020) and use the negative squared & distance. Items not in æ are assigned with 0 probability based on the indicator function Iumwy
Finally, the next token is sampled from the mixture distribution P. of the non-parametric distribution PA-NN and the parametric distribution PLM using fixed
À [o, 1]:
PM X (LM(w]2, v<e) (1 A) PL-NN(wl2,, Ves).
Nearest Neighbor Speculative Decoding
3.1 Two-Stage k-NN Search
As mentioned in Section 2.2, maintaining token-level key-value store can be expensive in terms of both latency and storage. To provide better trade-off between latency and accuracy, we adopt the two-stage design, which is widely applied in information retrieval and search engines.

---

First-stage passage retrieval Given the corpus D, we segment the documents into separate passages of less than m tokens each. We then encode the corpus and use a hybrid retrieval system select the relevant passages, as dense retrievers are good at handling semantics in queries (Karpukhin et al., 2020) and sparse retrievers are good at lexical matching (Sciavolino et al., 2021).
Second-stage N-NN token search After obtaining the top-b retrieved passages {d1, d2, def at time step t, we use the encoder f(+) of LM to encode the prefixes of all tokens as keys as shown the fly. Similarly, we use the negative squared &2 distance as the similarity function and qt. as the queries to search for the top-r nearest neighbors in (K'', V*).
in Figure 1. The key-value datastore (K., V) therefore is created
The two-stage design provides trade-off between search latency and accuracy and the passage-level index only takes a fraction of the token-level index in Section 2.2. In addition, the first-stage passage search also acts filter to remove deceptively similar tokens in non-relevant contexts.
3.2 Confidence-Based Output Interpolation
Similar to Equation (3), we linearly interpolate the language model's distribution PLM and non- parametric distribution PE-NN using coefficient Xe for a time step in generation. The difference is that we use the token retrieval score to compute de
(( (
min; (s k:) (5 k4]
a)) /T) /+),
==
where is the sigmoid function and the min-max ratio expresses the uncertainty of the k-NN component. We use the sigmoid activation to re-center and re-scale this uncertainty, where a is the offset and is the scale for the sigmoid function. We refer to this method as Relative Retrieval Confidence (RRC).
If the downstream task does not involve generation, such as perplexity evaluation and multi-choice tasks, our method will end at Equation (4). The mechanisms introduced in the following sections are only applied to generation, including token/span selection and post-hoc revision.
3.3 Dynamic Span Selection
Directly sampling tokens from the mixture distribution PM might escalate the exposure bias since the seemingly coherent tokens might be retrieved from completely different sources. To maintain coherence, extend the context of the current sampled token by using its n-gram continuation in the corpus. Given the current time step t, we first select the next token from the mixture distribution P.M. However, the sampled token w, may correspond multiple retrieved w; (i.e., the value v,), in the neighbors which have different n-gram continuations. We use a simple max max-pooling strategy²
select the starting token (1) of the n--ram from n:
wi
argmax (wa Jurguny (0EE))
PR-NN(W- = wi/w (2, Vex)
(5)
The corresponding n-gram for time step t is (w(") (1) The final output is determined by the interpolation coefficient X, in Equation (4):
w. w (n) where is fixed hyper-parameter.
(2)
Wt:
if Ax 6;
yt
[w]
w
wi
(n))
otherwise.
(6)
where 8 is a threshold and y, is the segment output time step L.
3.4 Relaxed Speculative Decoding
Despite the dynamic selection, the produce spans with adaptive length, we take inspiration from Leviathan et al. (2023), where use M to revise the proposed n-gram. However, the proposal distribution q(w|a; y<t) is unknown
n is hard to control over different tasks. To
We used a slightly different implementattion to ensure the sampled token is in я. Please see the code here: //tttuuu
htt.:///ooo.ttreeeeeeeee.///aaaa wtwpeeeesnnnnnntnnnneeeeeeeeeeee transformers

---

besides the first token w."). Therefore, we use relaxed version of speculative decoding that upper bounds the acceptance probability, The probability of accepting the token w, a span is:
P(accept token ( = min (1,
1, PM(w = =. x. Met, w w mma PM x, Yey wy wi w (1-1))
ux (1-1))
(7)
where 7 E (0, 1] is the relaxation factor, which is referred to as "Ieniency" by Leviathan et al. (2023).
The smaller y is, the less often M rejects the draft. If token wi() is rejected, we will remove all the tokens from W to wh"", and then re-select a token wi from the distribution pM without going through the span selection. The computation for processing multiple tokens can parallelized and NEST can thus maintain the latency or even accelerate the generation. Moreover, suppose all tokens in the draft are not rejected. In that case, we will directly fetch the n-gram's continuation in the corpus and use it for the next draft proposal until rejection, removing the reliance on the hyper-parameter
(8)
Once the n-gram is accepted, the corresponding parts are masked the corpus and will never be used again in this generation. This is to prevent the k-NN component from repetitively retrieving the same segments in a small key-value store (K', V*). We provide the complete procedure in Algorithm 1.
Experiments
We evaluate NEST and other baselines various tasks including text completion, question-ansserrig. fict-verification, and multi-choice tasks, providing a comprehensive picture of factuality, fluency, and attribution of NEST in different domains. In all experiments, we focus on evaluating instruction- following models. We use Llama-2-chat under a zero-shot setting, where we remove the few-shot demonstrations from the instructions to simulate the realistic usage of these models.
4.1 Benchmark Datasets
Text completion. WikiTeex- 103 (Merity et al., 2017) is a standard benchmark for language model- ing, extracted from the set of verified articles on Wikipedia. Pile of Law (Henderson al., 2022) is a growing dataset of legal and administrative data. We use the datasets from Huggingface and further split the test data into validation and test sets. For language modeling, we report the perplexity score. For free-form generation, we report ROUGE-1, 2, L (Lin, 2004) and MAUVE (Pillutla et al., 2021).
Question answering. We select four knowledge-intensitg Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (TQA) (Joshi et al., 2017), HotpotQA (HQA) (Yang et al., 2018), and MedMCQA (MQA) (Pal et al., 2022). Since the in-context demon- strations are removed for free-form generation, et al., 2020) which checks if the output contains any correct answers instead of exact match.
datasets, including
answer-level recall (i.e., Hit@) 1) (Karpukhin
Fact verification. We evaluate biography- generaaion task (Min et al., 2023) and Truth- fulQA (Lin et al., 2022) which is benchmark for testing false beliefs or misconceptions. We use FACTSCORE (Min al., 2023) for biography. For TruthfulQA, we follow Lin et al. (2022) which uses the difference between the max similarity to true reference answer and the max similarity to false reference answer for BLEU and ROUGE-1.
Closed-set tasks. MMLU (Massive Multitask Language Understanding) (Hendrycks et al., 2021) benchmark covers 57 subjects across STEM, the humanitics, the social sciences, and more. We report the macro accuracy for each domain.
4.2 Implementation
Knowledge Sources. Wikipedia (CC BY-SA 3.0): For tasks except text completion on Pile of Law, we use the Wikipedia 2021 dump released by Izacard et al. (2024) as the knowledge source and follow the same pre-processing procedures in RA-DIT (Lin et al., 2024), yielding ~33M passages with each
5

---

less than 200 tokens. Pile of Law (CC BY-NC-SA 4.0): We use the training split from Huggingface and select only the English data. We then follow the same procedure applied in Wikipedia, yielding a corpus containing ~15M passages after filtering. More details are provided in Appendix A.
Inference setting. &NN-LM and NEST share the same first-stage retriever. We use DRAGON+ (Lin et al., 2023) and BM25 (Robertson and Zaragoza, 2009) to encode the segments into dense and sparse vectors, respectively. Given the input, we query both the dense and sparse indexes at the same time and retrieve their corresponding top-(b 1) passages. We linearly interpolate the similarity scores between the two search results (also known as fusion) and sort them before selecting the top-b passages. The number of passage candidates b set to be 40 and the scaling factor is set be 100. For RA, we use the top-3 passages in the prompt due to the context window limit. We further combine NEST and RA since they are independent methods. Greedy decoding is used during generation. More details about retrieval, decoding, and hyper- parameters are described in Appendix B.
Evaluation setting. For text completion tasks and perplexity evaluation, we use 128 tokens as the prefix and the consecutive 256 tokens as the target. For the other tasks, we use 128 tokens as the max generation length for question answering and 512 for fact verification. For retrieval-based models, only the prefix will be used for retrieval. Hyper-paaammeter of all baselines and NEST are tuned on the dev set of WikiText- 103, NQ, and Biography. Each baseline uses the same preer-parameters for all tasks evaluated. We first tune the related apper-parameters for perplexity and then tune the rest for generation metrics to reduce the search space. More details are provided in Appendix B.
4.3 Baselines
Base LMs. evaluate publicly available, instruction -tuned language models, Llama-2-cha series", with model sizes ranging from 7B, 13B to 70B.
Two-Stage &NN-LM. We apply the two-stage strategy described in Section 3.1 to KNN-LM as well, where we retrieve the top-b passages and encode key-value datastore (K", V*) the fly.
In-Context Retrieval Augmentation (RA). A common -aggemttttaugmtitttioonn method is adding the retrieved evidence into the prompt. We perform retrieval given the only input instead of retrieving new passages every h step due to the expense of refreshing the kv-cache.
4.4 Main Results
Table 1 shows the main results of NEST and other baselines. For language modeling, RA-NEST is both Wik and Pile of Law. For text completion,
able to achieve the lowest complexity RA has the best MAUVE scores and ROUGE scores in Wikitext- 103 while RA-NEST works better for 7B and 13B models on Pile of Law. We observe that for legal documents, quoting the exact clauses from the source might be more favourable compared to Wikipedia.
For question.answering, RA-NEST tends to work better for smaller models (7B and 13B) in general. The gap between base LMs and other methods diminishes for 70B LMs, which is consistent with previous work where retrieval is found most useful for smaller models (Borgeaud et al., 2022).
For fact-verification, NEST is able to outperform the base LMs but underperform RA in terms of the FACTSCORE. RA-NEST'S is able to outperform RA for the 70B model. The degradation for RA-70B caused by generating shorter claims which punished by the FACTSCORE. On TruthfulQA, the semi- aani-paramttric LMs consistently outperform base LMs and RAs where in-context retrieval seems to have a negative effect on the scores. This is because TruthfulQA is an adversarial dataset containing difficult questions where in-context RA is more susceptible to the "evidence" in the prompt (e.g.. astrology and myths). In contrast, NEST only interpolates the results at the output level and therefore performs better in this case. The combination RA-NEST is also affected by the in-context retrieval.
For closed-set tasks, NEST is comparable to RA and RA-NEST manages to achieve the best macro scores on average. Overall, NEST is able to outperform base LMs and ANN-LM's on most tasks while being on par with RA. The combination of RA and NEST further improves over the two methods
"nttps://huggingfaac" co/neta- 1llamaLlamaa 70b- chat-hf

---

Models
Wikitext-
Pile Law
PPL())
MAUVE
RG-1
RG-2
RG-L
Avg. Len
PPIAI)
MAUVE
RG-1
RG-2
RG-1.
Avg. Len
Llama-2-Chatey
14.6
58.8
15.8
3.7
14.4
175.4
10.1
80.7
19.1
5.5
17.1
211.4
+RA
7.2
74.6
35.7
23.1
34.4
204.5
7.1
84.7
23.1
8.9
21.1
222.0
+ANN-LM
9.8
82.5
23.7
7.7
21.7
238.2
8.8
81.1
19.4
5.7
17.4
214.3
+NEST
844
73.2
28.4
14.2
27.1
218.4
8.1
88.0
23.7
8.7
21.5
226.5
+RA-NEST
6.4
72.6
35.2
22.7
34.0
202.0
6.7
90.0
24.4
9.0
22.2
232.1
Lama 2. 3.Chaliss
12.0
75.9
19.9
4.9
18.0
218.4
8.2
72.8
17.5
5.3
15.7
181.7
+RA
6.5
91.5
38.9
24.2
37.2
249.3
5.9
86.6
23.6
9.1
21.5
228.7
+ANN-LM
8.6
76.3
23.7
8.2
21.9
238.5
7.4
71.5
17.7
5.3
15.99
183.7
+NEST
7.2
67.1
29.3
15.6
28.1
207.1
6.8
86.0
22.9
8.7
20.9
212.3
+RA-NEST
588
86.8
38.6
24.0
37.0
245.5
5,7
90.1
24.7
9.2
22.4
229.4
Llama-2-Chatos
9.9
88.6
22.9
6.2
20.8
239.6
6.9
93.4
23.0
7.1
20.7
250.1
+RA
5.3
91.6
40.5
26.1
38.8
235.9
4.9
95.5
26.3
10.1
24.0
253.2
+ANN-LM
7.1
83.6
26.1
9.6
24.1
253,9
6.3
94.4
23.1
7.2
20.8
251.3
+NEST
6.3
82.6
32.6
17.2
31.1
236.3
5.9
95.4
25.6
9.4
23.2
251.3
+RA NEST
4.8
90.0
40.2
25.9
38.6
233.1
4.7
97.6
26.2
9.5
23.7
253.6
Models
TQA
NQ
HQA
MQA
Avg-
TruthfulQA
Biogruphy
MMLU Social
Answer Level Recall
ABLEU
ARG-1 1
FS
Facts
Human.
STEM
Other
Avg.
Lluma-2-Chat-p
61.1
38.9
9.3
35.0
-0.02
0.42
27.2
39.6
37.2
+RA
69.5
48.4
44.1
12.8
43.7
-0.34
0.18
56.5
67.1
41.8
35.3
42.2
43.3
40.7
+ANN-LM
63.4
42.4
33.5
9.5
37.2
0.13
0.66
30.6
59.8
38.0
33.1
39.2
40.1
37.6
+NEST
61.5
10.2
37.1
0.45
38.9
42.0
42.0
+RA-NEST
69.0
48.8
45.3
13.3
44,1
-0,32
0.21
55.1
57.7
37.9
32.7
39.3
39.8
37.4
Llama-2 Manaa--CCeee 2-Cholys
63.5
42.3
32.6
10.2
37.2
0.13
0.81
28.8
49.9
41.5
35.0
40.2
43.8
40.1
70.9
14.0
45.3
0.25
46.4
42.7
+ANN-LM
64.7
43.5
34.2
11.2
38.4
0.20
0.95
31.1
46.1
41.4
34.7
40.6
44.2
40.2
+NEST
64.2
44.2
34.3
10.9
38.4
0.29
0.98
35.7
47.2
41.3
34.9
40.2
43.7
40.0
+RA-NEST
70.9
45.3
14.7
45.7
0.25
58.4
52.4
43.5
43.5
46.7
42.8
Llama- 2- 2-Chatzon
74.0
50.1
12.8
44.1
0.14
0.70
34.2
58.8
43.5
44.4
47.0
43.2
+RA
75.5
55.4
52.5
16.0
49.9
-0.13
0.40
52,9
42.1
45,9
39.7
46.2
48.6
45.1
+ANN-LM
74.6
51.2
40.2
13.5
44.9
0.08
0.58
36.1
54.4
44.0
37.4
44.1
47.1
43.2
+NEST
74.2
51.6
41.4
13.8
45.2
0.17
0.70
41.6
56.2
43.8
38.0
44.4
47.8
43.5
+RA-NEST
75.4
55.2
52.4
16.3
49.8
-0.19
0.31
59.2
53.8
45.8
39.7
46.2
48,9
45.1
Table 1: Results on text completion (upper table) and other tasks (lower table). Bold numbers indicate the best performance. PPL.: Perplexity. RG: ROUGE score. Avg. Len: Average generation length. ABLEU/ARG: The difference between the max score to correct references and the max score to incorrect references. FS: FACTSCORE with length penalty.
on some tasks. Despite the limited improvement, we will show that NEST is able to provide better attribution and latency in the following sections.
4.5 Latency Analysis
Latency breakdown. The combination of dynamic span selection and relaxed speculative decoding can improve the latency of the LLM generation by quick draft proposal and processing multiple tokens at time step. Figure 2a shows the latency breakdown of NEST-70B model (a: 0.3, T 0.1,8 8 = 0.5) for different relaxation factors on the Biography validation data. The latency experiment done on 8 100 GPUs (for model parallelization) and 32 CPU threads (for search). The batch size is set to 1. We use internal, research- purpose implementation of the base Alama-2-chat model which did not optimize for latency. As we can see, the LM encoding time takes about half of the latency, while the sum of the others takes the rest. Noticeably, the cost of passage search and token index building stay relatively constant per query, while the others are related the number of tokens processed per time step. Still, even with extra retrieval overheads, the slowest NEST model is faster than the base LM, showing the efficacy of span selection and speculative decoding.
Latency-aceuracy trade-off. To understand why NEST can accelerate generation, we first show the latency-accuracy trade-off by tuning the relaxation factor in Figure 2b. The smaller is, the less often NEST rejects a segment retrieved from the corpus, which enables more tokens to be processed in parallel. The average proposed span length in Figure 2a can increase from 5 tokens to 35 tokens each time step as the relaxation factor gets smaller. Combined with Figure 2a, we can reach the conclusion that fetching longer spans from the corpus results in lower generation latency per query. For the accuracy, the FACTSCORF on Biography validation data shows that there is a sweet spot around y 5e 2 where both low latency and high accuracy can be achieved at the same time.
7

---

70
latency
60
Passage search
0.50
39
Token index buildling
Latancy (slevery)
Token search
encoding
13
Avero Length
Rejection sampling
FactSrore
40
0.46
30
0.44
FactScore
Span length
5e-1
5c-4 1e:4
5e1 16.1
1e:2 5e Se 1c44 Se.5
Relasation
(a) Latency Breakdown.
(b) Span length and FACTSCORE on Biography.
Figure 2: Latency-aceuracy trade-off and breakdown on Biography using Llama-2-68 2-Chat 7EB+NEST. As the relaxation factor 7 decreases, NEST tends to accept longer spans from the corpus. We choose y = 5c- our main experiments, which accelerates the generation and improves the FACTSCORE.
Task
Models
Ratio (%)
Length
Example (Generated by NEST-70B)
NEST NEST 138
93.4
18.4
Question: who sings does he love me with reba?
52.4
4.4
Answer: The song "Does He Love You" [1] is duet
NEST 700
58.8
4.3
Linda Davis, It was Reba's compilation album Greatest Hits Volume 2. It of country music's several songs about love triangle. [2]
between Reba McEntire
RA-NEST 7B
77.5
released in August 1993 the first single
NQ
RA-NEST 198
33.2
3.0
RA-NEST 700
44.9
3.9
Question:
Tell me bio of Taral Hicks.
NEST 78
95.5
27.9
Answer: Taral Hicks (bom September 21, 1974) is American actress and singer. Hicks is best known for her acting in such films as 1993's American crime drama film A Bronx Tale and her singing in such work as her 1997 [3] debut studio album This Time which peaked at No. 4 on
NEST
53,9
10.6
NEST 708
58.6
7.0
Bio
RA-NEST
50.3
5.1
RA-NEST 130
48.5
5.9
Billhoard Bubbling Under Hot Singles. Raised in Teaneek,
RA-NEST 708
80.7
11.0
New Jersey, Hicks graduated from Teaneck High School 1994 [3]. and singer D' Hicks.
She is the younger sister of
Table 2: Attribution analysis. (Attribution) Ratio: Proportion
tokens that are taken from the corpus. (Attribution) Length: Average length of consecutive spans in the generation that are taken from the same document. Green: Segments taken from the corpus. Gray: Reference.
4.6 Attribution and Qualitative Analysis
One of the most important features of NEST is providing attribution directly at a span level, where the reference for the corresponding statement is accurate since it is directly taken from the corpus. Table 2 shows the attribution ratio, average attributed span length, and two examples for analysis. For NQ and Biography tasks, depending on the model and syper-parameters in Equation (4) and (7), the ratio of tokens that can be traced back to the corpus ranges from 33.2% to 95.5%. In addition, more desirable to have consecutive segments that come from the same source so that consistent attribution can be provided. and the average length of spans taken from the corpus ranges from 3.0 to 27.9 tokens. This feature provides span-level attribution for most claims in the LLM generation. To our knowledge, neither of the baselines can achieve the same granularity and preciseness for the attribution as NEST. We provide more analyses on sensitivity and ablation for NEST Appendix
Related Work
5.1
Retrieval-Angmennation
Retrieval Augmentation involves external knowledge sources to improve the effectiveness of language models on Knowledge-intensive tasks. Chen et al. (2017) propose DrQA which combines extractive models and independent retrievers for open-domain question-answering Follow-up works on retrieval- augmentation such REALM (Guu et al., 2020), RAG (Lewis et al., 2020), and Atlas (Izacard

---

et al., 2024) further combine the retrieval component in pre-training and fine-tuning for downstream knowledge-ine intensive tasks. Asai et (2024) further divide them into three categories:
eoonledge-intensive
Input augmentation. REPLUG (Shi et al., 2024a) and in-context RALM (Ram et al., 2023) propose to pre-pend the retrieved passages in the prompts for zero-shot factual support. Recently, Self-
RAG (Asai et al., 2023b) leverages special tokens to perform adaptive retrieval and different critics to iterative refine the RALM's output. RA-DIT (Lin et al., 2024) retrofits LLMs with retrieval capabilities via instruction fine-tuning.
Intermediate fusion. RETRO (Borgeaud al., 2022) employs novel attention mechanism to incorporate multiple pre- pre-processed text fragments in intermediate layers for more efficient integration of retrieved results. This approach has been successfully applied to larger decoder-only language models as demonstrated by RETRO++ (Wang et al., 2023b) and InstructRetro (Wang et al., 2024). FiD (Izacard and Grave, 2021) applies similar an encoder-decoder structure in zero-shot manner and achieves better effectiveness at a document level.
Output integration. 6NN-LM (Khandelwal et al., 2020) pioneers this direction and proposes to interpolate the retrieval distribution and LM's prediction. Follow-up works further propose adaptive interpolation methods which involve training (He et al., 2021; Bhardwaj et al., 2023) and excessive tuning (Drozdov et al., 2022). Another line of work proposes to joint train the phrase encoder and LM to expand the vocabulary dynamically using the retrieved phrases, such as Copy- Sopy-Generator (Lan et al., 2023) and its follow-up work (Cao et al., 2024). Martins et al. (2022) proposes a chunk-based &NN machine translation model which retrieves chunks of tokens from the datastore.
5.2 Inference-Time Revision
Speculative decoding (Leviathan et al., 2023; Chen et al., 2023; Miao et al., 2023; Spector and Re, 2023) is an acceleration method that leverages small model to generate drafts for a large model to evaluate. The lateney is improved as the larger model can process multiple tokens in parallel at each time step. Recently, REST (He et al., 2024) proposes to draw multiple drafts from datastore and leverages a prefix trie tree to compute the proposal distribution, which is the closest concurrent work. Yang et al. (2023) also utilizes prefix matching to select draft sentences from a datastore, and keep the continuation of the draft sentence as long as the token matches with the model generation.
In general. speculative decoding can be categorized as unbiased self-revision method. In com- parison, NEST changes the LM output distribution through interpolation with non- -on-paraettric probability distribution. Previous work focusing on fact-checking follows a similar idea to generate factually consistent texts with a set of evidence via post-hoc editing, such as FRUIT (Iv et al., 2022) and PEER (Schick et al., 2022). Recently, RARR (Gao et al., 2023) leverages more complex planning with LLMs to verify the retrieved evidence and generate attribution reports.
Limitations
While being able to directly retrieve segments from the corpus and apply them in the generation, the output of NEST might still contain factual errors depending on the accuracy of the first-stage passage retrieval and the second-stage token retrieval. Moreover, as a plug-and-play method, our main goal is to provide a flexible solution that can combine different L.MMs and data stores in zero- and few-shot manners. Without further fine-tuning, the integrated system might be sub-optimal and the results can be better if it is fine-tuned on appropriate tasks. Lastly, such semi-parametrie LMs may not improve the ability of in-context learning, since the demonstrations in the prompts are unlikely to appear in any contexts that can be found in the database. An observation from preliminary experiments is that the current neural retrievers do not have the capability to process the in-context few-shot information, where techniques such as query reformulation might be needed for parsing the demonstrations.
7 Conclusion
This paper presents NEST, an inference-time revision method for LMs that improve their factuality and attribution through nearest neighbor speculative decoding. Leveraging two-stage k-NN search, relative retrieval confidence, dynamic span selection, and relaxed speculative decoding, NEST improves both validation perplexity and free-form generation quality nine different tasks. Its

---

effectiveness can be further improved when combined with in-context retrieval augmentation. With these results, we demonstrate that NEST is capable of generating text grounded to real-world sources in low latency while maintaining fluency.
8 Broader Impact
The ability to copy real-world texts from existing data stores is useful for finding the source of the claim (credibility), preventing hallucination (factuality), as well as protecting copyright (risk management). It helps to resolve the dispute that often happens in AI tools by acknowledging the contents that are borrowed from existing human works (e.g., arts, books, and other creative content). Meanwhile, the information on the Internet is mixed and it is important to filter out false and sensitive information before directly injecting them into the generation.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia. Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayant Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0231/ 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023a.
Rohan Anil, Sebastian Borgeaud, Jean-Baptin Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari- dou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham. Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawezyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni. Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Sev- eryn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Itycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah. Emanuel Taropa, Majd Al Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olean Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Syg- nowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James
10

---

Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong. Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iñaki Iturrate Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankara- narayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomenech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin. Shariq lqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming- Weig Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang. Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys. Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson. Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery. Dipanjan Das, Dominika Rogozińskka, Vitaliy Nikolaev, Pablo Sprechmann. Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo. Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung. James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang. Fan Yang. Shuo yiin Chang, Paul Komarek, Ross Mcllroy, Mario Lučié, Guodong Zhang. Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung. Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee. Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma. Polina Zablotskaia. James Besley, Da- Wo-ooon&& Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty. Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos. Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang. Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo. Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Victor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGer- ald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg.
11

---

James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Has- son, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Pagaanii, Jean-Raptiste I Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao. Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang. Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kępa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang. Richie Feng, Milad Gholami, Kevin Ling. Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wan- dekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David. Ankur Garg. Yifan He, Oleksii Duzhyi, Anton Algmyr, Timothée Lottaz, Qi Li. Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine. Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He. Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François- Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze. Francesco Bertolini, Liana-Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejezuk, Dominik Rabiej. Vipul Ranjan, Krzysztof Styre, Pengcheng Yin. Jon Simon. Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky. Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Hom, Assaf Israel, Francesco Pongetti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh. Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchan- dran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghorbani. Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo. Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV. Sarmishta Velury. Himadri Choudhury, Jamie Hall, Premal Shah. Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu. Soo Kwak, Victor Ahdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojeiech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders. Michael Mandl, Ante Kärrman, Pawet Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed. Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-
12

---

David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanezyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Chenn Sring-Cheng Chiu, Jaume Sanchez. Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving. Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Sugamthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikotaj Rybinski, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Mareus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng. Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang. Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal. Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Lebiond, Shirley Chung. Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang. Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp. Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig. Jennifer Beattie, Emily Caveness, Libin Bai. Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong. Phuong Dao. Zeyu Zheng. Frederick Liu, Fan Yang. Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp. Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thortton, Jordi punt-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu. Chenxi Pang. Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamaria-FFernnddez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee. Zonglin Li. Kyle Levin, Raphael Hoffmaan, Dan Molltmann-Ricc Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh. Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini. Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang. Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo. Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srini- vasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre- Chimoto,, Hanna Klimezak- Pliccikkkk David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendoliechio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru. John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian I Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham
13

---

Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang. Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M.. Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong Amnuta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Thitelhaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang. Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listik, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Himschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlag. Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, lanna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: family of highly capable multimodal models. arXiv preprint ar arXiv:2312. 1880, 2024.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JML,. 2023.
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41-46, Toronto, Canada, July 2023a. Association for Computational Linguisties.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through Memorization: Nearest Neighbor Language Models. In International Conference on Learning Representations (ICLR), 2020.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katic Mil- lican, George van den Driessche, rean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick. Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Im- proving language models by retrieving from trillions of tokens. arXiv preprint 112.422 2022.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. REPLUG: Retrieval- -uggmeett black-box language models. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2024a.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lánguistics. Association for Computational Linguistics, 2024b.
14

---

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton Brooon and Yoav Shoham. In-context -augeeeal- language models. Transactions of the Association for Computational Linguisties, 1 11:1316-1331, 2023.
Shufan Wang, Yixiao Song, Andrew Drozdov, Aparn Garimella, Varun Manjunatha, and Mohit Iyyer. ANN-LM does not improve open-ended text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15023-15037, Singapore, December 2023a. Association for Computational Linguistics.
Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. Copy is all you need. In The Eleventh International Conference on Learning Representations, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274- 19286. PMLR, 23-29 Jul 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fermandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu. Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.000 0928888,, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, November 2020. Association for Computational Linguistics.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric questions challenge dense retrievers. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 6138-6148, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Peter Henderson, Mark Simon Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and 256GB open-source legal dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Chin-Yer Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
15

---

Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lce, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-We Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yes Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse. explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujüi, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369 2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann, editors, Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248 260. PMLR, 07-08 Apr 2022.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-graine's atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076-12100, Singapore, December 2023. Association for Computational Linguistics.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint NrXiv:2009.03300, 2021.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24(1), mar 2024. ISSN 1532-4435.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. RA-DIT: Retreeeal-augmeente dual instruction tuning. In The Twelfth International Conference on Learning Representations, 2024.
Sheng- Chigh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguisties: EMNLP 2023, pages Association for Computational Linguistics.
Singapore, December 2023.
Stephen F. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-389) 2009.
16

---

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguisties.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: retrieval- augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML' 20. oog.. 2020.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. generation for mnowledge-intensii nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen tau Yih. Reliable, adaptable, and attributable language models with retrieval. arXiv preprint arXiv:2403.03188 2024.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310. 15/1, 2023b.
Boxin Wang, Wei Ping, Peng Xu, Lawrence MeAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, and Bryan Catanzaro. Shall we pretrain autoregressive language models with retrieval? comprehensive study. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7763-7786, Singapore, December 2023b. Association for Computational Linguistics.
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro. Instructretro: Instruction tuning post retrieval-augmeened pretraining. arXiv preprint
arXiv:2310.07713, 2024.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874 880, Online, April 2021. Association for Computational Linguisties.
Junxian He, Graham Neubig, and Taylor Berg- King-Kikkaatrik. Efficient nearest neighbor language models. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 5703-5714, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Rishabh Bhardwaj, George Polovets, and Monica Sunkara. Adaptation approaches for nearest neighbor language models. arXiv preprint arXiv:2211.07822, 2023.
Andrew Drozdov, Shufan Wang, Razich Rahimi, Andrew McCallum, Hamed Zamani, and Mohit lyyer. You can't 't pick your neighbors, or can you? when and how to rely on retrieval in the KNN-LM. In Findings of the Association for Computational Linguisties: EMNLP 2022. pages 2997-3007, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
Bowen Cao, Deng Cai, Leyang Cui, Xuxin Cheng, Wei Bi, Yuexian Zou, and Shuming Shi. Retrieval is accurate generation. arXiv preprini arXiv:2402. 17532, 2024.
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins, Chunk-based nearest neighbor machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 0228-4245 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
17

---

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative Ilm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023.
Benjamin Spector and Chris Re. Accelerating Ilm inference with staged speculative decoding. arXiv preprint arXiv:2308.0465 2023.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D. Lee, and Di He. Rest: Retrieval-based speculative decoding. arXiv preprini arXiv:233 1.08252, 2024.
Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models, 2023. URL attpa//b/////0//// 0.304.04487.
Robert Iv, Alexandre Passos, Sameer Singh, and Ming- Wei Chang. FRUIT: Faithfully reflecting updated information in text. In Marine Carpuat, Marie-Catherinn de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3670-3686, Seattle, United States, July 2022. Association for Computational Linguisties.
Timo Schick, Jane Dwivedi- Yu. Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard,
Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. Peer: A collaborative language model. arXiv preprint arXiv:2208. 11663, 2022.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RaIr: Researching and revising what language models say, using language models. arXiv preprint arXiv:2210.08726. 2023.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint 17Xiv:2401.08281, 2024.
Jimmy Lin, Xueguang Ma, Sheng-Chiel Lin, theng-Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2356-2362, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 9781450380379.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. Generating literal and implied subquestions to fact-check complex claims. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3495-3516, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4170- Toronto, Canada, July 2023. Association for Computational Linguistics.
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Ex- pertQA: Expert-curated questions and attributed answers. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies ( (Volume 1: Long Pupers), pages 3025-3045, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
18

---

A Additional Implementation Details
Two-stage k-NN search For the first-stage passage search, we use a Faiss (Douze et al., 2024) dense index and Pyserini (Lin et al., 2021) BM25 index for efficient search. For the dense index, we first use DRAGON+ to encode each passage in the corpus into single vector, and then use Faiss (index string "VF65536.PQ2566) to cluster the vectors into 65536 centroids and quantize them into 256 codes of 8 bits each. For the sparse index, we use the default spper-parameters and the "optimize" option in Pyserini to reduce the index size. For the approximate nearest neighbor retrieval, we use nprobe= 4096. During passage search, we retrieve 4000 passages from each index and keep the similarity score for fusion. The fusion coefficient 7) is determined by the relative confidence of dense and sparse retrievers similar to Equation (4). We set the dense coefficient 'ftense = - top-100 ddense (g, / max ddense (g. d) and same for the sparse coefficient 7pparse. The final interpolation coef is 0.7 (1 3/sparse) + 0.3 * Tjdense- The fusion score for each document s(q, d) = * Sdense + (1 1-7) 77) * Ssparse- If a document is missing in either dense or sparse retrieval results, we set its score the minimum similarity of the dense/sparse retrieval results. The first-stage search is done on RAM and CPUs with 32 threads. The final Wikipedia dense index size is about 8.96GB, and the sparse index size is about 3.48GB disk.
For the second-stage token search, we use the LLM to encode the sequence and use the input to the final layer's 's feed-forward network after layer normalization as the key and query vectors following Khandelwal et al. (2020). We retrieve the top-1024 tokens using the squared &g distance and compute the non-parametric probability according to Equation (2).
Rest of NEST For relative retrieval confidence, we set a = 0.3, 0.1 for all Wikipedia-based tasks and a = 0.2, r = 0.1 for Pile of Law for all model sizes in Equation (4). For dynamic span selection, we the n-gram length to be 64 and 8 0.5 for all model sizes and all tasks in Equation (6). For relaxed speculative decoding, we set r = 5e - 4 for Pile of Law tasks for all model sizes in Equation (7). For Wikipedia-bule -assed tasks, we set 7 5e
for the 7B model, y = 5e for the 13B model, and = 5e 2 for the 70B model. For RA-NEST, all models use the same y y= 5e- for all tasks except Pile of Law which still uses 7 = 5e 4 We observe that as the model gets stronger, using larger y which leads to more rejection, is more beneficial to generation quality. The complete NEST procedure is provided in Algorithm 1.
B Evaluation Details and Hyper-parameter Tuning
Datasets. We sample subsets of WikiText- 103. NQ and Biography as dev sets for hyper- hyper-parameter
tuning. We use the validation sets of WikiText- 103 (CC BY-SA 3.0), NQ (Apache License 2.0). TriviaQA (Apache License 2.0). MedMCQA (MIT License), HotpotQA (Apache License 2.0). MMLU (MIT License), and Biography (MIT License) for validation. TruthfulQA (Apache License 2.0) only has the test set. We finally test all datasets shown in Table For HotpotQA and MedMCQA, we do not have access to the test set and therefore the validation results are reported. For Biography, we use the labeled data that have human annotation as the validation and dev set, and the unlabeled data as the test set. In the original SCCRE paper, the authors use InstructGPT (Ouyang et al., 2022) to perform fact decomposition before verification. We hereby train our own decomposition model by further fine-tuning Llama-2 7B using publicly available datasets (Chen et al., 2022; Liu et al., 2023; Malaviya et al., 2024). For fact verification, we use the option to evaluate the decomposed atomic facts.
Inference and prompts. For language modelling and text completion, we use a context length of 128 tokens and a max generation length of 256 tokens. For the other tasks, we use 128 tokens as max generation length for question answering and 512 for fact verification. We remove all the in-context demonstrations from the prompt to test the zero-shot effectiveness of our model. We use greedy decoding in our experiments as the randomness in sampling can undermine factuality.
Regarding the prompts we use for evaluation, for MMLU, we compare the perplexity of each option concatenated with the question and select the one with the minimum perplexity.
For text completion, we use the following prompt " [[NST] Write an article. n Article: 
 (prefix)" where the 
 is format tag for Llama-2-Chi
19

---

Algorithm 1 NEST w/ Greedy Decoding
Inputs: Language model LM, hidden state encoder f, first-stage retriever Rt corpus C, input a. First-stage retrieval: Retrieve documents d1. d2 .2....dd dg from corpus d1, .i,.d...,d0 dg t R(x,C, C)
Second-stage retrieval: Construct token-level key-value memory
(A'', +)+++)+
for 1 do
d. w W3 ...,, with 4- di d: hg. ..... h f(d,)
for i 1 m d: d1
do
(KC', V") aadd
end for end for Generation
Yet f 2
fort = to T do
Compute query embedding
dr < [[v=))[11)
Token embeddings search, return top-r scores and values (IC', - r)
(1), (2), 02),,... (Sr, ())) f
Compute non-parametric distribution
PaNNN w |-tr +- 0, Ve E vocabulary
for to do
Pk- = valver) t PN-NN(w viluer) + exp(μ st)/ Eiz, exp(e. 85)
end for
Confidence-based Interpolation
A sigmoid( min; a)/T)
max; #:
P. (x(e)= + b PLM (w(yeee + (1 Ae) (w)New)
Dynamic span selection
Wi <- argmax (w
0, + argmax PL-NN = v.lyer)
Utatn Wes if
C.get-ngramm (Ve, n)
8; V1 Utetns otherwise.
n. 4- 1el
Relaxed Speculative Decoding
for = 1 n do
Paccept (w)
Pove (wo (1)
mn (2)
un 1)
7-maxx (w
w... (1-1)
Break if Paccopr (w: (8) < 0.5
end for
if' n and n I then
wi + argmax PMnWWlees wi (1) m ..... wi (1-1)
(2)
"
end if
Yzt t concatenate(yze..t wi w/?) .... (1))
end for
Return Vet
20

---

(a) Passages and tokens.
(b) Interpolation coefficient.
(c) Threshold for span selection.
Figure 3: Sensitivity analysis WikiText-103 and NQ dev for the NEST-7B model with the above typer-parameters in the sub-figures.
For question-answeee and vertfveriiiiation tasks, we use the following template: " "
 Question: {question] | Answer: 
" where we format the input question in the bracket.
For the RA models, reference. Background: (retrieved passages) ) n Article: 
 (prefi)"" for text completion. For -urreval-augmenteed question-answering answering and fact-verificationn tasks, use "
 Answer the question with the background context as reference. Background: [retrieved passages} n Question: (question) J Answer: 
".
the prompt " [
 Write article with the background context as
Hyper-parameters and baselines. For the base LM, we do not tune the hyper- parameters released with the original Llama-2-chat models. For the in-context retrieval augmented baseline, select the top-3 retrieved passages. For kNN-LM, we follow Equation (3) and use an interpolation coefficient of 0.7 for Wikipedia-based tasks and 0.9 for Pile of Law. For NEST and RA-NEST, we first tune the hyper-parameters in Equation (4) on language modelling tasks using perplexity. We then fix those hyper-parameters and then tune the rest of the parameters in Equation (6) and (7) on generation tasks. All hyper- pepeeeprrameters in the above methods are tuned on the dev sets of WikiText-1001 NQ. and Biography.
C Analysis
The following analyses are performed on the validation set of WikiTexx--033, NQ, and Biography data with the Llama-7B-chat model.
C.1 Sensitivity
Number of retrieved passages and tokens Khandelwal et al. (2020) show that increasing the size of the database and the number of tokens can improve the perplexity with proper setting. We also verify whether our two-stage k-NN search and RRC approach follow the same trend. Figure 3a shows the validation perplexity on WikiText-103. For fixed number of passages, the perplexity decreases as the number of tokens increases; for a fixed number of tokens, the perplexity decreases about 0.5
the number of passages doubles. However, as NEST needs to encode the retrieved passages on the fly, the latency also increases linearly w.r.t. the number of passages. Therefore, we set the passage number be 40 and the token number to be 1024 in the main experiments.
Interpolation coefficient Figure 3b shows the sensitivity of the hyper syper-parameters a (offset) and (temperature) in Equation (4) on 103. When 7 is big, d is close to uniform distribution and therefore the offset a does not have a big impact on the perplexity. When small, the impact of is enlarged and the sweet spot is achieved around 0.1 and 0.4.
Threshold for dynamic span selection Figure shows how threshold 5 Equation (6) affects the generation on NQ. A bigger 5 means selecting the span instead of token more often. We can see that the answer-level recall on NQ first increases and then decreases as we increase the value of where the sweet spot is around 0.5.
21

---

Models (7B)
Wiki/ROUGE-1
NQ/ALR
Bio./FS
ANN-LM (two-stage)
20.1
40.8
34.8
Relative Retrieval Confidence
24.7
44.4
41.6
Dynamic Span selection
24.5
44.6
41.6
Relaxed speculative decoding
26.8
45.4
46.8
Table 3: Ablation study on the validation of WikiText-103. NQ, and Biography. ROUGE-1 is
reported for WikiText- 103, ALR is reported for NQ, and FACTSCORE is reported for Biography.
C.2 Ablation Study
Table 3 shows progressive ablation of NEST WikiText-103, NQ, Biography. As mentioned in Section 3.1, is extremely expensive to encode billion-token corpus with billion-parameter models. Therefore, we directly start with the two-stage implementation of KNN-LM and gradually add the methods applied in NEST. As we can see, adding the RRC component gives the first effectiveness boost. The second dynamic span selection method does is crucial to give consistent attribution for consecutive spans and tokens. The last relaxed speculative decoding method further improves the final generation quality.
to increase the effectiveness, yet it
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that NEST provide better generation (Section 4.4), attribution (Section 4.6), and latency (Section 4.5) compared to the base LM and &NN-LM models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the complete algorithm of NEST Algorithm and implementa- tion in Appendix A for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
The code base is released at https://githe: com/facebokresearcch WEST/tree/main All the datasets we used are publicly available as discussed in Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
cisscccccyy)) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips.co.ooocc cwtppblic/geteeeeeceeccccceeecce gewgguuiiceessiie wwt.SSummissiiooooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details about datasets, inference, evaluation. hyper- hyper-paramette parameters, and baseline Section 4 and Appendix B. Our method is training-free method and therefore does not involve training-related hyper-parameters such as training steps and learning rate.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This paper does not include significant tests considering the performance gap between the proposed approach and the baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computational resources and hardware information for inference are reported in Section 4.5 and Appendix A. Our method does not involve training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp://neette httpar:///ee....//eee httppppllictthetiiiiieeeoo
Answer: [Yes]
Justification: Our paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 8.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our method is general framework that can be applied to different language models and knowledge sources, and therefore does not require safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Except the internal code base, all the existing assets including code and data used in this paper are properly cited in Section 4.2 and Appendix B.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/comm/ttt/o datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "UTGG2h8JUC", "reviewer": "Reviewer_nNiG", "summary": "The paper introduces NEST, a novel semi-parametric language modeling approach that enhances the generation quality and attribution of Large Language Models (LLMs) by incorporating real-world text spans. NEST employs a two-stage k-NN search and speculative decoding, achieving improved performance and reduced inference time.", "questions": "How does NEST compare with other advanced language models in terms of handling long-tail knowledge?\n\nWhat are the implications of using NEST for models pre-trained with different objectives or datasets?\n\nCould the authors elaborate on the impact of NEST on the diversity and creativity of LLM generations?\n\nOn page 2, the authors mention a 1.8× speedup; could they specify if this is consistent across different model sizes?\n\nHow does NEST handle potential biases in the retrieved text spans from the corpus?\n\nCan the authors provide more insight into the decision process behind the choice of hyperparameters?\n\nIs there a risk of overfitting the corpus used for training the key-value datastore?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The two-stage k-NN search is a smart optimization that balances search accuracy and efficiency.\n\nThe paper demonstrates significant improvements in generation quality and speed, offering a competitive edge over traditional methods.\n\nThe approach of providing direct attribution to sources is valuable for enhancing the reliability of LLMs.", "weaknesses": "The paper could benefit from a more detailed comparison with state-of-the-art methods.\n\nThe generalizability of NEST across different domains and languages needs further exploration.\n\nThe potential impact of NEST on the in-context learning ability of LLMs is not thoroughly discussed.\n\nThe paper lacks a comprehensive analysis of error rates and statistical significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the contributions of NEST. Below, we address each weakness point raised:\n- **Comparison with SOTA models**: We appreciate the reviewer's comment on comparing NEST with state-of-the-art (SOTA) models. To clarify, NEST is an algorithm that can be used to combine any SOTA LLMs with a desired data store to improve generation and attribution. While the retrieval-augmented generation (RAG) baseline outperforms NEST on several tasks, we highlight that NEST can be further combined with RA to achieve better performance (Table 1, last row). This flexibility is a key strength of our approach.\n- **Generalization of NEST across different domains**: We evaluated NEST across multiple domains, including general, legal, and medical fields, as well as a diverse set of subjects covered by MMLU (sec 4.4). Our results show that NEST consistently improves performance across these domains. While our current focus is mainly on English due to the availability of robust open-sourced multi-lingual LLMs, we believe that NEST can be applied to other languages with minimal modifications.\n- **In-context learning ability**: As mentioned in the Limitation section, NEST may not improve or even slightly hurt in-context learning since the demonstrations in the prompts are unlikely to appear in the natural text predominantly present in the retrieval data store. To quantify the impact, we conducted an experiment using Llama-2 (7B and 13B, not instruction-tuned) with 5-shot demonstrations on the Natural Questions dev set. Our results show that NEST does not improve performance in this setting. We will include these additional results in future versions of the paper. \n\n| Models |\t                EM |\t   F1 |\tALR|\n| -------- | ------- | ------- | ------- |\n| Llama-2-7B |\t24.05\t| 33.53\t| 27.10 |\n| +NEST\t    |          23.50 |\t32.87 |\t27.06 |\n| Llama-2-13B |\t30.34\t| 41.17\t| 33.76 |\n| +NEST\t  |            30.07\t| 40.77\t| 34.10 |\n\nFor the Questions:\n- **Handling long-tail knowledge**: Semi-parametric LMs such as NEST have stronger capabilities in memorizing long-tail facts because of the direct access to a non-parametric data store. Our experiment on MedMCQA, which involves medical questions that are often \"long-tail,\" shows that NEST outperforms the base LM model by 0.7~1 point in answer recall (Table 1).\n- **Different base LMs**: NEST is primarily designed for auto-regressive language models to improve text generation. Within this scope, NEST is designed to be flexible and can be combined with different LLMs and data stores in zero-shot, and we used Llama-2-chat as an example in the paper. We notice the notion of “models **pre-trained with different objectives or datasets**” can be ambiguous, and we’re happy to further clarify if we misunderstood the reviewer’s question.\n- **Diversity and creativity**: While NEST primarily focuses on improving factuality and attribution, we believe that our approach can be extended to incorporate diversity and creativity in future work. The Confidence-Based Output Interpolation mechanism we propose (sec 3.2) can determine whether to interpolate the LM prediction with the retrieved evidence. For creative generation scenarios, NEST will primarily rely on the base LLM for generation. \n- **Efficiency across different model sizes**: The speedup is not the same across models of different sizes. The overall generation latency per query is dominated by the LM forward pass and the rejection sampling process, while the other parts (retrieval and index building) stay almost constant. Therefore, the speedup is more obvious for larger LLMs and less prominent for smaller models, which is consistent with other speculative decoding methods.\n- **Biases in retrieval**: The Confidence-Based Output Interpolation mitigates the noise and biases in retrieval. By using the confidence of the retrieval, the LLM can adjust the frequency to interpolate with the retrieval distribution. This is verified in the TruthfulQA experiments, where adversarial queries are prevalent. NEST outperforms normal RAG methods by 0.3~0.73 points in Rouge-1 (line 219 - 223).\n- **Choice of hyperparameters**: We provide a detailed hyperparameter tuning scheme in Appendix A and B. The parameters $\\alpha$ and $\\tau$ in Equation (4) reflect the prior of how trustworthy the retriever is, and using larger $\\alpha$ means trusting the retriever more.\nFor $\\delta$ in Equation (5), we find 0.5 works best across all models and tasks, which is reasonable as the interpolation coefficient is well-calibrated in Equation (4).\nFor $\\gamma$ in Equation (6), it determines how often the LM rejects a segment retrieved from the corpus. A smaller $\\gamma$ means less rejection. We find that using a larger $\\gamma$ for a stronger LM (more rejection) tends to work better. \n- **Risk of corpus overfitting**: We would like to clarify that our LLMs and retriever are not fine-tuned on the corpus we used for retrieval. On the other hand, we tuned the hyperparameters in Equation (4), (6), and (7) for each retrieval corpus, assuming that the information distribution and quality of the data were unknown until tested. We recommend practitioners adopting the NEST approach also tune the hyperparameters on their retrieval corpus. This is cheap to complete and may significantly boost downstream performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QGHBJW29jC", "reviewer": "Reviewer_L2uT", "summary": "This work presents Nearest Neighbor Speculative Decoding (NEST), a technique to better inject real-world text spans into the output of existing language models. NEST is a kNN-LM approach adding an initial passage retrieval step. During inference, NEST uses Relative Retrieval Confidence (RRC) for confidence-based interpolation, dynamically extends selected tokens to include text spans when confidence is high, and employs a relaxed speculative decoding process that accepts only highly probable token spans.\n\nAccording to the evaluation conducted by the authors with Llama 2 chat, NEST significantly outperforms both the base LM and standard kNN-LM, in terms of speed and accuracy.", "questions": "It’s unclear to me what are the most important contributions of this work. What is really new and has the most impact on the results? Even with the help of the related work section I cannot answer this question, hence my low confidence score", "limitations": "The limitations are pointed out and sufficiently discussed.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- well-written: The paper is clear and easy to understand. Figure 1 itself is enough to understand what the paper presents\n- well-motivated: The paper explains the problem that it tries to solve and provides the motivations for why it needs to be solved \n- Extensive experiments demonstrating that the proposed approach works", "weaknesses": "- It’s unclear to me what are the most important contributions of this work. What is really new and has the most impact on the results? Even with the help of the related work section I cannot answer this question, hence my low confidence score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and would like to clarify the main contributions of NEST:\n- **Source Attribution**: By injecting retrieved segments into the generation of LLM, NEST provides direct attribution for the generation at a span level, enabling users to verify the reliability of the generation. Methods based on prompting/finetuning to provide citations can still make mistakes, while for NEST, given the same set of retrieved documents, the attribution of the claim is always correct since the segments are directly taken from the source.\n- **Copyright Protection**: Using LLMs for content generation is controversial because these models were probably trained on copyrighted content, which may infringe on the rights of data producers. Storing the proprietary data separately in a data store avoids memorizing high-risk knowledge; providing direction attribution to each claim avoids the legal risk of using licenced data without acknowledgement.\n- **Factuality**: Grounding the generation using spans taken from a verified knowledge source can improve the factuality of LLM generation. Unlike normal RAG settings where the retrieved evidence is injected into the prompt, NEST can choose not to use the retrieved results during interpolation. This is also showcased in adversarial question-answering tasks such as TruthfulQA, where RAG can easily be misled by the retrieved “evidence” that is seemingly related but incorrect, while NEST can avoid such misleading knowledge by not interpolating with them at the output. Please see line 219 - 223 for more details.\n\nMoreover, we would like to point out some technical improvements NEST made in LLM and retrieval-augmented generation:\n- **Dynamic Interpolation**: Retrieval is not always required for tasks such as creative writing, while RAG and baseline kNN-LM methods both used a fixed scheme to incorporate retrieved “evidence”, which might hurt the generation in such cases. Other methods that can adaptively choose to combine the retrieved knowledge such as CoG or Retro require further finetuning, which couples the LLM with the knowledge base and might not be able to transfer to new databases. On the other hand, NEST can dynamically adjust the interpolation between LLMs and retrievers in a zero-shot fashion, making it easy for developers to update the LLM and knowledge base.\n- **Generation Efficiency**: Injecting the retrieved segment into LLM generation also improves the generation speed thanks to speculative decoding. We did not optimize for this aspect but using a faster and better retriever should further improve the generation latency.\n- **Attribution-Fluency Tradeoff**: It might not be obvious in the paper but injecting segments into LLM generation (attribution) is risky as it might introduce artifacts during transition such as repetition or grammatical errors (fluency). This is hard to evaluate but has a major impact in real application. Our proposed dynamic span selection and relaxed speculative decoding method provide a solution for attribution-fluency trade-off by tuning the selection and rejection hyperparameters in Equation (6) and (7).\n\nIn summary, NEST not only solves the technical challenges in previous generation methods, such as when to combine with retrieval and attribution-fluency tradeoff but also has real impacts on applications such as copyright protection and providing accurate attribution for verification. We again thank the reviewer for pointing this out and we will elaborate on the contribution and novelty of NEST in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SFE8WT1fd7", "reviewer": "Reviewer_vMVQ", "summary": "This paper:\n\n* Introduces NEST, a semi-parametric language modeling approach that integrates real-world text spans into language model generations.\n* Enhances generation quality and reduces latency by using token-level retrieval and speculative decoding.\n* Outperforms conventional kNN-LM and competes well with in-context retrieval methods.\n* Demonstrates significant improvements with various models.", "questions": "* How scalable is the NEST approach in real-world scenarios?\n* What measures are taken to mitigate the impact of noise and errors introduced during the data retrieval stages?\n* How does NEST address potential biases that may arise from the retrieved data?\n* How well does NEST adapt to the evolving nature of language?", "limitations": "* The overall effectiveness of NEST is contingent upon the precision of both stages of text retrieval.\n* The approach may struggle with ensuring that the retrieved content is contextually relevant and free from biases.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Introduces a novel semi-parametric language modeling technique for improved attribution and generation quality.\n* Demonstrates a significant increase in speed and efficiency in language model generation.\n* Outperforms conventional kNN-LM and shows competitive results against in-context retrieval methods across various knowledge-intensive tasks.\n* Effective across a range of tasks including text completion, question answering, and factuality-aware generation, showcasing the method's adaptability to different content requirements.", "weaknesses": "* Performance heavily relies on the accuracy of the first-stage passage retrieval and second-stage token retrieval.\n* The retrieval process may still be complex and resource-intensive for practical deployment.\n* The gains in performance and efficiency are less pronounced in larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the novelty and effectiveness of NEST. Below, we address each weakness point raised:\n- *“Performance heavily relies on the accuracy of the first-stage passage retrieval”*:  We acknowledge the performance of NEST is impacted by the first-stage passage retrieval. However, all retrieval-augmented models have this limit, and our work focuses on improving 1) the integration of the retrieved content into the LLMs and 2), rather than further improving the first-stage retrieval accuracy.\n- *“Retrieval process may still be complex and resource-intensive for practical deployment”*: We politely disagree that the retrieval process is complex. Our two-stage retrieval system is designed to reduce the resources required for building a token-level datastore of the entire corpus, which is a significant advantage over other approaches. We have empirically demonstrated that our approach strikes a good balance between accuracy and efficiency.\n- *“The gains in performance and efficiency are less pronounced in larger models”*: We understand the reviewer's point that the gains in performance and efficiency are less pronounced in larger models. However, this is a natural consequence of the model's increased capacity to memorize nuanced facts. Nevertheless, retrieval augmentation remains meaningful in practice for long-tail knowledge, faster response, source attribution, and knowledge update. Moreover, our results show that the speedup is more prominent in larger models (the overall generation latency per query is dominated by the LM forwarding and rejection sampling process), making NEST a more efficient choice for large-scale applications.\n\nFor the Questions:\n- **Scalability of NEST**: We believe that our experiments are representative of real-world scenarios, with the largest model having 70 billion parameters and the largest knowledge store having 33 million passages. Further engineering optimization can enable scaling to even larger models and knowledge stores.\n- **Measures to reduce noises and biases in retrieval**: The Confidence-Based Output Interpolation (sec 3.2) is our exact measure to mitigate the noise and biases in retrieval. Not incorporating the retrieval information at the input reduces the risk of biases and interpolating based on the token-retrieval confidence further filters out contradictory knowledge. This is also verified in the TruthfulQA experiments when adversarial queries are encountered, where NEST outperforms normal RAG methods by 0.3~0.73 points in Rouge-1. Please see line 219 - 223 for more details.\n- **Adapting to changes in language**: We interpret the reviewer's question as referring to the evolving nature of language, where new phrases and knowledge are added frequently. NEST's zero-shot combination of LLMs and knowledge sources enables it to adapt to these changes more easily. By tuning the hyperparameters on the development set, NEST can combine different LLMs with different knowledge sources, making it more flexible and adaptable to the evolving nature of language.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iTlh5QBezj", "reviewer": "Reviewer_kPHS", "summary": "The paper presents a new semi-parametric language modeling approach that can incorporate text spans from a datastore into LLM-based generation improving both quality and attribution of generated texts. They propose a two-step approach that requires constructing an on-the-fly token-level datastore based on a small number of retrieved passages from a passage-level datastore. The current token is generated from the mixture distribution between the base LLM and the retrieved token-level distribution interpolated using a relative retrieval confidence score capturing the uncertainty of the token retriever. Furthermore, the approach enables extending the generation from token to an n-gram span based on a speculative decoding that rejects or accepts the tokens in the continuations, thus enabling span-level generation, improving efficiency.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments across tasks and datasets shows the efficacy of the proposed approach over standard LLM decoding, KNN and Retrieval augmented incontext learning variants.\n2. Ablations show how the relaxation factor used for speculative decoding can enable flexible-length n-gram continuations to be incorporated based on the domain and can provide a good tradeoff between accuracy and attribution.", "weaknesses": "1. Multiple-token nearest neighbor generation has been proposed in prior work but was not discussed or compared, see: [1] Chunk-based Nearest Neighbor Machine Translation (https://aclanthology.org/2022.emnlp-main.284/).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the novelty and effectiveness of our work. The chunk retrieval and the neighbour selection process in the suggested paper are indeed related to NEST. For example, it is possible to improve the span selection process of NEST with the batch-beam-level neighbours in the paper which evaluate multiple chunk candidates simultaneously. We thank the reviewer again and will add a more detailed discussion of this paper in the related work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a competitive technique to incorporate real-world text spans of arbitrary length into LM generations, thus enhancing also attribution.
All major concerns from reviewers appear resolved during rebuttal.
Decision: Accept (poster)