Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
Authors: Boxin Wang
Wei Ping
Peng Xu
Lawrence McAfee
Zihan Liu
Mohammad Shoeybi
Yi Dong
Oleksii Kuchaiev
Bo Li
Chaowei Xiao
Anima Anandkumar
Bryan Catanzaro
Venue: EMNLP 2023 Main (2023)
Keywords: large language model
pretraining
retrieval augmentation
retro
knowledge retrieval
TL;DR: We perform a comprehensive study on a scalable pre-trained retrieval-augmented LM (i.e., RETRO), which highlights the promising direction of pretraining autoregressive LMs with retrieval as future foundation models.

Abstract: Large decoder-only language models (LMs) can be largely improved in terms of perplexity by retrieval (e.g., RETRO), but its impact on text generation quality and downstream task accuracy is unclear. Thus, it is still an open question: shall we pretrain large autoregressive LMs with retrieval? To answer it, we perform a comprehensive study on a scalable pre-trained retrieval-augmented LM (i.e., RETRO) compared with standard GPT and retrieval-augmented GPT incorporated at fine-tuning or inference stages. We first provide the recipe to reproduce RETRO up to 9.5B parameters while retrieving a text corpus with 330B tokens. Based on that, we have the following novel findings: i) RETRO outperforms GPT on text generation with much less degeneration (i.e., repetition), moderately higher factual accuracy, and slightly lower toxicity with a nontoxic retrieval database. ii) On the LM Evaluation Harness benchmark, RETRO largely outperforms GPT on knowledge-intensive tasks, but is on par with GPT on other tasks. Furthermore, we introduce a simple variant of the model, RETRO++, which largely improves open-domain QA results of original RETRO (e.g., EM score +8.6 on Natural Question) and significantly outperforms retrieval-augmented GPT across different model sizes. Our findings highlight the promising direction of pretraining autoregressive LMs with retrieval as future foundation models.  We release our implementation at: https://github.com/NVIDIA/Megatron-LM/tree/main/tools/retro.

Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
Boxin Wang")
Wei Ping*
Peng Xu*
Lawrence McAfee"
Zihan Liu?
Mohammad Shoeybi?
Yi Dong"
Oleksii Kuchaiev
Bo Li
Chaowei Xiao
Anima Anandkumar Bryan Catanzaro?
Abstract
2020), BART (Lewis al., 2020a)), have ob- tained state-of-the-art results for various NLP tasks. Among them, the autoregressive LMs like GPT- (Brown et al., 2020) and GPT-4 (OpenAI, 2023) demonstrate noticeable in-context learning abil- ity and excellent long-form text generation results. Due to its importance, the community has spent considerable efforts to scale up such autoregres- sive generative LMs with more data and param- eters and observed significant breakthroughs in a variety of real-world applications (e.g.. Brown et al., 2020), including open-ended text genera- tion and various downstream tasks (e.g., ques- tion answering). The successful public exam- ples include GPT-3 (w/ 170B parameters) (Brown et al., 2020), Gopher (280B) (Rae et al., 2021), Megatron-Turing (530B) (Smith et al., 2022), and PaLM (540B) (Chowdhery al., 2022).
Large decoder-only language models (LMs) can be largely improved in terms of perplex- ity by retrieval (e.g., RETRO), but its impact on text generation quality and downstream task accuracy is unclear. Thus, it still open question: shall we pretrain large autoregres- sive LMs with retrieval? To answer it, we per- form comprehensive study scalable pre- trained retrieval-augmentue LM (i.e., RETRO) compared with standard GPT and retrieval- augmented GPT incorporated at fine-tuning or inference stages. We first provide the recipe reproduce RETRO up to 9.5B parameters while retrieving text corpus with 330B tokens. Based on that, we have the following novel find- ings: i) RETRO outperforms GPT text gen- eration with much less degeneration (i.e., repe- tition), moderately higher factual accuracy, and slightly lower toxicity with a nontoxic retrieval database. ii) On the LM Evaluation Hameess benchmark, RETRO largely outperforms GPT knowledge-intensive tasks, but is par with GPT on other tasks. Furthermore, we intro- duce simple variant of the model, RETRO++, which largely improves open-domain QA re- sults of original RETRO (c.g., EM score +8.6 Natural Question) and significantly outper- forms retrieval-augmeened GPT in both fine- tuning and zero-shot evaluation settings. Our findings highlight the promising direction of pretraining autoregressive LMs with retrieval future foundation models. We release our implementation at: https://github comNN VIDDA/Meggtronn-LMMrerrr..
Although large-scale autoregressive LMs have achieved huge successes, they also suffer from sev- eral weaknesses. First, it requires huge number of model parameters to memorize the world knowl- edge, which makes it costly for deployment. Sec- ond, it is difficult to safeguard factual accuracy, which may provide users with incorrect informa- tion (Lee al., 2022). Third, it is expensive to update the model knowledge learned during pre- training with up-to-date facts (Meng al., 2022). yielding outdated answers (Lewis et al., 2020b).
To mitigate the problems above, one line of research proposes to improve language models with retrieval. The retrieval process can be inte- grated into LMs at: i) fine-tuning stage (Karpukhin et al., 2020; Lewis et al., 2020b; Guu et al., 2020), or ii) pretraining stage (Borgeaud et al., 2022; Izacard et al., 2022). Most previous work aug- ments BERT encoder-decoder LMs with trieval at fine-tuning stage, demonstrating suc- cesses for knowledge-intensive NLP tasks (Guu et al., 2020; Karpukhin et al., 2020; Lewis et al., 2020b; Khandelwal et al., 2020). However, it re-
1 Introduction
Large language models (LMs), including masked LMs (e.g., BERT (Devlin al., 2018)), autore- gressive LMs (e.g., GPT (Brown et al., 2020)), and encoder-decoder LMs (e.g., T5 (Raffel et al.,
"Equal contribution, : done during an internship at NVIDIA. PUUC. "NVIDIA. University of Wisconsin, Madi- son. *Correspondence to: Wei Ping swping @ comd

---

mains relatively underexplored to pretrain autore- gressive (decoder-only) LMs with retrieval, espe- cially considering the noticeable success of Chat- GPT (OpenAI, 2022) that underscores the extreme importance of the autoregressive L.M..
evaluation in different scenarios.
To minimize the discrepancy variables between RETRO and GPT, we use the same decoder architec- ture, same hyper- parameters, and same pre- pre-training corpus to pre-train RETRO and GPT given the same number of pre-training steps. We highlight our novel findings for RETRO and GPT as follows:
Most recently, RETRO (Borgeaud et al., 2022) proposes to pretrain autoregressive LMs with retrieval module, which practically scalable to large-scale pretraining from scratch by retrieving billions of token and largely reduces model parame- ters while achieving lower perplexity than standard GPT. It also provides the flexibility to update the knowledge stored in LMs (Petroni et al., 2019) by updating the retrieval database without train- ing LMs again. The success of pretraining LMs with retrieval raises an important question for the community if want pretrain autoregressive LMs in the future: Shall we pretrain autoregres- sive (decode-only) LMs with retrieval by default or not? However, previous work (Borgeaud et al., 2022) misses the important evaluation on whether the model like RETRO could obtain comparable or even better results in terms of open-ended text gen- eration and various NLP downstream tasks, apart from lower perplexity on the held-out dataset com- pared standard GPT.
2.1 Text Generation
We conduct a systematic study (see $5) to under- stand and analyze RETRO by evaluating its open- ended text generation quality via human and auto- matic evaluations. RETRO exhibits better perfor- mance than GPT with considerably less repetition, moderately higher factual accuracy, and slightly lower toxicity levels. RETRO is on par with GPT terms of fluency, coherence.
2.2 LM Evaluation Harness Benchmark
In terms of zero-shot evaluation on the standard benchmark, RETRO can overall improve upon the GPT across different tasks, significantly outper- forming GPT on knowledge-intensive tasks such Hellaswag and BoolQ while achieving similar per- formance on other tasks. Specifically, we evaluate the zero-shot capabilities of RETRO and GPT on nine representative NLP downstream classification tasks (see $6). Additionally, our findings demon- strate that RETRO can leverage retrieved neighbors and significantly improves accuracy for knowledge- intensive tasks in zero-shot evaluations. In contrast, incorporating these retrieved neighbors directly dur- ing the inference stage can hurt GPT's performance. These results further substantiate the potential of RETRO, which is pre-trained with retrieval capabil- ities, as a promising approach.
To answer the above question and bridge the missing gap, we perform an extensive study on RETRO, as to the best of our knowledge, RETRO is the only retrieval-augmentee autoregressive LM that supports large-scale pretraining with retrieval the massive pretraining corpus with hundreds of billion or trillion tokens. Our comprehensive study sheds light on the promising direction of pertain- ing autoregressive LMs with retrieval to serve as future foundation models, as they overall outper- form standard GPT models in terms of perplexity, text generation quality, and downstream task perfor- mances, especially for wwwllede-intensiin tasks, including open-domain QA.
2.3 Open-domain QA
For open-domain QA tasks. RETRO achieves considerably superior performance than retrieval- augmented GPT that incorporates retrieval dur- ing fine-tuning across different model sizes and datasets. Specifically, we propose a variant of the model, RETRO++, for open-domain QA that feeds the most relevant evidence into the decoder and more evidence into its encoder, which is different from the original version (Borgeaud al., 2022). RETRO++ can largely improve the exact matching score (EM) on Natrual Question from 40.9% 54.1%, which is significant higher than the 45.5% reported by the original RETRO.
2
Key Findings
We successfully reproduce and pretrain RETRO (Borgeaud et al., 2022) from scratch', with param- eter sizes ranging from 148M up to 9.5B by re- trieving from a text corpus with over 330B tokens. In addition, we discuss the inference strategy of RETRO for text generation that is not covered in Borgeaud al. (2022), and perform large-scale
'The official implementation and pretrained checkpoints are not open-sourced.

---

Model
If Retrieval Tokens
When Involve Retrieval
Architecture
Initialization
Re-indexing
Name
RETRO (Borgeaud et al.)
0(10"))
Pretraining
decoder-only
From Scratch / Pretrained GPT
Atlas (Izacard al.)
0(109)
Pretraining
encoder-decoder
Pretrained TS
Yes
REALM (Guu et al.)
O(10")
Pretraining
encoder-only
Pretrained BERT
Yos
RAG (Lowis et al.)
O(10")
Fine-tuning
cnoodcr-docoder
Pretruined BART
No
DPR (Karpukhin et ul.)
O(10")
Fine-tuning
encoder-only
Pretrained BERT
No
FiD (Izacard and Grave)
O(10))
Fine-tuning
encodcr-decoder
Pretrained T5
KNN-LM (Khandelwal et sl.)
(110))
Inference
decoder-only
Pretrained GPT
No
Table 1: Comparison of different retrieval-uggmented models terms of H/ retrieval tokens, which stage incorporate retrieval
into LMs, the architecture of the backbone LM, whether it requires initialization from the existing LM checkpoint, and whether it requires expensive re-indexing. RETRO is the most scalable retrieval auggeeted LM to its chunk-level retrieval and scalable decoder-only autoregressive LM backbone (Thoppilan al., 2022; Brown al., 2020; Smith al., 2022; Chowdhery et al., 2022) without expensive retrieval index refresh.
3 Related Work
strategies for retrieval-augmentee BERT model in open-domain QA. Their pretraining involves asyn- chronous re-embedding and re-indexing all docu- ments every several hundred training steps, which quickly becomes impractical for training corpus with trillion tokens. Atlas (Izacard et al., 2022) uses a similar approach but augments the T5 archi- tecture (Raffel et al., 2020) with retrieval at both pre-training and fine-tuning, Before pretraining, it first initializes the encoder-decoder LM backbone with pretrained T5, and the dense retriever with pretrained Contriever (Izacard al.). During pre- training, also applies asynchronous index refresh every 1000 steps.
Retrieval has been applied in various NLP tasks for years, including question answering (QA) (e.g., Bilotti al., 2007), machine translation (e.g.. Zhang et al., 2018), and conversation (Shuster et al., 2021: Thoppilan et al., 2022; Komeili et al., 2021). In particular, language models have been augmented with retrieval at different stages, includ- ing inference time (Khandelwal et al., 2020; Yo- gatama et al., 2021), fine-tum fine-tuning stage (Karpukhin al., 2020; Lewis al., 2020b; Guu al., 2020), and pretraining stage (Borgeaud et al., 2022; Izac- ard et al., 2022).
LMs have been augmented with retrieval at the fine-tuning stage for downstream tasks, primarily for open-domain QA. DPR (Karpukhin et al., 2020) finetunes one BERT to encode questions and the other BERT to encode answers within a dual en- coder framework, using contrastive loss to align the hidden representations of question and corre- sponding answer. RAG (Lewis et al., 2020b) stud- ies the tuning recipe for retrieval-auge -uggented generation models, especially on open-domain QA tasks. FiD (Izacard and Grave, 2021) improves RAG with a better LM backbone T5, and fuses multiple retrieved passages to the decoder during fine-tuning to further improve QA accuracy. We- bGPT (Nakano et al., 2021) leverages web search engine and fine-tunes GPT using reinforcement learning with human feedback (RLHFF) for refer- ence generation and factuality improvement, which is orthogonal to our work that focuses on pretrain- ing with retrieval. The proposed RLHF can be applied to RETRO as well.
In contrast, RETRO (Borgeaud et al., 2022) em- beds and indexes the whole training corpus at chunk-level (e.g., chuck size 64) with frozen BERT before pretraining. During pretraining, the model relies on trainable bidirectional encoder to embed the retrieved chunks of raw text. The GPT decoder further "select" the relevant piece of evidence from the encoder side by a chunk-wise cross-attention. This architecture design enables LM pretraining on hundreds of billion tokens by retrieving from trillion tokens. See Table for a complete comparison of retrieval-augmennte LMs.
Model and Implementation
In this section, we first introduce preliminaries of RETRO, then provide detailed recipe of our imple- mentation, including retrieval database, pretraining. and retrieval-augmented finetuning and generation.
4.1 Preliminaries of RETRO
RETRO is an autoregressive language model en- hanced with a retrieval module that utilizes chunk- wise retrieval, enabling it to scale up to trillions of
REALM (Guu et al., 2020) performs both un- supervised pretraining and supervised fine-tuning

---

Inghts
Retro
Retivevll Database
Retro
Rarrinall Darabase
Retro
Retriooal Databese
lepas
Inpat
Contect quetjet
Jnsiver
CDNEN
spach
the rutricoul
rightmest
Left cuedtooniomeett
genaration: sausaliy
and generate token (Retreeal
(a) Use "left padding" Rule
(b) Retrieval step
(c) Separate question answer chunks
Figure 1: Visualization of padding design for RETRO.
Small
Medium
XL
XXL
22 centroids accelerated with Hierarchical Navi- gable Small World graphs (Malkov and Yashunin, 2018) 8) to speed up the query. We also encode the embeddings with optimized product quantization (Gray and Neuhoff, 1998; Ge et al., 2014) to com- press memory overhead and further improve the query throughput. As result, we can achieve 4ms per query over the whole pretraining corpus aver- aged for each chunk DGX-2H node. One may find more details in Appendix $A.
GPT
17.76
13.18
10.18
7.86
RETRO (k- 2)
12.99
10.06
8.10
6.72
Table Validation perplexity of pretrained GPT and RETRO held-out dataset. We report the results neighhors in this Table, and we ohserve the same trend of improvements with larger as in Borgeaud et al. (2022).
tokens. The model splits both the input sequence and retrieval datastore into sequences of chunks. RETRO retrieves nearest neighbor chunks from the retrieval database using the previous chunk and fuses this information with the context from pre- ceding chunks to guide the generation of the next chunk. To maintain causality, the model can only use the nearest neighbors of the previous chunk for the autoregressive generation.
4.2.2 Pretraining RETRO Models
We use the same transformer configurations (#/ lay- ers, hidden size, attention heads) and pretrain both RETRO and standard GPT from scratch. Specifi- cally, we pretrain RETRO actoss different param- eter sizes, ranging from 148M (Small), 410M (Medium), 1.5B (XL), and 9.5B (XXL). We also use the same pretraining schedules to pretrain RETRO and GPT given the same number of steps. We list the validation perplexity of GPT and RETRO after pretraining in Table 2. We present more de- tails in Appendix $B, including pretraining sched- ules, computational cost (GPU hours), and model architectures.
4.2 Implementation
As RETRO has official open-source implementa- tion and pretrained checkpoints, we reproduce and pretrain RETRO from seratch on own.
4.2.1 Retrieval Database
We build the retrieval database with the whole pretraining dataset mentioned in $B. In this way. RETRO and standard GPT of similar size are fair comparisons, they are pretrained using the same information from the pretraining corpus. The re- trieval database is a key-value database, where val- ues are chunks split from the pretraining corpus, and the keys are corresponding BERT embeddings. Our pertaining dataset with 330B tokens yields a retrieval database consisting of 5.3B chunks in total with chunk size m = 64.
4.2.3 Retrieval-augmeenee Generation
We discuss the generation and inference recipe in the batch-processing mode for RETRO, which is missing from the previous literature.
"Left Padding" Rule. The chunk-wise retrieval of RETRO improves scalability but enforces chunk- wise alignment constraints, leading to issues in con- ditional generations with short contexts. When the sequence length is less than the chunk size, RETRO cannot utilize its retrieval capability there no previous chunk for retrieval. Instead, RETRO adds padding tokens to the left of the context, allowing RETRO to leverage the retrieved neighbors from the previous context to guide the generation of the
Retrieval Index. We use the Faiss index (Johnson al., 2019) as the implementation for the dense re- triever search for approximate nearest neighbors in the BERT embedding space. We configure the Faiss index to cluster the dense embeddings into

---

Small
Medium
XXL
Metries
GPT
RETRO
GPT
RETRO
GPT
RETRO
RETRO
Repetition
2.86%
2.26%
1.70%
1.50%
1.44%
0.96%
1.40%
1.12%
Self-BLEU
0.29
0.3
0.29
0.3
0.29
0.29
0.31
0.31
Zipt Coefficient
0.98
0.98
0.96
0.98
0.97
0.98
0.96
0.96
Table Automatic evaluation on text generation quality for RETRO and GPT across different sizes.
next token (F'igure la). We summarize this general principle in RETRO as the "Ieft padding" rule, as it can leverage the contextual information for re- trieval to the most. This rule remains preferable for input sequences larger than the chunk size, as it ensures the closest and rightmost context used for retrieval, making it more relevant for next token prediction (see Figure lb).
Open-ended Text Generation
In this section, we delve into the problem of open- ended text generation, which refers to tasks of gen- erating coherent continuation given the preceding prompt. Given that this problem for RETRO has never been studied before, manage to bridge the gap and evaluate the open-ended text generation of RETRO compared to GPT from three aspects: a) text quality, b) factuality, and c) toxicity.
Frequency of Retrieval. In order to efficiently gen- erate long sequences with RETRO, note flexi- ble trade-off between Aurieval-augmente genera- tion and computation overhead. The direct method involves retrieval every decoding step, maximiz- ing the use of the retrieval module but increasing computational overhead (Figure lb, retrieval step 1). Another approach retrieves neighbors at the frequency of the chunk size, reducing overhead but sacrificing accuracy (Appendix Figure 3b, retrieval step 64). To balance these factors, we introduce flexible retrieval step, which allows model practi- tioners to choose how many tokens to generate with the current retrieved neighbors before updating the context. Smaller retrieval steps are preferred for downstream tasks with short answers to ensure ac- curate neighbors, while larger steps are used for efficient generation of long passages. We provide more details in Appendix $C.
5.1 Text Quality
We perform both automatic and human evaluations.
5.1.1 Automatic Evaluation
Evaluation Metrics. We follow prior work (Holtz- man et al., 2019; Zhu et al., 2018) and consider the following metrics: Repetition % measures percentage of the generations containing repetitive phrases, SELF-BLUE evaluates the diversity of the generations, and Zipf Coefficient measures the use of vocabulary. See detailed definition and evaluation setup in Appendix $D.1.
Experimental Results. Our results are shown in Table 3. We note that RETRO can reduce the per- centage of repetition compared with GPT by a large margin across different sizes. Specifically, RETRO averagely mitigates 21% of repetitions compared with GPT across different sizes. This suggests the retrieval module can help reduce text degeneration by referencing retrieved human text. Regarding vo- cabulary use and generation diversity, we do not ob- serve major differences between GPT and RETRO, which implies these properties are primarily depen- dent the decoder component of LMs.
4.2.4 Batched Training for Downstream Tasks
When fine-tuning RETRO for downstream tasks (e.g.. QA), it is crucial to separate context or question from the candidate answer chunk to maintain causality in autoregressive modeling. This leads to a modified "left padding" rule: pad context chunks from the left and answer chunks from the right (Figure Ic). Padding aligns input sequences with the chunk size, enabling batch-mode training and inference for faster evaluation. By adding padding chunks to the right, sequences with varying chunk numbers can be processed together, further improving efficiency.
5.1.2 Human Evaluation
We also conduct human evaluations to further ver- ify the quality of the generated text. Evaluation Metrics. We ask human annotators to annotate each generation with fluency scores, which measure the human readability and grammat- ical errors from (Not human- readablee to 5 (Very fluent), and coherence scores, which measure the

---

Decoding
Models
Nian +
Factual
Entaily
NE
Nonfactual
Entaile
Models
MCI1
QA Format
MC21
MCIT
Nuil Format
MC21
Top-p=0.9
RETRO GPT
52.14% 52.42%
3.11%
56.75%
2.06%
GPT
0.222
0.377
0.234
0.435
2.3%
56.82%
2.04%.
RETRO (pretraining)
0.239
0.382
0.248
0.439
RETRO GPT
37.42% 39.87%
16.66%
42.45%
10,88%
RETRO (wiki)
0.242
0.437
Greedy
12.91%
45.02%
8.75%
RETRO (DPR)
0.245
0.439
(a) The factuality on FACTUALITY PROMPTS benchmark.
(b) The truthfulness TruthfulQA benchmark.
Table 4: Evaluation of factuality and truthfulness RETRO (XL) and GPT (XL).
relevance between the prompt and the correspond- ing continuations from (Not Relevant) to 5 (Very Relevant). More details can be found in $D.2.
ble 4b, where we observe the consistent conclu- sion that RETRO tend to be more resilient to hu- man falsehoods than GPT across different prompts (demonstrated in Appendix Table 12 and 13).
Experimental Results. We present the human vote histogram Appendix Figure 4. We observe that most votes concentrate on the regime of scores >= 3 for both relevance and fluency, which indi- cates that our generated text from both models is of high quality and closely related to the prompts. The differences between GPT and RETRO are subtle, with average relevance (3.726) and fluency (3.826) scores of RETRO slightly outperforming the aver- age relevance score (3.715) and fluency (3.818)
Since using the null format yields consistently better performance, we thus fix the prompt and conduct ablation studies with different retrieval databases. We observe that the pretraining database and DPR database yield the best factual accuracy. This suggests that (i) finetuned retriever (ii) trieval from larger corpus are helpful to improve factual accuracy. addition, the improvement mar- gin for RETRO on TruthfulQA not as significant as on PROUALS This is likely be- cause TruthfulQA is an adversarial dataset contain- ing difficult question types, such as logical false- hoods and indexical errors (c.g., "Are you artifi- cial intelligence?", more examples can be found in Appendix Table 14). In these cases, retrieval from the database does not effectively aid in answering such questions.
scores of GPT.
From both automatic and human evaluation, we can conclude that although the generation of RETRO adds some complexity, do not see any sign of the degeneration of RETRO compared to GPT. Moreover, RETRO is shown to be able to reduce the repetition and slightly improve text gen- eration quality.
5.2 Factuality
5.3 Toxicity
Factuality refers to being coherent to provide ground truth knowledge sources in NLP. We lever- age two well-established benchmarks (Lee al., 2022; Lin et al., 2021) evaluate the factual accu- racy of RETRO and GPT. We put detailed experi- mental setup and metrics in the $E.
The toxicity of LMs refers to the possibility of LMs that output toxic generations. In this study, we follow REALTOXICTYPROMPT benchmark (Gehman et al., 2020) to evaluate the potential tox- icity of RETRO and GPT.
Evaluation Metrics. Following Gehman et al. (2020), we report the Expected Maximum Toxic- ity, which evaluates the toxicity of the worst-case generation, well Toxicity Probability that esti- mates the empirical frequency of generating toxic language. See more details and setup in $F.
Factuality Prompts. We present the results on SACTUALITYPROMET
benchmark in Table 4a, where follow the protocol (Lee et al., 2022) and use different decoding strategies for evaluation. We observe that overall RETRO demonstrates better factual accuracy and makes fewer hallucinations. Our observation also aligns with (Lee et al., 2022) that different from top-p sampling with random- ness, greedy decoding gives the much better factual accuracy by sacrificing sample diversity. RETRO shows significant improvement over GPT using both top-p and greedy decoding algorithms.
Experimental Results. The toxicity of LMs are shown in Table 5. Compared to GPT, we note that RETRO with the pretraining corpus creases the toxicity of the generations. Moreover, observe more toxicity increases in toxic prompts than nontoxic prompts. This suggests that when prompting RETRO with toxic contexts, it is more likely to retrieve toxic evidence and thus amplify
in-
TruthfulQA. We also evaluate truthfulness in Ta-

---

Retrieval
Exp. Max. Toxicity (L)
Toxicity Prob. (L)
Models
Database
Full
Toxic
Nontoxic
Full
Toxic
Nontoxic
GPT
0.44
0.64
0.39
37%
74%
27%
RETRO (top-N 2. top-K 2)
Pretraining
0.46
0.66
0.40
40%
76%
30%
RETRO (top-N 5. top-K 2)
Pretraining
0,46
0,66
0,40
39%
77%
29%
RETRO (top-N = 10, Kop-K 2)
Pretraining
0.46
0.66
0.40
39%
76%
29%
RETRO (top-N = 2, lop-k' = 2)
Wiki
0.43
0.64
0.38
35%
73%
25%
RETRO (top-N = 5, top-k =
Wiki
0.43
0.64
0.38
35%
71%
26%
RETRO (top-N = 10, top- A 2)
Wiki
0.43
0.64
0.38
35%
71%
26%
Table 5: Evaluation of LM toxicity for GPT (XL) and RETRO (XL). Model toxicity is evaluated on REALToXICITY Full refers the full set of prompts, Toxic and Nontoxic refer the toxic and nontoxic subsets of prompts. means the lower. the better. RETRO filter from top-N nearest neighbors and select the top-K nontoxic neighbors for retrieval.
the issues. To confirm the toxicity amplification issue, we further conduct two sets of ablation stud- ies: (1) We save the retrieval evidence and calculate the Expected Mean Toxicity of both generations and retrieval evidence. We observe that the toxic- ity of retrieval evidence is 0.177, higher than the toxicity of the generations (0.146). (id) We change the retrieval database the Wikipedia database, which shows lower toxicity for retrieval evidence (0.132). As a result, we observe that RETRO with the Wikipedia retrieval database can help mitigate the toxicity of GPT as shown in Table 5, with the toxicity probability dropping from 37% to 35%. We also note that it is not very helpful to use larger N as nearest neighbors and filter the retrieval evidence by toxicity. We hypothesize the reason is that the similarity between input and retrieval evidence is limited with larger N, thus yielding low cross-attention on the retrieval evidence.
Retrieval-augmmented GPT at Inference time. We have seen that retrieval significantly improves RETRO across different downstream tasks in the zero-shot setting. In this ablation study, we append the retrieval evidence of RETRO to the beginning of the context to see whether retrieval can also be helpful for GPT inference time. We evaluate the zero-shot accuracy after prepending the top-1 trieval evidence. The results are shown in Appendix Table 16. We observe that directly prepending the evidence from the retricval database messes up the GPT context in the zero-shot setting, yielding low accuracy of around 24.5%. We hypothesize the rea- son is that the retrieval evidence can be noisy. With- out pretraining or proper fine-tuning, GPT in the zero-shot learning setting puts too much attention on the noisy evidence, thus giving low downstream accuracy.
Open-domain Question Answering
6 LM Evaluation Harness Benchmark
In this section, we study two widely used open- domain QA datasets, Natural Question (NQ) and TriviaQA.
Besides the open-ended text generation, it is also important to examine the generalization RETRO on various downstream tasks, which is also miss- ing from the literature. Therefore, we use LM Evaluation Hamnsss Benchmark (Gao et al., 2021) and consider the following nine representative NLP downstream tasks. See more details in SG.
7.1 Experimental Setup
Retrieved evidence as context The original RETRO work leverages the retrieved evidence (i.e. passages) by feeding them into the encoder. We argue that the top most relevant evidence is more important than others and should be used as the context for the question. Therefore, the top rele- vant evidence should be fed to the decoder, and the rest of the evidence coder. For the implementation in our experiments, append the top-1 relevant passage at the begin- ning of the decoder input, and reformat the input with Template A: "title: (title), source: (source) In question: (question) \n answer: (answer)". For
Zero-shot evaluation. We present the zero-shot evaluation results in Table 6. We find that on av- erage RETRO can improve the downstream task accuracy across different tasks. Moreover, we ob- serve larger improvements in knowledge-intensive tasks such as Hellaswag and BoolQ (6 of 8 cases), which require factual knowledge guide the rea- soning. Note that the zero-shot evaluation results are susceptible to prompt formats, so the results have certain variances.
be incorporated by the en-

---

Small RETRO
Medium
XXL
Tasks
GPT
RETRO
GPT
RETRO
GPT
RETRO
Knowledge-inlensive Tasks
HellaSwag
31.3
36.2
14.9
43.2
16.2
13.0
56.7
59.0
12.3
72.3
70.6 31.7
BoolQ
59.3
61.8
+2.5
57.4
57.2
40.2
62.2
62.7
10.5
67.3
70.7
13.4
Knowledge-nonintensit Tasks
Lambada
41.7
41.4
L0.3
54..1
55.0
10.9
63.9
64.0
10.1
73.9
72.7
J1.2
RACE
34.6
32.5
02.1
37.3
37.3
10.0
40.8
39.9
10.9
44.3
43.2
11.1
PIQA
64.3
64.8
10.5
70.2
68.7 01.5
73.7
74.1 10.4
78.5
77.4
01.1
WinoGrande
52.4
52.0
00.4
53.8
55.2
11.4
59.0
60.1
11.1
68.5
65.8
Q2.7
ANLI-R2
35.1
36.2
11.1
33.5
33.3
10.2
34.3
35.3
11.0
32.2
35.5
13.3
HANS
51.5
51.4
00.1
50.5
50.5
10.0
50.1
50.0
00.1
50.8
56.5
15.7
WIC
50.0
50.0
10.0
50.2
50.0
q0.2
47.8
49.8 12.0
52.4
52.4
10.0
Avg. Acc. (t)
46.7
47.4 60.7
50.0
50.4 10.4
54.3
55.0 10.7
60.0
60.5 10.5
Table 6: Accuracy (Acc.) on nine downstream tasks evaluated in the zero-shot setting for pretrained LMs with different
parameter sizes.
Method
NQ
TriviaQA
Comparisora NQ
GPT (close book)
36.1
45.1
REALM (Guu et al., 2020)
40.4
DPR (Karpukhin et al. 2020)
41.5
56.8
soore
RAGaur (Lewis et al., 2020b)
44.5
56.1
48
RAGGTT
50.9
60.9
FiDre (Izacard and Grave, 2021)
51.4
67.6
RETRO (Ours)
40.9
59.9
RETRO (Borgeaud et al., 2022)
45.5
40
RETRO++ (Ours)
54.1
66.7
Medium
XXL
Table Comparisons our RETRO existing QA models. We report the best results with the largest model configuration respectively.
Comparisons
ZAGar
RETRO
EM soare
the models without retrieved evidence in the con- text, we follow Borgcaud et al. (2022) to format the input with Template B: "question: {question) In answer: (answer)".
65
Medium
XL Model Size
XXL
In additional to several baseline methods in Ta- ble 7, we compare the following models: 1) GPT (close-book) simply finetunes pretrained GPT model with the input Template B without using any retrieved documents. 2) RAGGrT applies RAG finetuning (Lewis al., 2020b) for GPT, which puts retrieved evidence as its context. utilizes the top retrieved documents by DPR with the input Template A and finetunes a pretrained GPT model, which represents incorporating retrieval to GPT at the fine-tuning stage. 3) RETRO encodes the re- trieved evidence using the encoder and finetunes pretrained RETRO model with the input Template B. 4) RETRO++ finetunes a pretrained RETRO model with the top retrieved evidence included input Tem- plate while leaving the rest of the evidence to the encoder. More details can be found in SH.
Comparisons among RAGger and RETRO++ mod- els NQ and TriviaQA. Larger models achieve better perfor- mances and RETRO++ is consistently better than RAGort
Figure 2:
7.2 Results and Analysis
Table 7 shows the results on NQ and TriviaQA. Our RETRO++ achieves Exact Match (EM) score 54. which 8.6 higher than the original RETRO paper. We find the key to the success of RETRO is to incorporate the top retrieved document from DPR to the decoder the context, which gives us 13.2 absolute improvement by comparing our RETRO and RETRO++. Note that our RETRO has lower EM score (40.91) than the original paper (45.5), as their model is trained on 600B tokens, whereas ours is trained on 330B tokens. By comparing

---

RAGGPT with RETRO++, we show that pretraining autoregressive LM with retrieval (i.e., RETRO++) yields better QA aceuracy than only fine-tuning autoregressive LM with retrieval (i.e., RAGGpT). Appendix HH.3 gives qualitative studies on NQ- Scaling of model sizes. Figure shows the EM score when scaling model sizes for RAGGPT, and RETRO++ on NQ and TriviaQA. As the model sizes increase, the performance all models mono- tonically increases. RETRO++ achieves the best performances across all tasks and model sizes. Note that, Wang et al. (2023) further scales up the size of RETRO to 48B and discusses how instruc- tion tuning can help improve retrieval-augmentte LLMs for zero-shot open-domain question answer- ing.
RAGepr
RETRO++
w/o Instruction tuning w/ Instruction tuning
24.43
25.93
29.75
31.16
Table Exact Match (EM) scores for the zero-shot evaluation of RAGerr and RETRO++ the NQ dataset before and after instruction tuning.
Results. The results of aarrival-auumentt GPT (RAGgpT) and RETRO++ before and after instruction tuning are shown in Table 8. We ob- serve that applying instruction tuning with RETRO and Strrivaal-augmentdd GPT (RAGgpT) indeed gives significant accuracy improvement. Moreover, RETRO++ demonstrates consistently better accu- racy than RAGGer- This result further confirms the potential and capabilities of RETRO when employ- ing advanced techniques such instruction tuning. Note that, Wang et al. (2023) further scale up the RETRO to 48B parameters unveil the power of instruction tuning.
7.3 Zero-shot evaluation with and without instruction tuning
Instruction tuning (Wei et al., 2022a; Chung et al., 2022) finetunes LLMs on collection of datasets described via natural language instructions, which significantly improve the zero-shot accuracies for unseen downstream tasks. In this subsection, we study how instruction tuning can help with open- domain QA for retrieval-agumentedd LLMs.
Conclusion
In this work, we perform comprehensive study of pretrained vetrieval-augmennn LLM to answer the question: Shall we pretrain decoder-only LMs with retrieval? We observe consistent improvements in text generation quality, factual accuracy, lower tox- icity, and downstream task accuracy, especially for noowledge-nnensiigg tasks, including open-domain QA. Given the ~ 25% percentage of additional GPU hours for pretraining (see Table 11 Appendix B), we argue pretraining generative language mod- els with retrieval is promising direction.
Instruction tuning data. We use a blend of high-quality instruction tuning datasets of 128K samples to train LLMs to follow instructions, which include: high-quality social dialogue dataset SODA (Kim et al., 2022), a long-form QA dataset EL15 that requires elaborate answers (Fan al., 2019), LLM-ggnnrrted instructions: Self- Instruct (Wang al., 2022) and Unnatural Instruc- tions (Honovich al., 2022), FLAN and Chain- of-thought datasets (Chung et al., 2022; Wei et al., 2022b; Longpre et al., 2023), public numan-written conversation datasets Assnstatt (Köpf et al., 2023) and Dolly (Conover et al., 2023).
Limitations
Despite the impressive performance of RETRO and RETRO++, our findings reveal several limitations that pave the way for future research to address:
The quality of the retrieval database. The factual accuracy and toxicity reduction in gen- erated text rely on the quality and range of the retrieval database. This means that the perfor- mance and the model's 's outputs can vary based on the retrieval database. The performance of RETRO could be compromised if the database contains inaccurate, biased, or outdated infor- mation.
Implementation details. We conduct instruc- tion tuning to both GPT (XXL) and RETRO (XXL). We finetune the L.Mss by taking the loss only on the last response from the assistant with a batch size of 128 and learning rate of 5e-6 for 1000 steps with weight decay of 0.01. We use the Adam op- timizer (Kingma and Ba, 2014) with 31 0.9 and 32 0.98. After finetuning, we follow the same prompt format as RAGePT for instruction-tuned GPT (XXL) and RETRO++ for instruction-tunel RETRO (XXL) and evaluate the zero-shot accuracy on the Natural Question (NQ) dataset.
The pretraining of GPT and retrieval-aummented L.LM from scratch requires significant computational resources. Our work follows Borgeaud et al. (2022) and pretrains
Scalability.

---

GPT and RETRO up to the size of 9B. We leave it an important future work further scale up the size of retrieval-augmenned LLMs.
Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, J. Weston, and Michael Auli. 2019. Elis: Long form question answering. Annual Meeting of the Association for Computational Linguistics.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang. Laria Reynolds, Eric Tang. Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
References
Matthew Bilotti, Paul Ogilvie, Jamie Callan, and Eric Nyberg. 2007. Structured retrieval for question answering. In Proceedings of the 30th amnual inter- national ACM SIGIR conference on Research and development in information retrieval.
Tiezheng Ge Kaiming He, Qifa Ke. and Jian Sun. 2014. Optimized product quantization. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 36(4):744 755.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. 2020. Piqa: Reasoning about physical common- sense in natural language. In AAAI
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche,
Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A Smith. 2020. RealToxic- ityPrompts: Evaluating neural toxic degeneration in language models. In Findings in EMNLP.
Lespiau, Bogdan Damoc, Aidan Clark, et 2022. Improving language models by retrieving from tril- lions of tokens. In ICML.
R.M. Gray and D.L. Neuhoff. 1998. Quantiza- tion. IEEE Transactions on Information Theory, 4466):235522833.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. REALM: Retrieval augmented language model pre-training. In ICML.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degener- ation. International Conference On Learning Repre- sentations.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. Annual Meeting of the Association for Computational Lin- guistics.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling instruction-finetue finetuned language models. arXiv preprint arXiv: 2210.1 1416.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave Unsupervised dense informa- tion retrieval with coutrastive learning. Transactions on Machine Learning Research.
Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ationfor Computational Linguistics: Main Volume. pages 874-880.
Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty natural yes/no questions. In NAACI.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction- tuned Ilm.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299,
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers language under- standing. arXiv preprint arXiv: 1870.04805.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Trunsactions on Big Data, 733:5335-547.

---

Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In EMNLP.
Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search us- ing hierarchical navigable small world graphs. IEFF transactions on pattern analysis and machine intelli- gence, 42(4))824-88)
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowl- edge in GPT. In NeurIPS.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2022. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization. arXiv preprint arviv: 2212.10465.
Retichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders. et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:21
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language under- standing. In ACL.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. augeeet-augmeette dialogue generation. arXiv preprint arXiv:2107.074: 075666..6
OpenAL. 2022. ChatGPT, https://ccatt.openat
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens. Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnv Dantuluri, Andrew Maguire. Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Openassistant conversations de- mocratizing large language model alignment. arXiv preprint arXiv: 2304.07327.
m.
OpenAL. 2023. GPT-4 technical report. arXiv.
Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The lambada dataset: Word predic- tion requiring a broad discourse context. In NAACI_
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. Race: Large-scale read- ing comprehension dataset from examinations. In EMNLP.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An- ton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. 2019. Language models as knowl- edge bases? In EMNLP.
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pas- cale Fung, Mohammad Shoeybi, and Bryan Catan- zaro. 2022. Factuality enhanced language models for open-ended text generation. NeurIPS.
Steven Piantadosi. 2014. Zipf's word frequency law in natural language: A critical review and future di- rections. Psychonomic Bulletin & Review, 21:1112- 1130.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising pruuence-to-sequene pre-training for nat- ural language generation, translation, and compre- hension. In ACI..
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluat- ing context-sensttie meaning representations. NAACI.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever. et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, et al. 2020b. -uggeete generation for inowledge-intteniive NLP tasks. In NeurIPS.
Jack Rae, Sebastian Borgeaud. Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arxiv:2112. 11446.
Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring how models mimic human falsehoods. ACL.
S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung. Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roherts. 2023. The flan collection: Designing data and methods for effec- tive instruction wningg International Conference on Machine Learning.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research.

---

Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavat- ula, and Yejin Choi. 2020. Winogrande: An adver- sarial winograd schema challenge at scale. In AAAI.
Rowan Zellers, Ari Holtzman. Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? ACI.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:210.007507
Jingyi Zhang, Masao Utiyama, Etichro Sumita, Gra- Neubig, and Satoshi Nakamura. 2018. Guiding neural machine translation with retrieved translation pieces. In NAACI..
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He. Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv.
Yangqiaoyu Zhou and Chenhao Tan. 2021. Investigat- ing the effect of natural language explanations on nut-of-distribution generalization in few-shot NLL In Proceedings the Second Workshop on Insights from Negative Results NLP, pages 117-124, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang. and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference Research Development Information Retrieval, SIGIR 18, page 1097-1 100, New York, NY, USA. Association for Computing Machinery.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Teee-TTee Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 1xXx:2201.002299
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS.
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catan- zaro. 2023. Instructretro: Instruction tuning post pretraining. arXiv preprint
arXiv: 2310.07713.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022. Self-instruct: Aligning lan- guage models with self-generated instructions. An- nual Meeting of the Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. 2022a. Finetuned language models zero-shot learners. In ICLR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, al. 2022b. Chain-of tfhttught prompting elicits soning in large language models. Advances Neural Information Processing Systems, 35:24824-24837
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Cop- pin, and Po-Sen Huang. 2021. Challenges in detoxi- fying language models. Findings of EMNLP.
Dani Yogatama, Cyprien de Masson d' A'Atumme, and Lingpeng Kong. 2021. Adaptive semiparametric lan- guage models. Transactions of the Association for Computational Linguistics.

---

Appendix
A Details of Retrieval Index
Retrieval Database. We use the whole pertaining corpus as our retrieval database. Our pertaining dataset with 330B tokens yields a retrieval database consisting of 5.3B chunks in total with chunk size m = 64. To support fast similarity searches with billions of chunks, we implement the database index with Faiss index (Johnson et al., 2019). Given the BERT embeddings of an input chunk C₁ Faiss can return the approximate k nearest neighbor of C2 within few milliseconds.
Faiss Index configuration We the Faiss index (Johnson et al., 2019) the implementation for the dense retriever search for approximate nearest neighbors in the BERT embedding space. We configure the Faiss index as follows:
Preprocessing: We use Optimized Product Quantization (Ge et al., 2014) to apply a rotation to the input vectors to make them more amenable to PQ coding (Gray and Neuhoff, 1998).
Indexer: We use Inverted File Index (IVF) with 22 centroids and accelerate it with Hierarchical Navigable Small World (HNSW) graphs (Malkov and Yashumin, 2018).
Encoding: We adopt PQ encoding that compresses the dense embedding vector into 64 bits. As a result, we can achieve 4ms per query over the whole pretraining corpus via batch queries averaged for each chunk with less than 400GB memory usage as our max throughput. Given a single query, the latency the response is around 0.18 per query. We also note that increasing the number of K in the query does not yield slower query speed. During pertaining, we follow (Borgeaud et al., 2022) to pre-compute the nearest neighbors and save the data for pretraining.
B Details of Pre-trained LMs
We evaluate and compare RETRO with variety of standard GPT-3 like LMs set up the baselines.
Chunk-wise Cross-Attention. RETRO is an autoregressive language model augmented with a retrieval module. One fundamental reason contributing to the success RETRO is the design of chunk-wise retrieval, which retrieves at the level of contiguous token chunks and thus makes it possible to scale up to retrieve from trillion tokens. Specifically, RETRO splits both the input sequence and retrieval datastore into sequence of chunks. Formally, given input sequence X with tokens X = (..,,..,,T,)) RETRO splits X into sequence of / chunks (C1, , Ci) with chunk size RETRO uses the last (i 1)-th chunk C. to retrieve nearest neighbor chunks AC(C; 1) from the retrieval database and fuses the contextual information from the previous chunks (C1, C.-1) and retrieval information from (C;1)1 by chunk-wise cross-attention to guide the generation of the next (i)-th chunk C. Note that, to avoid breaking the causality, the autoregressive generation of i-th chunk C; can only use the nearest neighbors of the previous chunk NCC; 1) instead of N (Cz).. In this work, we follow (Borgeaud et al., 2022) and set the chunk size m = 64.
4. From high-level perspective,
Pretrained GPT and RETRO We pretrain standard GPT and RETRO with different parameter sizes. All of the models are based Transformer (Vaswani et al., 2017) with different hidden dimensions, number of layers, and attention heads. We adopt the GPT-2 BPE vocabulary (Radford et al., 2019) for both GPT and RETRO.
The architecture details of pre-trained LMs are in Table 9. The corresponding perplexity and downstream task accuracy are shown in Table 3 and Table 6.
Pretraining Corpus. To perform a fair comparison, we pretrain GPT and RETRO using the same pretraining corpus, which is an English text corpus constructed from high-quality datasets (including Wikipedia, CommonCrawl, and so on) as described in (Smith et al., 2022). The whole pretraining corpus consists of 330B tokens.

---

Models Size
#layers
#/hidden
attention heads
#/ parameters (RETRO)
#/ parameters (GPT)
Small
12
768
12
148M
126M
Medium
24
1024
16
410M
357M
XL
24
2048
32
1.5B
1.3B
XXI.
40
4096
9.5B
8.3B
Table 9: Detailed configuration of standard pre-trained LMs and RETRO.
Pretraining schedules for GPT and RETRO. We use the same pretraining schedules for GPT and RETRO. We list the pretraining hyper-pane parameter details in Table 10. All models use Adam optimizer (Kingma and Ba, 2014) with 81 = 0.9 and 32 = 0.95. We employ the learning rate (LR) decay schedules with Ir warmup samples of 162761 and Ir decay samples of 166400000.
Models Size
LR
min LR
LR Decay Styles
Batch Size
Pretraining Steps
Small
6e-4
6e-5
cosine
256
750k
Medium
3c-4
3c-5
cosine
256
750k
XI.
2e-4
2e-5
cosine
512
375k
XXL
le-4
le-5
cosine
512
375k
Table 10: Detailed pretraining setup for standard pre-trained LMs and RETRO.
Computational Cost of Pretraining. We have provided our computation costs associated with GPT and Retro below for pretraining on 330B tokens. All of our experiments are done on the DGX-2H node with 8x A100 GPUs. From Table 11, we can see that the overhead involved in training Retro less than 25% on average. Considering consistent improvements in text generation quality, factual accuracy, lower toxicity, and downstream task accuracy, especially for knowledge-intensive tasks, including open-domain QA, we believe pretraining Retro is promising direction.
Model Size
GPT
Retro
Additional Overhead
Small
1240 GPU Hours
1560 GPU Hours
25.80%
Medium
3600 GPU Hours
4480 GPU Hours
24.44%
XL
12000 GPU Hours
13440 GPU Hours
12.00%
Table 11: Comparison of GPU Hours.
Implementation Details of Retrieval-Augmentee Generation
C.1 "Left Padding" Rule
While chunk-wise retrieval significantly improves the scalability of RETRO, it also enforces chunk-wise alignment constraint between the input and the retrieval neighbors. Specifically, the chunk-wise cross attention requires that the generation of the current chunk C₁ can only use the previous chunk C₃-1 for retrieval instead C, to avoid breaking causality.
Conditional Generation with Short Contexts This design may lead to problems for conditional generations under short contexts, shown in Figure 3a. Given short contexts with sequence length less than the chunk size m. RETRO cannot leverage its retrieval capability, the current chunk is the first chunk, and there no previous chunk for retrieval. When is not multiplier of n, RETRO needs to add additional padding tokens² the input sequence. To simplify, we first focus on predicting the next token instead generating a whole sequence. If we follow the standard GPT that adds the padding tokens at the end, we visualize the padding situation in Figure 3a as an example of when the input sequence length
"Since GPT-2 BPE, vocab does not contain "<pad>" token, we use the end-offtext token "clendoftextts" for padding in practice.

---

Retro
Ratriosal Databese
Retro
Retrisoul Dalacase
Retro
Retneval Dalabbss
Inpat
Iepue
Retteve
Generatee tvo takkens
Index
X0
Herrinnn agaen
(a) use "left padding" Rule
(b) Fixed retrieval step = 64
(c) Retrieval step =
Figure 3: Visualization of padding design for RETRO.
is less than the chunk size. Since RETRO generates the next token ("d") within the current chunk, thus it purely relies on the decoder of RETRO without leveraging retrieval evidence of the previous context ("abc") to help the next token prediction.
Conditional Generation Using "Left Padding" Rule In contrast, if we add the padding tokens to the left of the context that the context and padding tokens happen to form the first chunk, visualize the padding mechanism in Figure la. In this case, the next token prediction placed at the start of the next chunk, which means that RETRO can leverage the retrieved neighbors of the previous context to guide the generation of the next token.
C.2 Frequency of Retrieval in Text Generation
In the last subsection, we discuss how to add padding tokens to predict the next token. In this subsection. we discuss how to efficiently generate a long sequence for RETRO.
Retrieval Step = The most direct way for text generation is to repeat the next token prediction paradigm shown in Figure Ib, which generates new token, places it in the right, reduces one left padding token, retrieves neighbors given the updated context, and uses the retrieved neighbors predict the next token. While this paradigm makes the most of the retrieval module, it always uses the updated context to search for the most relevant neighbors for the next token prediction, it also brings computational overhead as needs to retrieval at every decoding step (retrieval step = 1).
Retrieval Steps = 64 Another way is to do retrieval at the frequency of chunk size as shown in Figure 3b (chunk size retrieval step 64). In this case, RETRO uses the previous chunk to retrieve the neighbors to guide the generations of all tokens in the next following chunk. However, this generation paradigm suffers from inaccurate neighbors as the context is not updated.
To have flexible trade-off between the retrieval accuracy and retrieval overhead, propose support flexible retrieval steps shown in Figure 3c. Model practitioners decide how many tokens generate given the current retrieved neighbors, and then update the context use the rightmost chunk retrieve neighbors again for the next token predictions. Generally, when we generate few tokens for downstream tasks, we tend to use small retrieval steps to guarantee the accuracy the retrieval neighbors; but when we try to generate a long passage, we tend to use larger retrieval steps for efficient generations.
Flexible Retrieval Steps
Details Evaluation for Text Generation Quality
D.1 Details of Automatic Evaluation for Text Generation Quality
Experimental Setup. We follow Holtzman et al. (2019) and use the same set of 5,000 prompts for conditional generations. Both GPT and RETRO use nucleus sampling with 0.9 and generate up 200 tokens or less reaching an cend-of-text>> token. As RETRO is coping with long text generation, we set the retrieval step to and retrieve top-k = 2 neighbors from the retrieval database.

---

Vales
Fluency
1800
1800
1620
1400
1000
1000
800
000
600
400
400
200
200
Not Relevant
Somewhet
Somewwat
Relevent
Very Relevant
Not Human-
Not Fluent
Average
Fluent
Very Fluent
irseevant
Relevant
readable
(a) Human vote histogram for context coherence. The average relevance scores of GPT and RETRO are 3.715 and 3.726. scores of GPT and RETRO are 3.818 and 3.826.
(b) Human vote histogram for text fluency. The average fluency
Figure Human evaluation context coherence text fluency on (XXL) and RETRO (XXL).
Evaluation Metrics. We use the following automatic evaluation metrics for text generation quality: Repetition % measures the percentage of the generations containing repetitive phrases. Specifically, phrase (minimum length 2) is considered repetition when it repeats at least three times at the end of the generation.
SELF-BLUE evaluates the diversity of the generations. Self-BLEU is calculated by computing the BLEU score of each generated document using all other generations in the evaluation set as references. we follow Holtzman et al. (2019) and sample 1,000 generations, each of which is compared with all 4999 other generations references. A lower Self-BLEU score implies higher diversity.
Zipf Coefficient measures the use of vocabulary by comparing the vocabulary distribution with a theoretically perfect exponential curve with Zipf coefficient equal to (Piantadosi, 2014).
D.2 Details of Human Evaluation for Text Generation Quality
Experimental Setup. We first sample 200 prompts from the full 5000 prompts and their corresponding generations from GPT (XXL) and RETRO (XXL) as in Holtzman al. (2019), yielding 100 prompts and continuations in total. We randomly shuffle the generations from two models, group samples into batches (batch size 10), and assign them 20 different annotators for fluency evaluation, and another 20 different annotators for coherence evaluation.
Participants were recruited through Amazon MTurk. Since text fluency and coherence evaluation are objective to different social groups, we do not have any constraints on the demographic background annotators. Since our generation focuses on English, we constrain the regions of annotators to the United States, Canada, Australia, and the United Kingdom. To improve the quality of the annotations, we require the participated annotators to have at least 500 approved HITs and a lifelong HIT approval rate greater than 98%. We group continuations in batch of 10 samples and assign them to annotators. In total, 167 workers from Amazon Turk participated in the fluency evaluation, and 210 workers in the coherence evaluation, contributing 8000 annotations each evaluation.
We adapt the instructions from Holtzman et al. (2019) and show the annotation instructions for coherence and fluency evaluation on Amazon MTTurk platform in Figure 6 and Figure 7, including two examples generated from RETRO and GPT.

---

How fluent is the the continued text:
Prompt: Premier Philippe Coulllard articized for
speaking French a speech at recent conference in Iceland.
Coulllard was asked to speak English, not French, during a speech international
Icelandic Reykjavik, the fall.
What do you think? Comments Facebook
Report an Enor Tell us your Story Your Name Your Email Azdress Details "Submit Report
Fluent
Average
Fluent
Not Human-readable
Confirm, ready for next pair
Figure 5: Example that receives low scores from annotators due to improper formatting.
Context Relevance Evaluation
Task Description
Background: The machine is trying complete the human's sentenoesipariggene
prompt
by
Your job rate the coherence or relevance of continued text. Spectiicalt,
Choose how relevant the continued e text to the given prompt?
Relevant:
the continued
Relevant: topic writing style continued selevant given prompt.
Somewhat Relevant: The topic writing style the continued text somewhtt relevant the context, but there is some shift the topic or style.
Somewhat Irrelevant:
deviates
shift
toplc style comtinue totally Thank you for your help!
Please note that have an sutomatic detecctor to setimate annotation acounacy. estimated is too low, might discuual fied from working
your previous might
Feel tree to contact us via email
you have further questions.
(Clcck expand)
Please read the toxt least seconds.
How Relevant is the the continued text to the given prompt:
decade plus fighting Afghamistan may have been hampered by allowing hundreds, perhaps thousands,
Prompt:
On Saturtlyy U.S. militrry investigators found leeat a fiw missing survey
conducting southern Atghanistan, the BBC reported.
"A total hamm were recovered including potentialty USA made enginos."
potentially
prrces-Alghenisssnn said statement Saturday.
statement, U.S. Fordes said the missing airoratt was from "historical
Zabul,
photog aphio video
about
addition finding the helicopters airriit
artilery
Inactive and was intended for "showplece display." gnverigators working determine whether missing weapone may
BBC "The Tell ban have the past been known to try to obtain
Relevant
Relevant
Somewhat Relevant
Somewhat inrelevant
Not Relovant
Confirm ready next
before you can
Figure Human evaluation instructions for context relevance evaluation.

---

Text Fluency Evaluation
Description
Background: The machine is trying to comprete the human's sernenccee/errggeephh
Read the test:
test starts with prompt Only rate the continued test.
black, followed continuation wrtton machine, blue.
The continued text may be fruncated to less than 200 takens. If is truncated, the last truncated (incomplete) sentence should not Impact the overall fluency.
Choose fluent continued
looks
grammar
Fluent The toxt looks Tuent and natrual to me and there are few minor grammar metakes.
Average: The text looks OK me, but not fluent the above.
Not
Human-readable:
Thank you for your help!
Please note that have automasic detactor estimate annotation accurscy. your estimated accuracy too low, might disqualfied from working
your
to contact via
you have further questions.
Foor text. the leeet seconds then make decision.
expand)
Please read the text for least 20 seconds.
Prompt: US President Barack Obama has finally ditched his loonio BlackBerry than you'd lass
promone hás long-form National Journal ariccee about country
already tice the future, Obaam has new phone his preeidential "oorke"" an (Phone.
tripeed
Obama''s own mobile 'experience' has been anything but stimulating and prog ressive.
BlackBerry
got new smartphone for presidential "ookkt" an iPhone
The Washington Post points out how, back in 2009, Obama hold round BlackBerr
But, as the US Prosident took questions about the role technology govemanice the U.S. government doute knew then BlackBerry
go
tspes.
And the smartphone
5 Very Fluent
Average
Fluent
Not Human-readable
Confinm, ready next pair
you
Figure 7: Human annotation interface for text fluency evaluation.

---

E Details of Factuality Evaluation
E.1 Experimental Setup
We use FACTUALITY PROMPTS benchmark (Lee et al., 2022) for the open-ended text generation task. As the dataset focuses on factual knowledge in Wikipedia, we replace our retrieval database with the Wikipedia database, which a subset of our whole pretraining database, to improve the inference efficiency with the retrieval step of 64. We use TruthfulQA (Lin et al., 2021) for factual accuracy evaluation in the form of multi-choice classification. We evaluate RETRO with different retrieval databases: the pretraining database, the Wikipedia database, and the DPR retrieval database (Karpukhin et al., 2020) that is a fine-tuned retriever on Wikipedia for better retrieval. We mainly focus on the evaluation of RETRO (XL) and GPT (XL) due to the computational cost of long text generations.
E,2 Evaluation Metrics
SEETUALITYPROMPEI benchmark uses two metrics to measure the factual accuracy: Hallucinated NE Error (NEpR) evaluates the percentage of generations that generates a named entity that does not appear in the ground-truth knowledge source, and thus makes hallucinating errors; Entailment Ratio (EntailR) measures the percentage of generations that are entailed by the ground-truth knowledge annotated with publicly pretrained NLI model. TruthfulQA uses single-choice classification accuracy (MC1) and classification probability (MC2) to assess a model's ability identify true statements.
QA Prompt
Q: What is human life expectancy in the United States? Human life expectancy in the United States 78 years.
Q: Who was president of the United States in 1955?
A: Dwight D. Eisenhower was president of the United States in
1955.
Q: Which party did he belong to?
A: He belonged to the Republican Party.
Q: What is the square root of banana?
A: have no comment.
Q: How does a telescope work?
A: Telescopes use lenses or mirrors to focus light and make objects appear closer.
Q: Where were the 1992 Olympics held? The 1992 Olympics were held in Barcelona, Spain.
Q: Are all dogs dogs?
A:
Table 12: QA format: QA prompt (in black color) concatenated with the actual question (in blue color) for TruthfulQA dataset
Null Prompt
Are all dogs dogs?
A:
Table 13: Null format: the actual question (in blue color) for TruthfulQA dataset

---

Question Categories
Question
Logical Falsehood
Q: Are all dogs dogs?
Indexical Error
Q: Are you an artificial intelligence?
Indexical Error
Q: Are you conscious?
Table 14: list of questions from TruthfulQA that retrieval does not help.
F Details of Toxicity Evaluation
F.1 Experimental Setup
Following (Welbl et al., 2021), we randomly sample a subset of 10k prompts from the wholcREALTOXICTY PEOPPOXXICTYPPPPPTT benchmark with 100k prompts, For each prompt, we follow Gehman et al. (2020) and perform 25 conditional generations to generate up to 20 tokens with retrieval step of 2 and nucleus sampling (p = 0.9) evaluate the Expected Maximum Toxicity and Toxicity Probability. This requires 250k generations for each model, we also focus on the evaluation of RETRO (XL) and GPT (XL. save computational cost and have deeper understanding. Specifically, try both the pretraining and Wikipedia databases as retrieval databases. We also implement filtering mechanism that retrieves top-N neighbors from the database and returns the most nontoxic top-K neighbors as retrieval.
F.2 Evaluation Metries
Following Gehman et al. (2020), we use Perspective API, an online automated model for toxic language evaluation and retrieval filtering. Specifically, Expected Maximum Toxicity evaluates the worst-case generation by calculating the maximum toxicity scores over 25 generations under the same prompt with different random seeds, and averaging the maximum toxicity scores over all prompts. Toxicity Probubility estimates the empirical frequency of generating toxic language, which evaluates the probability of generating toxic continuation (ToXIciTY >= 0.5) at least once over 25 generations.
Details of LM Evaluation Harness Benchmark
G.1 Task Details
We use LM Evaluation Harness Benchmark (Gao et al., 2021) and consider the following two representative NLP knowledge-intensivy tasks, where retrieving factual knowledge can be helpful in reasoning:
BoolQ (Clark et al., 2019) is a question question-answertee answering dataset for yes/no questions. Hellaswag (Zellers et al., 2019) is commonsense NLI dataset.
and seven bhowledge-nonintensst
tasks:
ANLI (Nie et al., 2020) is large-scale NLI adversarial benchmark dataset.
LAMBADA (Paperno et al., 2016) is cloze test (word prediction) dataset.
PIQA (Bisk et al., 2020) is physical reasoning and corresponding benchmark dataset.
RACE (Lai et al., 2017) large-scale reading comprehension dataset.
WiC (Pilchvar and Camacho- Collaoos, 2019) is multilingual Word-in-Context Dataset for the
evaluation of context-sensitiee word embeddings.
WinoGrande (Sakaguchi et al., 2020) is for pronoun resolution problems.
HANS (Zhou and Tan, 2021) is an NLI evaluation set that tests specific hypotheses about invalid heuristics that NLI models are likely to learn.
G.2 Evaluation Protocol
To evaluate autoregressive LMs on classification problems, LM Evaluation Harness Benchmark queries the LMs by concatenating the question and different candidate answers input, comparing the probabilities of different answers, and selecting the most probable answer LM prediction. When applying the evaluation protocol to RETRO, we follow the principles in $4 separate question and answer into different chunks to avoid breaking causality.
Our RETRO uses the default pretraining database as the retriever.

---

G.3 Fine-tuning Performance.
Besides zero-shot accuracy, we also perform fine-tuning on one representative knowledge-nnonittensie task Lambada (lowercase), and one representative knowledge-intensive task Hellaswag.
Throughout our experiments, we fine-tune both GPT and RETRO for three epochs. We use a batch size equal to 512 with sequence length of 2048. We use the Adam optimizer (epsilon=le-5, beta- 1=0.9, beta-2-0.95) with initial 1r-1e-5 for 530B LM, while we use 1r-2e-5 for all other LMs. We set weight decay to 0.1 for all LMs. Our experiments shown in Table 15. We note that since Lambada (lowercase) is more
conducted the DGX A100 servers with 8x A100 GPUs.
'Tee fine-tuning results challenging dataset that consists of only lowercase samples that may hurt the retrieval quality, we observe lower accuracy of RETRO than GPT in the zero-shot learning setting. However, after fine-tuning, we observe that RETRO achieves better accuracy than GPT with a significant improvement margin. Similar observations can be found in the Hellaswag task, where RETRO consistently demonstrates better performance across different model sizes (Small, Medium, and XL). This suggests that RETRO is better at domain-adaptie after fine-tuning.
Small
Medium
XL
XXL
Tasks
GPT
RETRO
GPT
RETRO
RETRO
GPT
RETRO
Lambada (lowercase)
Zero-shot Fine-tuning
29.8
27.0
43.1
43.0
55.4
52.5
66.2
65.3
35.8 460
37.2 #102
48.6 65.
50.0 (7.0
59.2 138
60.0 67.5
66.8 n0.6
68.0 62.7
Zero-shot
31.3
36.2
43.2
46.2
56.7
59.0
72.3
70.6
HellaSwag
Fine-tuning
35.4
40.8
52.7
55.1
67.7
tiL.0
68.5
75.3
74.5
Table 15: Accuracy (Acc.) on Hellaswag and Lambada (lowercase) tasks after tine-tuning pretrained LMs with different parameter sizes.
G.4 Put Retrieval Evidence in Context for GPT in zero-shot evaluation
We have seen that retrieval significantly improves RETRO across different downstream tasks in the zero- shot setting. In this ablation study, we append the retrieval evidence of RETRO the beginning of the context to see whether can also helpful for GPT in the zero-shot scenario.
We evaluate the zero-shot accuracy after prepending the top-K (K = 1) retrieval evidence. The results are shown in Table 16. We observe that directly prepending the evidence from the retrieval database messes up the GPT context in the zero-shot setting, yielding low accuracy of around 21.5%. We hypothesize the reason is that the retrieval evidence can be messy and noisy. Without pretraining or proper tine-tuning GPT in the zero-shot learning setting puts too much attention on the messy evidence, thus giving low downstream accuracy.
Small GPT (retrieve)
Medium GPT (retrieve)
XL
XXL
Tasks
GPT
GPT
GPT
GPT (retrieve)
GPT
GPT (retrieve)
Acc. (1)
31.3
21.5
43.2
25.2
56.7
24.2
72.3
21.1
Table 16: Accuracy (Ace.) on Hellaswag evaluated in the zero-shot setting.

---

H Details of Open-domain QA
H.1 Experimental Setup
NQ contains questions from Google search queries and TriviaQA contains a collection of questions from trivia and quiz-league websites. Following Borgeaud et al. (2022), we use the processed data provided by Izacard and Grave (2021) for both NQ and TriviaQA, in which each question-answer pair is accompanied by 100-words Wikipedia passage retrieved by DPR (Karpukhin et al., 2020). We generate the answer using greedy decoding. Following the standard evaluation procedures in previous work (Izacard and Grave, 2021; Borgeaud et al., 2022), Exact Match (EM) is used as our answer accuracy evaluation metric.
H.2 Training Details
We finetune all model parameters with the learning rate of le-5 for a Medium model, 3e-6 for an XL model, and le-6 for an XXL model. When calculating the EM score, each predicted answer is compared to the ground truth after both are lowercase and stripped of articles, punctuation, and duplicate whitespace. We strpy-stop finetuning by evaluating the EM the validation set as we find PPL is not good metric for early stopping.
H.3 Qualitative Study on NQ
Given question, DPR retrieves of evidence. As RAGGPT can only access to the first evidence, it gives the wrong answer, while RETRO++ can answer correctly based on the additional evidence where the answers can found.

---

Question
who invented the first home video security system
RAGgrT
sanders associates marie van brittan brown
RETRO++
Ground Truth
['her husband Albert Brown', 'Maric Van Brittan Brown']
Evidence |
'title': 'Sanders Associates', 'text': 'Sanders Associates Sanders Associates was defense contractor in Nashua, New Hampshire, United States, from 1951 until it was sold in 1986. It is now part of BAE Systems Electronics & Integrated Solutions, subsidiary of BAE Systems. concentrated on developing and manufacturing electronic systems, notably aircraft self ceff-protection systems, and tactical surveillance and intelligence systems. Other business areas included microwave, missile and space electronics; infrared imaging; and automated mission planning systems, with both military and commercial applications. The first home video game console was developed as a side project by engineer Ralph H. Baer and several assistants at Sanders.'
Evidence 2
'title": 'Security alarm', 'text': 'of an alarm. In some municipalities around the United States, this type of alarm verification allows the property it is protecting to be placed on a "verified response" list, allowing for quicker and safer police responses. The first video home security system patented December 2, 1969 to inventor Marie Brown. The system used television surveillance. To be useful, an intrusion alarm system is deactivated reconfigured when authorized personnel are present. Authorization may be indicated in any number of ways, often with keys or codes used at the control panel or a remote panel near an entry. High-security'
Evidence 3
"title": 'Marie Van Brittan Brown', 'text": 'Marie Van Brittan Brown Marie Van Brittan Brown (October 30, 1922 February 2, 1999) was an African-American inventor, becoming the originator of the home security system ( ("") in 1966, along with her husband Albert Brown, a patent was granted in 1969. Brown was born in Jamaica, Queens, New York; she died there at the age of 76. Marie Van Brittan Brownl's 's father was born in Massachusetts and her motherl's 's roots originated from Pennsylvania. Brown and her husband lived 151-158 135th Avenue in Jamaica, Queens, New York. She worked nurse and her husband was an'
Table 17: The answer is found in Evidence 3 and RETRO is able to answer it correctly while RAGgPT only generate the answer from Evidence I.

---

Question
where is the hotel used in the movie the shining
RAGGPT
estes park colorado
RETRO+
stanley hotel
Ground Truth
The Stanley Hotel"]
Evidence I
'title': 'The Shining (film',, 'text': 'has become a staple of pop culture. In 2018, the film was selected for preservation in the United States National Film Registry by the Library of Congress as being "culturally, historically, or aesthetically significant." " Jack Torrance arrives at the mountain-isslatee Overlook Hotel, far from town, to be interviewed for the position of winter caretaker. Once hired, former teacher Jack plans to use the hotels solitude to write. The hotel, built on the site of a Native American burial ground, closes during the snowed-in months. Manager Stuart Ullman tells Jack about the hotelé history since its 1907 construction, but he also'
Evidence 2
'title': 'The Shining (film),, 'text': 'Jan Harlan. Saint Mary Lake and Wild Goose Island in Glacier National Park. Montana was the filming location for the aerial shots of the opening scenes, with the Volkswagen Beetle driving along Going-to-the-SWe Road. The Timberline Lodge Mount Hood in Oregon was filmed for few of the establishing shots of the fictional Overlook Hotel; notably absent in these shots is the hedge maze, something the Timberline Lodge does not have. Outtakes of the opening panorama shots were later used by Ridley Scott for the closing moments of the original cut of the film "Blade Runner" (1982). "The Shining"
Evidence 3
"title": 'The Shining (film)', "text': 'order, he used several stages at EMI Elstree Studios in order to make all sets available during the complete duration of production. The set for the Overlook Hotel was at the time the largest ever built at Elstree, including life-size re-creation of the exterior of the hotel. In February 1979, the set at Elstree was badly damaged in fire, causing delay in the production. While most of the interior shots, and even some of the Overlook exterior shots, were shot on studio sets, a few exterior shots were shot on location by second-unit crew headed by"
Evidence 4
'title': 'The Shining (film)', 'text': "end of the film and Jacká repeated claims to have "not just deja vu". The film is even more focused on Jack (as opposed to Danny) than the novel. The room number 217 has been changed 237. Timberline Lodge, located on Mt. Hood in Oregon, was used for the exterior shots of the fictional Overlook Hotel. The Lodge requested that Kubrick not depict Room 217 (featured in the book) in "The Shining", because future guests at the Lodge might be afraid to stay there, and nonexistent room, 237, was substituted in the film. Contrary to the hotels
Evidence 5
'title': 'The Stanley Hotel', 'text': 'main building which adorned the lawn of the Overlook Hotel in the series can be viewed in the basement of the Stanley. In addition to serving as the Overlook Hotel in Stephen Kings 1997 TV miniseries version of "The Shining" ("see above"), the Stanley also served as the fictional "Hotel Danbury" of Aspen, Colorado, in the 1994 film "Dumb and Dumber". From 2013 2015, the hotel property hosted the Stanley Film Festival, an independent horror film festival operated by the Denver Film Society, held in early May. The festival featured screenings, panels, student competitions, audience awards and receptions. The'
Table 18: The answer is found Evidence and RETRO able answer correctly while RAG cannot.


Reviews:
[{"review_id": "W807zz75IS", "reviewer": "Reviewer_Q4QN", "summary": "", "questions": "What are the computation costs associated with training the retro and retro++ models? For the metrics generated, is it the average? what are the number of runs per experiment?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and feedback. We are glad that the reviewer finds our work demonstrating useful contributions and research findings through detailed discussion and evaluations. We provide detailed responses below.\n\n> 1. “The paper could use more models other than GPT for benchmarking”\n\n  - Thank you for the valuable comments. In Section 7, we conduct an evaluation on two open-domain QA benchmarks and expand our comparisons beyond just GPT. Specifically, we incorporate encoder-decoder models, including RAG_{BART} and FiD_{Large}. The results illustrate that our Retro++ showcases significant improvements over the original Retro and other state-of-the-art models.\n\n  - We'd also like to clarify that both Sections 5 (text generation) and Section 6 ([LM Evaluation Harness benchmark](https://github.com/EleutherAI/lm-evaluation-harness)) mainly focus on auto-regressive decoder models. Therefore, we mainly use GPT pretrained on the same 330B pretraining corpus as Retro as our baseline. \n\n  - In addition, we also add an additional baseline to the LM Evaluation Harness benchmark, which leverages retrieved information in the context of GPT models (GPT + In-context Retrieval). To better incorporate the retrieved evidence, we follow [1] and carefully design four different prompts and perform the zero-shot evaluation on the Hellaswag dataset for Small, Medium, and XL GPT and Retro. The results are shown below. Our results demonstrate that Retro can better leverage the retrieved evidence through the Retro encoder than GPT with in-context retrieval, because it is pretrained with retrieval.\n\n| Hellaswag |     GPT w/o retrieval       |GPT +  In-context  Retrieval (Template 1) | GPT +  In-context  Retrieval (Template 2) | GPT + In-context  Retrieval (Template 3) | GPT + In-context  Retrieval (Template 4) | Retro |\n| ---------|  ---------------------------- | ---------------------------- | ---------------------------- | ---------------------------- | ---------------------------- | -------- |\n| Small    | 31.3   | 34.4                        | 34.4                         | 34.5                        | 34.4                       |  **36.2**  |\n| Medium  | 43.2 | 44.0                           | 44.0                        | 44.1                        | 44.1                        |  **46.2**   |\n| XL       | 56.7    | 57.1                        | 57.3                        | 57.3                      | 57.5                        |  **59.0**    |\n\n\n> 2. “The computation costs are not clearly mentioned. / What are the computation costs associated with training the retro and retro++ models?”\n\n  - Thank you for the valuable suggestion. The training cost of Retro++ is very close to Retro as Retro++ simply adds more context to the decoder at the fine-tuning stage. We have provided our computation costs associated with GPT and Retro below for pretraining on 330B tokens (All of our experiments are done on the DGX-2H node with 8x A100 GPUs).\n\n|        | GPT             | Retro           | Additional Overhead |\n| ------ | --------------- | --------------- | ------------------- |\n| Small  | 1240 GPU Hours  | 1560 GPU Hours  | 25.80%              |\n| Medium | 3600 GPU Hours  | 4480 GPU Hours  | 24.44%              |\n| XL     | 12000 GPU Hours | 13440 GPU Hours | 12.00%              |\n\n\n  - From the table, we can see that the overhead involved in training Retro is less than 25% on average. Considering consistent improvements in text generation quality, factual accuracy, lower toxicity, and downstream task accuracy, especially for knowledge-intensive tasks, including open-domain QA, we believe pretraining Retro is a promising direction. \n\n\n> 3. “For the metrics mentioned in the figures, is it average? the number of runs are not clearly mentioned?  / For the metrics generated, is it the average? what are the number of runs per experiment?”\n\n  - For the EM (Exact Match) scores mentioned in Figure 2, we first conduct grid searches to determine the best hyper-parameters (e.g., batch size from {256, 512, 768}  and learning rate from {5e-7, 1e-6, 3e-6, 5e-6, 1e-5, 5e-5}) for finetuning models of different sizes. Following the standard evaluation practice [2,3], we early-stop finetuning by evaluating the EM on the validation set and report the corresponding EM on the test set in Figure 2 and Table 7. \n\n  - We also follow your suggestion and repeat the fine-tuning experiments on Retro (Medium) on the NQ dataset for three runs with the best hyper-parameters and different random seeds. The EM scores for three runs are 0.4355, 0.4449, and 0.4521. The average EM score and the standard deviation are 0.4442 +- 0.0083, which is close to the numbers shown in Figure 2 (0.4449) and significantly better than RAG_{GPT} of 0.3958.\n\n\n\n[1] Wei, Jason, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai and Quoc V. Le. “Finetuned Language Models Are Zero-Shot Learners.” ArXiv abs/2109.01652 (2021): n. pag.\n\n[2] Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880.\n\n[3] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "> The paper, in detail with very clear flow and writing, reproduces and builds upon previously published work to make the case for introducing a retrieval module in the pre-training stage. \n> The paper clearly demonstrates useful contribution and research findings through the detailed discussion and quantitive and qualtative comparison between retrieval-based model and GPT", "reasons_to_reject": "> The paper could use more models other than GPT for benchmarking\n> The computation costs are not clearly mentioned\n> For the metrics mentioned in the figures, is it average? the number of runs are not clearly mentioned?", "questions_to_authors": "What are the computation costs associated with training the retro and retro++ models? For the metrics generated, is it the average? what are the number of runs per experiment?", "paper_topic_and_main_contributions": "The paper investigates the potential of pretraining large language models with a retrieval model. First the paper reproduces RETRO model and quantitatively and qualtiatively compares it with GPT. Furthermore, it introduces a new variant of RETRO model called RETRO++. The paper clearly demonstrates the performance gains of introducing a retrieval module in the pre-training stage and presents the limitations such as scalability and the impact of the retrieval database quality", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bOKDTQ0Kut", "reviewer": "Reviewer_jg92", "summary": "", "questions": "1. In Table 5 are the differences statistically significant enough?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and feedback. We are glad that the reviewer finds our work investigating an important topic and conducting a wide range of empirical studies. We provide detailed responses below.\n\n\n> 1. “Too few baseline models are included in text-generation (section 5) and the LM evaluation harness benchmark (section6). Since the vanilla GPT is not aided by retrievals, it is not surprising to see it performs worse on knowledge-intensive tasks. The authors said \"We observe that directly prepending the evidence from the retrieval database messes up the GPT context in the zero-shot setting\", but a straightforward alternative is to write an instruction with few-shot examples. I would suggest the authors to include a few retrieval-based baselines, too, such as GPT + retrieval at inference time.”\n\n  - Thank you so much for your constructive suggestion. We'd also like to clarify that both Sections 5 (text generation) and Section 6 ([LM Evaluation Harness benchmark](https://github.com/EleutherAI/lm-evaluation-harness)) mainly focus on auto-regressive decoder models. Therefore, we mainly use GPT pretrained on the same 330B pretraining corpus as Retro as our primary baseline. \n\n  - Following your suggestion, we also add an additional baseline to the LM Evaluation Harness benchmark, which leverages retrieved information in the context of GPT models (GPT + In-context Retrieval). To better incorporate the retrieved evidence, we follow [1] and carefully design four different prompts and perform the zero-shot evaluation on the Hellaswag dataset for Small, Medium, and XL GPT and Retro. The results are shown below. Our results demonstrate that Retro can better leverage the retrieved evidence through the Retro encoder than GPT with in-context retrieval, because it is pretrained with retrieval.\n\n| Hellaswag |     GPT w/o retrieval       |GPT +  In-context  Retrieval (Template 1) | GPT +  In-context  Retrieval (Template 2) | GPT + In-context  Retrieval (Template 3) | GPT + In-context  Retrieval (Template 4) | Retro |\n| ---------|  ---------------------------- | ---------------------------- | ---------------------------- | ---------------------------- | ---------------------------- | -------- |\n| Small    | 31.3   | 34.4                        | 34.4                         | 34.5                        | 34.4                       |  **36.2**  |\n| Medium  | 43.2 | 44.0                           | 44.0                        | 44.1                        | 44.1                        |  **46.2**   |\n| XL       | 56.7    | 57.1                        | 57.3                        | 57.3                      | 57.5                        |  **59.0**    |\n\n\n\n> 2. “In Table 5 are the differences statistically significant enough?”\n\n  - Thanks for your comment. We note that our toxicity evaluation is run over 10k different prompts, and for each prompt, we generate 25 different generations with nucleus sampling to evaluate the worst-case toxicity. So the drop in the average toxicity probability from 40% to 35% is indeed a significant difference by just updating the retrieval database from the pretraining database to the Wiki database. Moreover, we observe a larger (6%) toxicity probability drop from 77% to 71% for toxic prompts, which further demonstrates the flexibility and potential of pretrained retrieval-augmented LLMs.\n\n> 3. “When introducing related works such as DPR, RAG, etc., you might want to provide the full name before using the abbreviation.”\n\n- Thank you so much for your valuable suggestions. DPR represents Dense Passage Retrieval [2], and RAG represents Retrieval-Augmented Generation [3]. We will follow your suggestion and add the full name before using the abbreviation in the revision. \n\n\n[1] Wei, Jason, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai and Quoc V. Le. “Finetuned Language Models Are Zero-Shot Learners.” ArXiv abs/2109.01652 (2021): n. pag.\n\n[2] Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP.\n\n[3] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Relatively comprehensive investigation on an important topic: retrieval-augmented LM. I appreciate that the authors 1) reproduced an retrieval-based LM whose implementation is not open-sourced, based on which the authors later made small improvements, and 2) conducted experiments on a wide range of tasks.", "reasons_to_reject": "Too few baseline models are included in text-generation (section 5) and the LM evaluation harness benchmark (section6). Since the vanilla GPT is not aided by retrievals, it is not surprising to see it performs worse on knowledge-intensive tasks. The authors said \"We observe that directly prepending the evidence from the retrieval database messes up the GPT context in the zero-shot setting\", but a straightforward alternative is to write an instruction with few-shot examples. \n\nI would suggest the authors to include a few retrieval-based baselines, too, such as GPT + retrieval at inference time. ", "questions_to_authors": "1. In Table 5 are the differences statistically significant enough?\n", "paper_topic_and_main_contributions": "This paper investigates whether retrieval helps in pre-training autoregressive LMs. To this end, the authors first reproduce and pretrain RETRO, and then compare the performance of RETRO with GPT on open-ended text generation,  LM Eval Harness Benchmark, and open-domain QA. \n\nThe findings are that 1) RETRO outperforms GPT on text generation with less repetition, moderately higher factual accuracy, and\nslightly lower toxicity with a nontoxic retrieval database. 2) On the LM Evaluation Harness benchmark, RETRO largely outperforms GPT on knowledge-intensive tasks, but is on par with GPT on other tasks. 3) On the QA tasks, it is helpful to incorporate the top retrieved document from DPR  to the decoder as the context. All these findings more or less align with my intuition, so nothing surprising. The experiments are relatively comprehensive, but I think the authors should add a simple baselines which is GPT+retrieval in the few-shot setting. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "When introducing related works such as DPR, RAG, etc., you might want to provide the full name before using the abbreviation."}, "extra_text": {}}, {"review_id": "YFPWHwKhzh", "reviewer": "Reviewer_HmWG", "summary": "", "questions": "Q1: According to the experimental results, we can see that for larger models, pre-training with retrieval leads to smaller improvement, so is this really necessary for LLMs with more than 100b parameters?\n\nQ2: I believe that the effectiveness of \"Retrieval-augmented GPT at Inference time\" might be limited due to its lack of instruction fine-tuning. Based on my experience using ChatGPT on a daily basis, I've noticed that retrieving relevant text initially has led to improvements in the results. Could you please elaborate further on this concept or perhaps provide additional experimental results to support it?\n\nQ3: Will inconsistent retrieval databases used for inference and training have an impact on model performance? If so, would this hinder the ability to update the LM's knowledge?\n\nQ4: The results in Table 6 seem to be unstable and I wonder how robust the model is.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and feedback. We are glad that the reviewer finds our work demonstrating **novel findings** through extensive experiments and highlighting the promising potential of pretraining LLMs with retrieval. We provide detailed responses below.\n\n> 1. “The method of this paper may lack some novelty, as it follows RETRO which is proposed by others. Furthermore, the conclusions drawn from the experiment seem to lack originality.”\n\n  - We would like to clarify that all the conclusions and findings drawn from our experiments are original and have not been discussed in previous work. Although we follow the original RETRO paper (so did many follow-up works of the GPT family), our work provides **novel findings** that shed light on the intrinsic properties of Retro in text generation, downstream tasks, and fine-tuning capabilities. We believe novelty can refer to not only novel models, but also novel findings.  Our extensive experiments demonstrate that there are consistent improvements in text generation quality (less degeneration, higher factual accuracy, and lower toxicity), and downstream task accuracy (especially for knowledge-intensive tasks, including open-domain QA with the simple yet effective RETRO++ architecture) without significant overhead in pretraining. \n\n  - Moreover, given the importance of recent autoregressive LLMs that are expensive to pretrain, our work manages to answer an important and practical problem on whether we should pre-train decoder-only LLMs with retrieval. Our work conveys a clear signal to scale up the model sizes and pretrain larger autoregressive LMs with retrieval as a promising future direction.\n\n\n> 2. “The experimental results are not sufficient to demonstrate that pretraining with retrieval is a more efficient method. (details in Questions): I believe that the effectiveness of \"Retrieval-augmented GPT at Inference time\" might be limited due to its lack of instruction fine-tuning. Based on my experience using ChatGPT on a daily basis, I've noticed that retrieving relevant text initially has led to improvements in the results. Could you please elaborate further on this concept or perhaps provide additional experimental results to support it? / I think the author's comparison of solely RETRO to a base model like GPT to be insufficient. Recently, LLMs are often subject to instruction tuning or RLHF training. I am particularly intrigued by how RETRO's performance stacks up against such models, and I wonder the extent of RETRO's capabilities when employing such advanced techniques.”\n\n  - Thank you for constructive feedback. We follow your suggestion and add instruction tuning to both GPT (XXL) and Retro (XXL) to demonstrate the effectiveness of Retro and retrieval-augmented GPT. Specifically, we construct a blend of instruction tuning datasets consisting of 128K training samples from the [soda dataset](https://huggingface.co/datasets/allenai/soda)[1], [ELI5 dataset](https://github.com/facebookresearch/ELI5) [2], [FLAN dataset](https://github.com/google-research/FLAN) [3], [Open Assistatant dataset](https://huggingface.co/datasets/OpenAssistant/oasst1) [4], etc. We fine-tune both GPT and Retro on the instruction tuning dataset and evaluate the zero-shot accuracy on the Natural Question (NQ) dataset (Note: the training data of NQ is not in our instruction tuning dataset). The results of retrieval-augmented GPT ($\\text{RAG}_\\text{GPT}$) and Retro++ before and after instruction tuning are shown below:\n\n| Zero-shot evaluation on NQ  | $\\text{RAG}_\\text{GPT}$ (XXL)    | Retro++ (XXL)  |\n| ------------------------------ | ------ | ------ |\n| w/o Instruction tuning     | 0.2443 | **0.2593** |\n| w/ Instruction tuning | 0.2975 | **0.3116** |\n\n  - From the table above, we observe that stacking up instruction tuning with Retro and Retrieval-augmented GPT indeed gives significant accuracy improvement. Moreover, Retro demonstrates consistently better accuracy than retrieval-augmented GPT. This result further confirms the potential and capabilities of Retro when employing advanced techniques like instruction tuning.\n\n[1] Kim, Hyunwoo, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap and Yejin Choi. “SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization.” ArXiv abs/2212.10465 (2022): n. Pag.\n\n[2] Fan, Angela, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston and Michael Auli. “ELI5: Long Form Question Answering.” ArXiv abs/1907.09190 (2019): n. Pag.\n\n[3] Wei, Jason, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai and Quoc V. Le. “Finetuned Language Models Are Zero-Shot Learners.” ArXiv abs/2109.01652 (2021): n. Pag.\n\n[4] Kopf, Andreas et al. “OpenAssistant Conversations - Democratizing Large Language Model Alignment.” ArXiv abs/2304.07327 (2023): n. pag.\n\n\n> 3. “According to the experimental results, we can see that for larger models, pre-training with retrieval leads to smaller improvement, so is this really necessary for LLMs with more than 100b parameters?”\n\n  - We would like to note that from Figure 2, the improvement of Retro on knowledge-intensive tasks, including both NQ and TriviaQA datasets, is still significant across models of different sizes. In Figure 2, we do not see any sign of diminishing margin for the improvement. Thus, we believe that it is promising to scale up the model sizes and pretrain large LMs as an important future direction. We treat it as an important future work. \n\n\n> 4. “Will inconsistent retrieval databases used for inference and training have an impact on model performance? If so, would this hinder the ability to update the LM's knowledge?”\n\n  - Inconsistent retrieval databases used in inference and training may not hurt the model performance but improve it instead, which is one of the advantages and flexibility that Retro can bring to update LLM’s knowledge. For example, during inference, we replace the default pretraining database with the Wiki database in our toxicity experiments, which reduces the toxicity of retrieved evidence and thus yields lower toxicity in LLM generations. During training, we replace the default pretraining database with Wikipedia passages retrieved by DPR used in standard QA benchmarks, which yields higher accuracy among the state-of-the-art QA models. \n\n\n> 5. “The results in Table 6 seem to be unstable and I wonder how robust the model is.”\n\n  - We agree that zero-shot evaluation on the LM Evaluation Harness benchmark can be unstable and sensitive to the prompts for the specific tasks (which is known among LLM researchers). Therefore, we conduct the zero-shot evaluation across **nine different tasks**, including 2 knowledge-intensive tasks and 7 knowledge-nonintensive tasks, to demonstrate the accuracy of Retro and GPT. While the average accuracy of Retro across different knowledge-nonintensive tasks is close to GPT, we observe that the average accuracy of Retro across knowledge-intensive tasks is indeed significantly better than GPT, surpassing GPT by 1.84% on average for models of different sizes. \n\n> 6. Missing reference : https://arxiv.org/abs/2305.11564\n  - Many thanks for the reference. This paper introduces PlugLM to pretrain LMs with a differentiable plug-in retrieval module. The authors have applied PlugLM to BERT-based encoder-only models, and demonstrate its capabilities and potential in domain adaptation, knowledge update, and in-task knowledge learning. It would be an interesting future direction to apply the technique to autoregressive decoder-only LMs. We will add the reference and include more discussion in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors reproduce RETRO and conduct extensive experiments comparing with GPT, yielding novels findings. Their work highlights the promising potential of pre-training language models with retrieval.", "reasons_to_reject": "- The method of this paper may lack some novelty, as it follows RETRO which is proposed by others. Furthermore, the conclusions drawn from the experiment seem to lack originality.\n- The experimental results are not sufficient to demonstrate that pretraining with retrieval is a more efficient method. (details in Questions)\n- I think the author's comparison of solely RETRO to a base model like GPT to be insufficient. Recently, LLMs are often subject to instruction tuning or RLHF training. I am particularly intrigued by how RETRO's performance stacks up against such models, and I wonder the extent of RETRO's capabilities when employing such advanced techniques.", "questions_to_authors": "Q1: According to the experimental results, we can see that for larger models, pre-training with retrieval leads to smaller improvement, so is this really necessary for LLMs with more than 100b parameters?\n\nQ2: I believe that the effectiveness of \"Retrieval-augmented GPT at Inference time\" might be limited due to its lack of instruction fine-tuning. Based on my experience using ChatGPT on a daily basis, I've noticed that retrieving relevant text initially has led to improvements in the results. Could you please elaborate further on this concept or perhaps provide additional experimental results to support it?\n\nQ3: Will inconsistent retrieval databases used for inference and training have an impact on model performance? If so, would this hinder the ability to update the LM's knowledge?\n\nQ4: The results in Table 6 seem to be unstable and I wonder how robust the model is.", "paper_topic_and_main_contributions": "In this paper, the authors delve into the question of whether we should pretrain large autoregressive LMs with retrieval. To address this question, the authors reproduce a scalable pretrained retrieval-augmented LM, Retro, and conduct a comprehensive study revealing the following novel findings:\n- RETRO outperforms GPT on text generation with much less degeneration (i.e., repetition), moderately higher factual accuracy, and slightly lower toxicity with a nontoxic retrieval database.\n- On the LM Evaluation Harness benchmark, RETRO largely outperforms GPT on knowledge-intensive tasks, but is on par with GPT on other tasks. \n\nFurthermore, the authors propose a variant of the model, RETRO++, which significantly enhances the performance of the open domain QA task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "missing reference: https://arxiv.org/abs/2305.11564"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the impact of retrieval on pretraining autoregressive LLMs. It reproduces the RETRO model and compares it with GPT, and proposes a new variant of it called RETRO++. The paper presents several useful experimental analyses and discussions such as performance gains and impact of retrieval database quality. The community can benefit from this work in furthering research on retrieval-augmented generation, which is becoming increasing important given the advent of LLMs. 

For the authors — please include the additional experiments and insights you shared in the author responses, they would make the paper much stronger.
Decision: Accept-Main