Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
Authors: Akari Asai
Zeqiu Wu
Yizhong Wang
Avirup Sil
Hannaneh Hajishirzi
Venue: ICLR 2024 oral (2024)
Keywords: Retrieval-augmented Generation
Language Models
Retrieval-augmented LMs
Factuality
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce Self-RAG, a new training and inference framework to enable an LM learn to retrieve, generate and critique.

Abstract: Despite their remarkable capabilities, large language models (LLMs) often produce responses containing factual inaccuracies due to their sole reliance on the parametric knowledge they encapsulate. Retrieval-Augmented Generation (RAG), an ad hoc approach that augments LMs with retrieval of relevant knowledge, decreases such issues. However, indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation. We introduce a new framework called **Self-Reflective Retrieval-Augmented Generation (Self-RAG)** that enhances an LM's quality and factuality through retrieval and self-reflection. 
Our framework trains a single arbitrary LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its generations using special tokens, called {\it reflection} tokens. Generating reflection tokens makes the LM controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements. 
Experiments show that Self-RAG (7B and 13B parameters) significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks. 
Specifically, Self-RAG outperforms ChatGPT and retrieval-augmented Llama2-chat on Open-domain QA, reasoning, and fact verification tasks, and it shows significant gains in improving factuality and citation accuracy for long-form generations relative to these models. Our code and trained models are available at https://selfrag.github.io/

Published as a conference paper at ICLR 2024
SELF-RAG: LEARNING TO RETRIEVE, GENERATE, AND CRITIQUE THROUGH SELF-REFLECTION
Akari Asai', Zeqiu Wu', Yizhong Wang Avirup Sil Hannaneh Hajishirzi
TUniversity of Washington Allen Institute for AI IBM Research Al fakari, zegiuwul, ynaiii,izzha rizhongw, hannaneh @cs. washington. edu, avi avieus. ibm. com
ABSTRACT
Despite their remarkable capabilities, large language models (LLMs) often produce responses containing factual inaccuracies due to their sole reliance on the paramet- ric knowledge they encapsulate. hoc approach that augments LMs with retrieval of relevant knowledge, decreases such issues. However, indiscriminately retrieving and incorporating fixed number of retrieved passages, regardless of whether retrieval is necessary. or passages are relevant, diminishes LM versatility or lead to unhelpful response generation. We introduce a new framework called ReffeReflective eration (SEL.F-RAG) that enhances an LM's quality and factuality through retrieval and self-reflection. Our framework trains a single arbitrary LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its own generations using special tokens, called reflection tokens. Generating reflection tokens makes the LM controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements. Experiments show that SELF- RAG (7B and 13B parameters) significantly outperforms state-of-the-art LLMS and vetrieval-augmentte outperforms ChatGPT and retrieval-augee -uggeettd Llama2-chat on Open-domain QA, reasoning and fact verification tasks, and it shows significant gains in improving factuality and citation accuracy for long-form generations relative to these models]
Generation (RAG), ad
-arrivaalAuugmented Gen-
models diverse set of tasks. Specifically, SELF-RAC
INTRODUCTION
State-of-the-art LLMs continue to struggle with factual errors (Mallen et al. 2023 Min et al. 2023)
despite their increased model and data scale (Ouyang al. 2022). Retrieval- Augmentted Generation (RAG) methods (Figure [ left; Lewis et al. 2020; Guu et al. 2020) augment the input of LLMs with relevant retrieved passages, redueing factual errors in knowledge-intensive tasks (Ram al.)
2023
Asai et al. 2023a). However, these methods may hinder the versatility of LLMs or introduce
unnecessary off-topic passages that lead low-quality generations (Shi al., (023) since they
retrieve passages indiscriminately regardless of whether the factual grounding is helpful. Moreover, the output is not guaranteed to be consistent with retrieved relevant passages (Gao et al.) (023) since
the models are not explicitly trained to leverage and follow facts from provided passages.
This work introduces Self-Reflective Retrieval-augmented Generation (SELF-RAG) to improve an LLM's generation quality, including its factual accuracy without hurting its versatility, via on-demand retrieval and self-reflection. We train an arbitrary LM in an end-to-end manner to learn to reflect on its own generation process given task input by generating both task output and intermittent special tokens (i.e., reflection tokens). Reflection tokens are categorized into retrieval and critique tokens to indicate the need for retrieval and its generation quality respectively (Figure right). In particular, given an input prompt and preceding generations, SELF-RA RELF-RAG first determines if augmenting the continued generation with retrieved passages would be helpful. If so, it outputs retrieval token that calls a retriever model on demand (Step 1). Subsequently, SELF-R AG concurrently processes multiple retrieved passages, evaluating their relevance and then generating corresponding task outputs (Step 2). It then generates critique tokens to criticize its own output and choose best one (Step 3) in terms of factuality and overall quality. This process differs from conventional RAG (Figure I left), which
Our code and trained models available httpt:://////..... ///e/ frag. frag.gi thub.

---

Published as a conference paper at ICLR 2024
Petrieval-Augmennt Generation (RAG)
Ours: Self-refiective Retrieval- Augmented Generation (Self-RAG)
Prompt did US thei names?
did thei names?
Step 1: Retrivee demand
Stop 1: Rebriove K documonts
US clates got their names from a varkety sources,
Dity states, dloven named
persoors
Step Generate segment parallel
Populler mammas by states, In Phooos, Xmmms populer taby namm
Prompt
Prompt
Prompt
3
Retriever
Californda was narzed after Sottonal Spanish:
Step 2: Prompt LM with docs and generate
11 of 50 state names
Insivann
Toean namand
Cel ric's name has is 1Bthicantury novel
Nitton Arrer inioo.
orgins
Prompt How did US states get their rames?
Sergas de Explandión.
Portily
Step 3: Critique outputs and select best segment
UG states got their names from 3 vartery of
Adk roies Flavin aratio we named wher
indivibull parsor: (S.A, Celifornio wisd nemed
after Cit stochor modumb Same states
get
inch/ Ioons Utsh. are ramed
Repeat....
states names are come from parsons. 26 states are named Natvo Amoricans, inclucing
Contadletooy
meroon 2l00 No information in pansages
Prompt: Wittle an essay of your best summer vacation
Prompi: Wrlte easny your
bost...
My best summmar vacalion is whan my family and emiaakes on road tr along...
Figure I: Overview of RELF-RAG. SELF-RAG learns retrieve, critique, and generate text passages to enhance overall generation quality, factuality, and verifiability.
consistently retrieves fixed number of documents for generation regardless of the retrieval necessity (e.g., the bottom figure example does not require factual knowledge) and never second visits the generation quality. Moreover, SELF-RAG provides citations for each segment with its self-assessment of whether the output is supported by the passage, leading to easier fact verification.
SELF-RAC trains an arbitrary LM to generate text with reflection tokens by unifying them as the next token prediction from the expanded model vocabulary. We train our generator LM on diverse collection of text interleaved with reflection tokens and retrieved passages. Reflection tokens, inspired by reward models used in reinforcement learning (Ziegler al. 2019 Ouyang et al., 2022). inserted offline into the original corpus by a trained crific model. This climinates the need to host a critic model during training, reducing overhead. The critic model, in part, is supervised on a dataset of input, output, and corresponding reflection tokens collected by prompting a propriety LM (i.e., GPT-4: A1peA1[2023] While we draw inspiration from studies that use control tokens to start and guide text generation (Lu et al. 2022 Keskar et al. 2019),, our trained LM uses critique tokens to assess its own predictions after each generated segment an integral part of the generation output.
SELF-RAG further enables customizable decoding algorithm to satisfy hard or soft constraints, which are defined by reflection token predictions. In particular, our inference-time algorithm enables us to (1) flexibly adjust retrieval frequency for different downstream applications and (2) customize models' behaviors user preferences by leveraging reflection tokens through segment-level beam search using the weighted linear sum of the reflection token probabilities as segment score.
Empirical results on six tasks, including reasoning and long-fop form generation, demonstrate that SELF- RaG significantly outperforms pre-trained and instruction-tunee L.MSs that have more parameters and widely adopted RAG approaches with higher citation accuracy. particular, SELF-R RAG outperforms eetrieval-augmente
ChatGPT on four tasks, Llama2-chat (Touvron et al.. (023) and Alpaca (Dubois) cct al. 2023) on all tasks. Our analysis demonstrates the effectiveness of training and inference with reflection tokens for overall performance improvements well as test-time model customizations (e.g., balancing the trade-off between citation previsions and completeness).
2 RELATED WORK
Retrieval-Augmeente Generation. Setrieval-Augmentee Generation (RAG) augments the input space of LMs with retrieved text passages (Guu et al. 2020 Lewis et al., 2020), leading to large improvements in knowledge-intensive tasks after fine-tuning or used with off-the-shelf LMs (Ram et al. 2023). A more recent work (Luo et al. 2023) instruction-tunes an LM with fixed number
2

---

Published as a conference paper at ICLR 2024
Type
Input
Output
Definitions
Retrieve
x v,y y
{yes, no, continuc) rrelevant, irrelevant)
Decides when to retrieve with R provides useful information to solve a. All of the verification- worthy statement in is supported by d. is useful response
ISREL
2, d
ISSUP
2, d. y
(fully supported, partially supported, no support) 15. 3, 1}
IsUSE
2, y
Table 1: Four types of reflection tokens used in SELF-RAG Each type uses several tokens to represent its output values. The bottom three rows are three types of Crriique tokens, and the bold text indicates the most desirable critique tokens. I, y, d indicate input, output, and a relevant passage, respectively.
of retrieved passages prepended to input, or pre-train a retriever and LM jointly, followed by few- shot tinettunig on task datasets (zacard al., 2022b). While prior work often retrieves only once at the beginning, Jiang et al. (2023) propose to adaptively retrieve passages for generation on top of a proprietary LLM [Schick et al. (2023) train an LM to generate API calls for named entities. Yet, the improved task performance of such approaches often comes at the expense of run-time efficiency (Mallen et al. 2023), robustness to irrelevant context (Shi et al. 2023), and lack of attributions (Liu et al.) [023a] Gao et al. (023). We introduce method to train an arbitrary LM to learn to use retrieval on on-demand for diverse mnsluuction-folloiing queries and introduce controlled generation guided by reflections tokens to further improve generation quality and attributions.
Training and generating with critics. Training LLMs with reinforcement learning (e.g.. Proximal Policy Optimization or PPO: |Schulman et al. 2017) from human feedback (RIHF) has proven effective aligning LLMs with human preferences (Ouyang et al., 2022 Wu et al. 2023 Though our work also studies fine-grained critique on retrieval and generation, we train our target LM on task examples augmented with reflection tokens from a critic model offline, with a far lower training cost compared to RLHE. Compared to prior work using control tokens to guide LM generation (Lu et al., 2022] Korbak et al., 2023), SELF-RAG uses reflection tokens to decide the need for retrieval and to self-evaluate generation quality.
SELF-RAG: LEARNING TO RETRIEVE, GENERATE AND CRITIQUE
We introduce Self-Reflective Retrieval-Aure Setrieval-Augmenneed Generation (EI.F-RAG) shown Figure SEIF-RAG is framework that enhances the quality and factuality of an LLM through retrieval and self-reficction, without sacrificing LLM's original creativity and versatility. Our end-to-end training lets an LM M generate text informed by retrieved passages, if needed, and criticize the output by learning to generate special tokens. These reflection tokens (Table T signal the need for retrieval or confirm the output's 's relevance, support, completeness. In contrast, common RAG approaches retrieve passages indiscriminately. without ensuring complete support from cited sources.
3.1 PROBLEM FORMALIZATION AND OVERVIEW
Formally, given input x, we train M to sequentially generate textual outputs y consisting of multiple segments y , Ur)] where y, indicates a sequence of tokens for the /-th segment. Generated tokens in ye include text from the original vocabulary as well as the reflection tokens (Table
Inference overview. Figure and Algorithm Maashmmmpprssese present overview of SELF-RAC inference. For every œ and preceding generation yeu. the model decodes a retrieval token to evaluate the utility of retrieval. If retrieval not required, the model predicts the next output segment, as it does in standard LM. If retrieval is needed, the model generates: a critique token to evaluate the retrieved passage's relevance, the next response segment, and a critique token to evaluate if the information in the response segment is supported by the passage. Finally, new critique token evaluates the overall utility of the response 3l To generate each segment, SELF-- R-RAC AG processes multiple passages in parallel and uses its own generated reflection tokens to enforce soft constraints (Section 3.3 or hard control
II this paper, we treat one sentence as a segment in our experiments, hut our framework applicable to any segment unit (i.c., sub-sentence). We follow Liu et al. (2023a) in using "perceived" utility value that is independent of retrieved passages.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 SELF-RAG Inference
Require:
Generator LM M, Retriever R, Large-scale passage collections fdi, ..... ,Ny
1: Input: input prompt z and preceding generation y<c. Output: next output segment y/
2: M predicts
Retrieve
given (x, (<+))
3: if |Retrieve
= Yes then
4: Retrieve relevant text passages D using R given (I, 3t-1)
b Retrieve
5: M predicts IsREL given æ, d and given 2, d, Vez for each d E D
> Generate
6: M predicts
ISSUP
and
IsUSE given 2, yt, d for each d D
>> Critique
7:
Rank based on ISREL
ISSUP
ISUSE
Detailed in Section 3.3
8:
else if Retrieve
No then Mgen predicts 1/ given 2
9:
> Generate
Mgen predicts IsUSE 10:
given I; 3/t
b. Critique
(Algorithm I over the generated task output. For instance, in Figure (right)) the retrieved passages di is selected at the first time step since d2 does not provide direct evidence [[SREL] is Irrelevant) and d3 output is only partially supported while di are fully supported.
Training overview. SELF-RAG enables an arbitrary LM to generate text with reflection tokens by unifying them as next token predictions from the expanded model vocabulary (i.e., the original vocabulary plus reflection tokens). Specifically, we train the generator model .M on a curated corpus with interleaving passages retrieved by retriever R and reflection tokens predicted by critic model (summarized in Appendix A Algorithm 2} We train C to generate reflection tokens for evaluating retrieved passages and the quality of a given task output (Section 3.2.1). Using the critic model, update the training corpus by inserting reflection tokens into task outputs offline. Subsequently. we train the final generator model (M) using the conventional LM objective (Section 3.2.2) enable M to generate reflection tokens by itself without relying on the critic at inference time.
3.2 SELF-RAC TRAINING
Here, we describe the supervised data collection and training of two models, the critic (Section 3.2. and the generator M (Section 3.2.2).
3.2. 1 TRAINING THE CRITIC MODEL.
Data collection for critic model. Manual annotation of reflection tokens for each segment is expensive (Wu et al. 2023). A statc-of-th -fr-r--rtttttttt LLM like GPT-4 (Open can be effectively used generate such feedback (Liu al. 20236). However, depending on such proprietary LMs can raise API costs and diminish reproducibility (Chen et al., 2023). Our method requires fine- grained evaluations on multiple passages as well as segments for each input-output instance from the training dataset, increasing the number of evaluations required to generate SELF-RAG training data exponentially, To overcome those issues, we create supervised data by prompting GPT-4 to generate reflection tokens and then distill their knowledge into an in-house C. For each group of reflection tokens, we randomly sample instances from the original training data: {xsample ysample) ~ (X,Y). As different reflection token groups have their definitions and input, shown in Table we use different instruction prompts for them. Here, we use Retrieve] as an example. We prompt GPT-4 with a type-specific instruction ("Given an instruction, make judgment whether finding some external documents from the web helps to generate a better response.") followed by few-shot demonstrations the original task input x and output y to predict an appropriate reflection token as text: p(r[T, y). Manual assessment reveals that GPT-4 reflection token predictions show high agreement with human evaluations. We collect 4k-20% supervised training data for each type and combine them form training data for C. Appendix Section D shows the full list of instructions, and A. contains more details and our analysis.
Critic learning. After we collect training data Deräsee we initialize C with a pre-trained LM and train it on Deritic using a standard conditional language modeling objective, maximizing likelihood:
max (((.))) ()) log logpe(r' (r(pe) v), r for reflection tokens.
(1)
4

---

Published as a conference paper at ICLR 2024
Input: Write an essay of your best summer vacation
Inputt How cld US states get thei names?
Output: My coastal town Santorini azuro charming White washed building are unforgettalee
mag escape
Output: states from per persons: For Louisiana ramed honor
XN Geargia after King George
Crttic
Retriever
Augmented Oulpul: Netsowal
Augmsented Output:
rp-orthe diy
persens/gs
vacation was magical escape the coastal town of charming white-
Relwwar statles remmess sorr from parsens
Suppated
pototttt Itrned
Santorrni, [outhssioa The
For Instrnnw. Loukbbea named wher King luue XV,
washed builiing are unlorgettable experience
U
Georgia was named after King George
Figure 2: SELF-RAC training examples. The left example does not require retrieval while the right one requires retrieval; thus, passages are inserted. More examples are in Appendix Table 4
Though the initial model can be any pre-trained LM, we use the same one as the generator LM (i.e., Llama 2-7B; Touvron et al. 2023) for C initialization. The critic achieves higher than 90% agreement with GPT-4-based predictions on most reflection token categories (Appendix Table 4
3.2.2 TRAINING THE GENERATOR MODEL
Data collection for generator. Given an input-output pair (1r, y). we augment the original output y using the retrieval and critic models to create supervised data that precisely mimics the SELF- RaG inference- timennee-tmeeeeee process (Section B.I). For each segment Yt E y, we run to assess whether additional passages could help to enhance generation. If retrieval is required, the retrieval special token Retrieve =Yes is added, and R retrieves the top K passages, D. For each passage, C further evaluates whether the passage is relevant and predicts [SREL] If passage is relevant, C further evaluates whether the passage supports the model generation and predicts [sSur] Critique tokens IsRea and [sSuP] are appended after the retrieved passage or generations. At the end of the output, y
(or yr), C predicts the overall utility token ISUSE] and an augmented output with reflection tokens
and the original input pair is added D Pyen. See the example training data in Figure 2
Generator learning, We train the generator model .M by training on the curated corpus augmented with reflection tokens D Dgen using the standard next token objective:
max E (e P.g r(z). M
Unlike training (Eq. M learns to predict the target output as well as the reflection tokens. During training, we mask out the retrieved text chunks (surrounded by <p> and </p> in Figure 2) for loss calculation and expand the original vocabulary with a set of reflection tokens
Critique
Retrieve
Connections to prior work on learning with eritique. Recent work incorporates additional critique (feedback) during training, e.g., RLHF (Ouyang et al. 2022) via PPO. While PPO relies on separate reward models during training. we compute critique offline and directly insert them into the training corpus, where the generator LM is trained with standard LM objective. This significantly reduces training costs compared to PPO. Our work also relates to prior work that incorporates special tokens to control generation (Keskar et al. 2019: Lu et al. [022]] Korbak et al. (023). Our SELF-RAC learns to generate special tokens to evaluate its own prediction after each generated segment, enabling the use of soft re-ranking mechanism or hard constraints at inference (discussed next).
3.3 SELF-RAG INFERENCE
Generating reflection tokens to self-evaluate its own output makes SELF-RAG controllable during the inference phase, enabling it to tailor its behavior to diverse task requirements. For tasks demanding factual accuracy (Min et al. 2023), we aim for the model to retrieve passages more frrequenty to ensure that the output aligns closely with the available evidence. Conversely, in more open-ended tasks, like composing personal experience essay, the emphasis shifts towards retrieving less and prioritizing the overall creativity or utility score. In this section, we describe approaches to enforce control to meet these distinct objectives during the inference process.
Adaptive retrieval with threshold. SELF-RAG dynamically decides when to retrieve text passages by predicting Retrivee] Alternatively, our framework allows threshold to be set. Specifically, if the prob-
5

---

Published as a conference paper at ICLR 2024
abilitt of generating the [Retriev)=YYes token normalized over all output tokens in Retrieve surpasses designated threshold, we trigger retrieval (details in Appendix Section A.44.
Tree-decoding with critique tokens. At each segment step t. when retrieval is required, based either on hard or soft conditions, R. retrieves K passages, and the generator M processes each passage in parallel and outputs K different continuation candidates. We conduct a segment-level beam search (with the beam size=B) to obtain the top-B segment continuations at each timestamp t, and return the best sequence the end of generation. The score of each segment Y/2 with respect to passage d is updated with a critic score S that is the linear weighted sum of the normalized probability of each Critique token type. For each critique token group G (e.g.. ISREL ) we denote its score at timestamp
t as of and we compute a segment score as follows:
(Jte, d.
Critique
=
p(Ue x d, Vct)) + S Critique )
where
(3)
Critique
wOs G St for G [IREL
IsSur
IsUsE
(4)
GEg
where o6
( stands for the generation probability of the most desirable reflection token
1 (r()
=Re 1 evant) for the critique token type G with N distinct tokens (that represent different possible values for G). The weights w in Eq. eq..aare are hyperparameters that can be adjusted at inference time to enable customized behaviors at test time. For instance, to ensure that result y is mostly supported by evidence, we can set a weight term for the [sSur] score higher, while relatively lowering weights for other aspects. Alternatively, we could further enforce hard constraints during decoding using Critique e.g., filtering out a segment continuation when the model generates an
i (e.g.- ISREL
undesirable token (e.g.. IsSur
=No support).
4 EXPERIMENTS
4.1 TASKS AND DATASETS
We conduct evaluations of our SELF-RAG and diverse baselines on a range of downstream tasks, holistically evaluating outputs with metrics designed to assess overall correctness, factuality, and fluency. Throughout these experiments, we conduct zero-shot evaluations, where we provide instruc- tions describing tasks without few-shot demonstrations (Wei et al. [0022] Sanh et al.) (022). Details of our experiments' settings, including test-time instructions, are available in the Appendix Section B.]
Closed-set tasks include two datasets, i.e., fact verification dataset about public health (PubHealth; Zhang et al. (023)) and a multiple-choice reasoning dataset created from scientific exams (ARC- Challenge: Clark et al. (018)). We use accuracy as an evaluation metric and report on the test set. We aggregate the answer probabilities of target classes for both of these datasets (Appendix Section B.2)
Short-form generations tasks include two open-domain question answering (QA) datasets, PopQA Mallen et al.) 2023) and TiivieeA-unfitereed (Joshi et al., 2017), where systems need to answer arbitrary questions about factual knowledge. For PopQA, consisting of 1,399 rare entity queries whose monthly Wikipedia page views are less than 100. As the AvuriaQA-ufflttreee (open) test set not publicly available, we follow prior work's validation and test split (Min et al. 2019; Guu et al. 2020), using 11,313 test queries for evaluation. We evaluate performance based on whether gold answers are included the model generations instead of strictly requiring exact matching, following Mallen et al. (2023;; Schick et al. (2023).
the long-tail subset,
Long-form generation tasks include biography generation task (Min et al. 2023 and a long-
QA task ALCE-ASQA Gao et al.] (2023); (Stelmakh et al. (2022). We use FactScore (Min et al. 2023) to evaluate biographies, and we use official metries of correctness (str-em), fluency based on MAUVE (Pillutla et al. 2021), and citation precision and recall (Gao et al. 2023) for ASQA.
4.2 BASELINES
Baselines without retrievals. We evaluate strong publicly available pre-trained LLMs. Llama2 73. (Touvron et al. 2023 instruction-tuned models, Alpaca7s,133 (Dubois et al. 2023)

---

Published as a conference paper at ICLR 2024
(our replication based Llama2); and models trained and reinforeed using private data, Chat- GPT (Ouyang et al. 2022) and Llama2-chatisa For instruction- tuned LMs, we use the official system prompt instruction format used during training if publicly available. We also compare method to concurrent work, CoVE6ss (Duuliawala et al. 2023), which introduces iterative prompt engineering improve the factuality of L.LM generations.
Baselines with retrievals. We evaluate models augmented with retrieval at test time or during training. The first category includes standard RAG baselines, where LM (Llama2, Alpaca) generates output given the query prepended with the top retrieved documents using the same retriever as in our system. also includes Llama2-FT, where Llama2 is fine-tuned on all training data we use without the reflection tokens or retrieved passages. We also report the result of retrieval-augmented baselines with LMs trained with private data: Ret-ChatGPPT and Ret-Llama2-chat, which deploy the same augmentation technique above, as well as perplexity.ai, an InstructGPT-basee production search system. The second category includes concurrent methods that are trained with retrieved text passages, i.e., SAIL (Luo et al. (023) to instruction-tune an LM on the Alpaca instruction-tuning data with top retrieved documents inserted before instructions, and Toolformer (Schick et al. 2023 to pre-train train an LM with API calls (e.g., Wikipedia APIs)
4.3 EXPERIMENTAL SETTINGS
Training data and settings. Our training data consists of diverse instruction-folloiing input-output pairs. In particular, we sample instances from Open- Open-Instruct processed data (Wang et al., 2023) and
sooledgg-intensive knowledge- intensive datasets (Petroni et al.] 2021 Stelmakh et al.) [2022] Mihaylov et al. 2018). 3) In
total, we use 150k instruction-outipu pairs. We use Llama2 7B and 13B (Touvron et al. 2023 as our generator base L.M, and we use Llama2 7B as our base critic LM. For the retriever model R, we use off-the-shelf Contriever-MS MARCO (Izacard et al.) 2022a) by default and retrieve up to ten documents for each input. More training details are in the Appendix Section B.1
Inference settings. As a default configuration, we assign the weight terms values of 1.0, 1.0 and 0.5, respectively. To encourage frequent retrieval, we set the retrieval threshold to 0.2 for most tasks and to 0 for ALCE (Gao et al., (023) due to citation requirements. We speed up inference using vllm (Kwon et al., 2023). At each segment level, we adopt a beam width of 2. For a token-level generation, we use greedy decoding. By default, we use the top five documents from Co MARCO (Izacard al., [202aa;; for biographies and open-domain QA, use additional top five documents retrieved by web search engine, following Luo et al. (003)); for ASQA, we use the author-provided top 5 documents by GTR-XXL (Ni al. 2022 across all baselines for fair comparison.
ISREL
ISSUP
ISUSE
5 RESULTS AND ANALYSIS
5.1 MAIN RESULTS
Comparison against baselines without retrieval. Table (top) (top) presents the baselines without retrieval. Our SELF-RAG (bottom two rows) demonstrates substantial performance advantage over supervised fine-tuned LLMs in all tasks and even outperforms ChatGPT in PubHealth, PopQA, biography generations, and ASQA (Rouge and MAUVE). Our approach also significantly outperforms a concurrent method that employs sophisticated prompt engineering: specifically, on the bio generation task, our 7B and 13B models outperform the concurrent CoVE (Dhuliawala et al. 2023), which iteratively prompts Llama265a to refine output.
Comparison against baselines with retrieval. As shown in Tables (bottom) our SELF-RAG also outperforms existing RAG in many tasks, obtaining the best performance among non-proprietary LM- based models on all tasks. Powerful instruction tuned LMs with retrieval (e.g.. LLama2-chat, Alpaca) show large gains from their non-retrieval baselines. However, we found that these baselines provide limited solutions for tasks where we cannot simply copy or extract sub-strings of retrieved passages. On PubHealth and ARC-Challenge, baselines with retrieval do not improve performance notably from their no-retrieval counterparts. We also observe that most baselines with retrieval struggle to improve citation accuracy. On ASQA, our model shows significantly higher citation precision and
'ee report numbers using the results reported in the paper as the implementations are available.
7

---

Published as a conference paper at ICLR 2024
Table Overall experiment results six tasks, Bold numbers indicate the best performance among con-poprrettary models, and gray-colored bold text indicates the best proprietary model when they outperforms all non-proprietary models. indicates concurrent recent results reported by concurrent work. - indicates numbers that are not reported by the original papers or are not applicable. Models are sorted based on scale. FS, em, rg, mau, pree, rec denote FactScore (factuality); str-em, rouge (correctness); MAUVE (fluency); citation precision and recall, respectively.
Short-form PopQA (acc)
Closed-set
Long-forn generations (with citations)
TQA (acc)
Pub (acc)
ARC
Bio
ASQA
LM
(acc)
(FS)
(em)
(rg)
(mau)
(pre) (rec)
LMs with proprietary data
Llama2-Cin
59.3
49.4
38.4
55.9
22.4
29.6
28.6
Ret-Llama2-C13e
51.8
59.8
52.1
37.9
79.9
32.8
34.8
43.8
19.8
36.1
ChatGPT
29.3
74.3
70.1
75.3
71.8
35.3
36.2
68.8
Ret-CharGPP
50.8
65.7
54.7
75.3
40.7
39.9
79.7
65.1
76.6
Perplexity.ai
71.2
Baselines without retrieval
1lama27
14.7
30.5
34.2
21.8
44.5
7.9
15.3
19.0
Alpacazs
23.6
54.5
49.8
45.0
45.8
18.8
29.4
61.7
13ama2
14.7
38.5
29.4
29.4
53.4
7.2
12.4
16.0
Alpacaı3s
24.4
61.3
55.5
54.9
50.2
22.9
32.0
70.6
CoVEssB
71.2
Baselines with retrieval
Toolformer* 6B
48.8
Llama27u
38.2
42.5
30.0
48.0
78.0
15.2
22.1
32.0
2.9
4.0
Alpacars
46.7
64.1
40.2
48.0
76.6
30.9
33.3
57.9
5.5
7.2
Llamaa-FT7s
48.7
57.3
64.3
65.8
78.2
31.0
35.8
51.2
5.0
7.5
SAII *7
69.2
48.4
Llama2
45.7
47.0
30.2
26.0
77.5
16.3
20.5
24.7
2.3
Alpacar3
46.1
66.9
51.1
57.6
77.7
34.8
36.7
56.6
2.0
3.8
Our SELF-RĀG 76
54.9
66..""
72,4
67.3
81.2
50.0
35.7
74.3
66.9
67.8
Our SELF-RAG 13n
55.8
69.3
74.5
73.1
80.2
31.7
37.0
71.6
70.3
71.3
PubHealth
PQA
Med (acc)
AS
70.5
(acc)
(em)
Precision
Accuracy
0.99
Froquen Prequency
0.5
SELF-RAS RAG (50k)
45.5
73.5
32.1
70.0
0.99
Training
No Retriever R.
43.6
67.8
31.0
PopQA
No Critic
42,6
72.0
18.1
Test
Mauve
Accuracy
0.8
0.75
Prequono
No retrieval
24.7
73.0
0.50
Hard constraints
28.3
72.6
0.6
Retrieve topl
41.8
73.1
28.6
0.25
Remove IsSuP
44.1
73.2
30.6
Weight for IsSupport
0.4 Retrieval Threshold
0.2
(a) Ablation
(b) Customization
(c) Retrieval
Figure 3: Analysis on SELF-RAG: (a) Ablation studies for key components of SELF-RAG training and inference based our 7B model. (b) Effects of soft weights on ASQA citation precision and Mauve (fluency). (c) Retrieval frequency and normalized accuracy on PubHealth and PopQA.
recall than all models except ChatGPT. Gao et al. (2023) found that ChatGPT consistently exhibits superior efficacy in this particular task, surpassing smaller LMs. Our SELF-RAG bridges this performance gap, even outperforming ChatGPT in citation precision, which measures whether the model- -eeerrtee claim fully supported by cited evidence. Llama2-FTza which is the baseline LM trained the same instruction-output pairs SELF-RAG without retrieval self-reflection and vetrieval-augmeetee at test time only, lags behind SELF- RAG. This result indicates SELF-RAG gains not solely from training data and demonstrate the effectiveness of RELF-RAC framework.
8

---

Published as a conference paper at ICLR 2024
5.2 ANALYSIS
Ablation studies. We conduct a set of ablations of our framework to identify which factors play key roles. We evaluate two model variants trained differently than model: No Retriever trains an LM using the standard cosruction-following method given instruction-outpuu pairs, without retrieved passages; No Critic trains LM trained with input-output pairs that are always augmented with the top one retrieved document without reflection tokens. This is similar to SAIL (Luo et al.] [2023). and we use instruction-output data instead of using the Alpaca dataset (Dubois et al. 2023), in SAIL. We also conduct ablation on our inference-time algorithm, including No retrieval disables retrieval during inference: Hard constraints indicates the model performance that retrieves when Retrieve YYes instead of using the adaptive threshold; Retrieve top 1 always retrieves and uses the top one document only, similar to standard RAG approaches; Remove [sSur] indicates the model performance that removes ISSUP score only during critique- guuddedd beam search in Eq. 4 In this ablation experiment, we use training instance size of 50k for more efficient exploration training variations. Later in this section, we conduct an analysis of the effect of training data size. We conduct the ablation studies on three datasets, PopQA, PubHealth, and ASQA. On ASQA, we evaluate models on sampled 150 instances and exclude ablations involving adaptive
retrieval processes.
We show in Table 3a the ablation results. The top part of the table shows results for training ablations, and the bottom part for inference ablations. We see that all components play important roles. We also observe a large performance gap between SELF-RAG and No Retriever or Critic baselines across tasks, indicating that training an LM with those models largely contributes to the performance gain of SELF-RAG. Using the top passages regardless of their relevance (Retrieve top 1) as in conventional RAG approaches causes large drop in PopQA and ASQA, and removing IsSur during the beam search results hurts performance on ASQA. This demonstrates the effectiveness of SEL.F-R -AAFF' capabilities of carefully selecting generations based on fine-graince multiple criteria, instead of naively using all passages from the retrieval model or solely depending on relevance scores.
Effects of inference-time customization. One key benefit of our proposed framework is that enables us to control how much each critique type affects the final generation sampling. We analyze the effects of different parameter weights on the top of our 7B model during inference time on ASQA, where multiple evaluation aspects are considered. Figure Bbjree33bshooor shows the effects of changing the weighting term for [sSuP] which criticizes how supported the output is by the text passage. As the figure shows, increasing the weight leads to positive effects on the models' citation precision since this puts more emphasis on whether model generation is supported by the evidence. On the contrary, larger weight results in lower MAUVE scores: when generation gets longer and more fluent, there are often more claims that are not fully supported by citations, consistent with findings by Liu et al. (2023a). Our framework lets practitioners choose and customize models' behaviors at test time by adjusting such parameters without requiring additional training.
Efficiency and accuracy trade-off. Using our framework, practitioners can adjust how often retrieval occurs using the token probability of reward tokens. We evaluate how this adaptive threshold affects the overall accuracy and frequency of retrieval, and we evaluate the performance with varying numbers of threshold 8 (larger å results in less retrieval) on PubHealth and PopQA. Figure Be shows that the model's retrieval frequencies dramatically change on both datasets. as ô varies. On one hand, performance deterioration by retrieving less is smaller on PubHealth but larger in PopQA.
6 CONCLUSION
This work introduces SELF-RAG, a new framework to enhance the quality and factuality of LLMs through retrieval on demand and self-reflection. SELF-RAG trains an LM to learn to retrieve, generate, and critique text passages and its own generation by predicting the next tokens from its original vocabulary as well as newly added special tokens, called reflection tokens. SELF-RAG further enables the tailoring of LM behaviors at test time by leveraging reflection tokens. Our holistic evaluations on six tasks using multiple metrics demonstrate that SELF-RAS RELF-RAG significantly outperforms LLMs with more parameters or with conventional aeggenell-uggmeneed generation approaches.

---

Published as a conference paper at ICLR 2024
ETHICAL CONCERNS
This work aims to improve the factuality of LLM outputs, the lack of which continues to cause nu- merous real-world problems (e.g., spread of misinformation and provision of incorrect and dangerous advice). While our method shows significant improvements in terms of performance, factuality, and citation accuracy, it can still generate outputs that are not fully supported by the citations. We hope that explicit self-reflection and fine- tnee-graied attribution may help users verify factual errors in the model outputs.
ACKNOWLEDGMENTS
We thank Sewon Min, Scott Wen-tau Yih, Sean Welleck, and Kawin Ethayarajh for fruitful discussions in the early stages of this work. We thank Sewon Min, Joongwon (Daniel) Kim, and Sandy Kaplan for valuable feedback on the paper, and Tianyu Gao and Weijia Shi for their help on evaluations. Akari Asai is supported by the IBM Fellowship. We thank Stability AI for providing computing to train and evaluate the LMs this work, and Microsoft Accelerate Foundation Models Research Program for the access OpenAI APIs. This work was funded in part by the DARPA MCS program through NIWC Pacific LN66001-19-2-40311 NSF IIS-2044660, and gifts from A12.
REFERENCES
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. Learn- ing to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations, 2020. URL
[d=SJgVHkrYDH
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-bused language models and appli- cations. In Proceedings the 61 Annual Meeting the Association for Computational Linguisties (Tutorial), 2023a. URL https:: http:////caaattohooooooo
aal- http//2222..cc............a.ll.. /tt.s//ooo.ttpp////aattoooooooo. www.:lllooooo..gg2222e
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. Task-aware retrieval with instructions. In Findings of the Associ- ation for Computational Linguistics, 2023b. URL http::/:///aclaaaaoog......o
Findings acl. 225
Bernd Bohnet, Vinh Q Tran, Pat Verga, Roce Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:22 2022. pRtppstss://axtttt www.:xii..orgaaaaoa. 2 221222002720771
Lingjiao Chen, Matei Zaharia, and James Zou. How is chatgpt's behavior changing over time? arXiv preprint arXiv:2307.0900 0000000097900 2023. http: /ttp////////tt//////////////////
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 18X3v:1803.055: 05457, 2018. URL/https: //aa://///xvvv wwaxxivv.rrgeeee wwtp///g//aaaaeeeeee.eeeeeeeeeee www.a////..... 1803.05457 05457
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with ao-awareness. In Advances in Neural Information Processing Systems, 2022. httptttt///ppnniie
net/ forum? iid 1ee2iid--400 H4Dq ibmx
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. -hain-of-verificatiin reduces hallucination large language models. arXiv preprins
arXiv:2309.11499 11495, 2023, httLhttss://aaa/ wwtp:xvvvvrr...ggm abs 2309. 2309.11499 11495
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge- -owweeed conversational agents. In International Conference on Learning Representations, 2019. URL http:htttt///opereeeeee
net. wttt:/orommdddrr11111
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that
10

---

Published as a conference paper at ICLR 2024
learn from human feedback. arXiv preprint arXiv:2305. 2023. URL https:://aaaat orq/abs/ aaaaab//2 abs/2305. 14387
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen, Enabling large language models to generate text with citations. arXiv preprint 2rXiv:2305.14622 2023. UtL.Bttpp:://rxiv..oo//////
2305. 10e..1666 14627
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International Conference on Machine Learning, 2020. URL
https://dl acm. org/ doi 0555:/15555555555555555555 3111/35223333335533333 3525306
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sehastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. net/
Transactions on Machine Learning Research, 2022a. URL
forum?id=jKN1px. 7b0.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint 2.xii:2208.032999, 2022b. https!
//arxiv. org/ abs/ 2208 03299
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint arXiv:2305.00688 06983, 2023. (ttps::::// wttp///wwwoi.ooaaooooooooooeeere http:/abb/2300.003 06983
Mandar Joshi, Funsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), 2017. URL
attpss:/acclattholoryy.. wtt.:/yy..og//111111. 1147
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05835 05858, 2019. /tt.httts////rr//..orrgg///////
05858
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, 2023. /ttp:::s::/ /RLjitttp/:/pezzeeeee... net/
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019. htt.atttss/aclaattoloooooo org/
019-1026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. URL
wwg. 2t.bs//399 06180
Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensing nlp tasks. In Advances Neural Infor- mation Processing Systems, 2020. /ttpttep::/proceee ings....g. neurips.......... ccc cc/paper/ Paper.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Scott Yih. Ra-dit: Retrieval- augmented dual instruction tuning, 2023. URL httllttaa/aaxii..ooo
abs/ aabs/2310.000 01352
Nelson Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. arXiv preprint 230.22304009849 2023a. URL URL http://ss//axxii..rrg./ wtt.orgg//bb///////////// 098488

---

Published as a conference paper at ICLR 2024
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16634, 2023b.
htth:tts:// pttttp::/aaxxi..ooo /ttpxaaaxaiv../o/.. org/ abs/ / /2/32333..20 12.33316655 16634
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems, 2022. URL httpss///peerreiiiii EoreEooommnndd-Haaaaaa
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng. and James Glass. Sail: Search-augmented instruction learning. arXiv preprint
wtwpg/g/a...ag.ooeee arXiv:2305..5525, 2023. URL http:tttss/axi...otoo
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2023. URL ctttss://aclanttholll
acl-long.50 546
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young. Lucy
Geoffrey Irving, et al. Teaching language models support answers with verified quotes. arXiv preprint 1rXiv:2203.1114,,000 2022. URL 2 11147]
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing, 2018. URUhttts::/acllatthon
org/D18 1260
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. A discrete hard EM approach for weakly supervised question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP)) 2019. URL httpstts:: / http///claattoloo..ooooooo org/
D19 D19-1284
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv: 1425, 2023. URL https:
7/arxiv. org/ abs/ / 2305. 1055..1555 14251
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assister question-answeriig with human feedback. arXiv preprini 12Xiv:2112.09332, 2021. URL https
//axxiv.org/abs/222 09332
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma. Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. URL https:
emnlp-nnaeaaa.ee mai... 669
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.087777777 2023. URL brg/abs/ 08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. htts::::::/// http:://:////peeeeee.e... wwteeeewwwwwweeeeeeeeeeeeeeeeeee forumm?
id= TG8KACxEON
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jemnite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel,
12

---

Published as a conference paper at ICLR 2024
benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021. URL pattps://aclantthlooo org 2021. nwaccacccl..m...mmm -mainmiin 200
and Sebastian Riedel. KILT:
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Advances Neural Information Processing Systems, 2021. URL https //openreview net/ / forum?id Tqx 7nJPPPR
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations
toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking. Storage and Analysis, 2020. URL https: / ac
org/doi/10.. 5555/ /55////0505555533311111111 3433701 3433727
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Leyton-Brown. and
Yoav Shoham. In-context retrieval-augmentie language models. Transactions of the Association for Computational Linguistics, 2023, URL https: //axxiv org/
bong//bbat abs/ 2302 00083
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Araad Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. /ttpppeee///pee.eeess....eee http://wweeeetrrorroooo--//S-
Timo Schick, Jane Dwivedi-Yis Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04771, 2023. URL http:ttsss//xxx..rrrrrrrese// 04761
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017. URL https://arxiv.ort
abs 0t///177.066
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https //proceed ngs. ml m.r. r.ress w202 sh 23a.ii23....mm html
Ivan Stelmakh, Yi Luan,
Bhuwan Dhingra, and Ming- Mii Wei Chang. ASQA: Factoid questions meet long- form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. URL a acl wntholoog.oogggoooooooolooo 2022. ewnlp-nanll.e.eeee 566
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long Pupers), 2018. URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint NrxXiv:2307.09288 2023. URL https::t.... //arxiv
orq/ abs///bb/aa 2307 09288
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.0475.. 2023. URL http:: / rx v. org/ abs/ 04751
13

---

Published as a conference paper at ICLR 2024
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022. htt///pppp///peeerreee...ttt///s
id==gEZGGoozqq
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A
Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693, 2023. URL https:
//arxiv.ogg/abs/ 2206. 01693
Xiang Yue, Boshi Wang. Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311, 2023. URL https
//arxiv. .rg/ abs/ 2305.06311
Tianhua Zhang. Hongyin Luo, Yung-Sung Chuang, Wei Fang. Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Interpretable unified language checking. arXiv preprint arXiv:2304.03723 2023. URL httpt::tpa/xiii> wttp:orgg//a/oooooo
abs 03728
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1199999.8999 2019, httht / //trxiv..vv//a////aeeeeeeaawavva
08593
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A
SELF-RAG Details
16
Reflection Tokens
16
A.2
Advantages of Learning based Methods
16
A.3
SELF-RAG Training
17
A 4
SELF-RAG Inference
19
Experimental Details
B.
I More Details of Training
19
More Details of Evaluations
19
Results
20
Analysis
20
Human Evaluation
................................
21
Qualitative Examples
Full List of Instructions and Demonstrations for GPT-
21
15

---

Published as a conference paper at ICLR 2024
A SELF-RAG DETAILS
A.1 REFLECTION TOKENS
Definitions of reflection tokens. Below, provide detailed definition of reflection type and output tokens. The first three aspects will be provided at each segment level, while the final aspect is only given at each output level.
[Retrieve): ): Given an input and previous-step generation (if applicable), an LM determines whether the continuation requires factual grounding. No indicates retrieval is unnecessary as the sequence does not require factual grounding may not be enhanced by knowledge retrieval, Yes indicates retrieval is necessary. We additionally have continue to use evidence, which indicates that a model can continue to use the evidence retrieved previously. For instance, a passage may contain rich factual information, and thus SELF-RAC generates multiple segments based on the passage.
rectreevalonnddmaaee
Relevant [SREL ): Retrieved knowledge may not be always relevant to the input. This aspect
indicates whether the evidence provides useful information (Relevant) or not [Irrelevants
Supported ( [SSuP] Attribution is the concept of whether the output is fully supported by certain evidence (Menick et al., 2022, Bohnet et al. 20022.. This aspect judges how much infor- mation in the output is entailed by the evidence. We evaluate attributions in three scale, Fully supported rted Part ially supported, and No support / Cont radi radiaatooo....... follow- ing Yue et al. (2033) Nakano et (202T).
): Following the definitions from Liu et al. (023a)) we define the perceived utility as whether the response is a helpful and informative answer to the query, independently from whether it is in fact factual or not. This can be also viewed as plausibility in Menick et al. (2022). For usefulness, we use a five-scale evaluation (I is the lowest and 5 is the highest).
Useful
IsUss:
Details of GPT-4-based data collections. We use the instruction and demonstration pairs to prompt GPT-4, listed in Section D Following an official recommendation, we separate instructions and outputs with "##". We use the temperature 1 and set the maximum output token counts to be 200. We discard instances where GPT-4 does not follow the designated output formats or output sequences that do not match our expected category names, As result, we collected 1,2594 for Retrieve 11,181 for IsSUr 19,317 for relevance, and 3,831 for utility.
Manual analysis of the GPT-4 predictions. The authors of this paper manually assess randomly sampled 20 instances for each aspect and check if GPT-4 predictions match their assessments given the same instruction, demonstrations, and test instances. We found our assessments show high agreement with GPT-4 predictions, especially for relevance (95%), retrieval necessity (95%). and the degree of support (90%). Agreement was slightly lower in usefulness (80%), mostly due to the disagreement between and 2 or 4 and 5. Compared to prior efforts on agreement of GPT-4 predictions and human annotators in pair- evaluations, we found our human annotators often agree with GPT-4 predictions. We hypothesize this is because absolute scoring systems, unlike such relative, overall pair-wise evaluation systems enables GPT-4 to generate more reliable and agreeable predictions. The effectiveness of GPT-4 evaluations in fine-grained aspects has shown to be effective in prior work (Liu et al. 2023a).
fine-grained evaluation with
A.2 ADVANTAGES OF LEARNING-RASED METHODS
While recent work @Jing et al. 2023) proposes prompting- method enable retrieval on- demand, we find feedback and inference-time control. First, Self-RAG requires careful multi-aspect fine- une-grained self-evaluations at inference time. To make an LM to comprehend fine- fine-grained aspects and scoring systems, precise and detailed instructions, as well as few-shot demonstrations, are necessary. This significantly increases the input sequence length, resulting in higher costs and latency. Nevertheless, we briefly tried prompting- based approaches in our preliminary experiments and found is nontrivial. When combine all instructions for all aspects and feed them to the target pre-traineds LMs (GPT-3 davinci-003 / 002, Llama2- 13B-chat), all models struggle to precisely follow our evaluation scheme, often generating output formats that do not suit our scheme whose reflections less accurate. To
based method is more suitable to enable fine-grained self-reflection
16

---

Published as a conference paper at ICLR 2024
Dataset name
Category
Data source
of instances
of Retrieve
Yes
GPT-4 Alpaca
Instruction- foolooowng
Open
Open-Instruct Instruct
26,168
53.2
Stanford Alpaca
caarruction-followwee
Open-Instruce
25,153
48.0
FLAN-V2
Instruction-following
17,817
15.8
ShareGPT
fosllwiionnfollowwngg
Onsn-Inttruct
13,406
76.8
Open Assistant 1
coinuctionfollowwiinn
Open-Instruct
9,464
77.1
Wizard Wikipedia
Knowledge intensive
Knowledge-intensive
KILT
17,367
22.7
Natural Questions
Knowledge-inten intensive
KILT
15,535
87.7
FEVER
Knowledge-intensive
KILT
9,966
63.2
OpenBoookQA
Knowledge-intent intensive Knowledge-intenet
HF Dataset
4,699
2.3
Arc-Fasy
HF Dataset
2,147
11.0
ASQA
Knowledge intensive
-oowledge-intessiee
ASQA
3,897
91.5
Table 3: The generator LM M training data statistics.
make the most use of the Self-RAG potential, we need to use the token probabilities for the reflection tokens, which may not be always available for black box proprictary LM APIs. Note that at the time of submission, ChatGPT and GPT-4 do not support long probability information, preventing us from applying the Self-RAG algorithm to such models. This limitation also discussed in the Active Retrieval paper, which also requires access to token probabilities.
A.3 SELF-RAG TRAINING
Overview of training.
Algorithm Aloviis provides a high-level overview of our training.
Algorithm 2 SELF-RAG Training
1: Input input-output data D (X, Y), generator M, C
2: Initialize C with a pre-trained LM
3: Sample data {X Xxammple Ysample) (X,Y)
T'raining Critic LM (Section 3.2.1
4: for (I,z) v) E (X (Xsampple y ssamle do
Data collections for C
5: Prompt GPT-4 collect reflection token for (x, v)
6: Add (r))
} to Deritic
7: Update C with next token prediction loss
Critic learning: Eq.. Training Generator LM (Section Data collection for M with Deritic
8: Initialize M with a pre-trained LM
9: for (I, y) E (X, Y) do
10: Run C to predict given (z, y)
Add (ac, y, r) D Dyen
12: Update M on D gen with next token prediction loss
Generator LM learning; Eq.
Full list of seed datasets. To sample diverse input-output pairs, sample instances of the Open- Instruct (Wang et al.) (023) dataset. In particular, we use their ShareGPT, GPT-4 Alpaca, Alpaca, OpenAssistant, and FLAN subsets subsets. We also sample instances from couple of knowledge- intensive datasets, Natural Questions (Kwiatkowski et al. 2019). Wizard of Wikipedia (Dinan et al.) 2019) and FEVER (Thorne al., 2018) from the KILT benchmark (Petroni et al., 2021). ASQA (Stel- makh et al.) 2022) and multiple QA datasets including ARC-Easy and Opon (Mihayloy al. 2018). Table showw the full list of training instances, and in total, we use 145,619 instances. We also present the percentage of the instances where RRetiieee=Yes appears at least once. While some instruction-followee
datasets such as FLAN-T5 show a lower percentage of instances with Retriewe>=Yes other datasets show significantly higher percentages, indicating that our Critic model predicts the necessity of retrieval according to the given instances. FLANNTS, many training data come from non-knowledge-ii intensive tasks such grammatical error collections or simple string manipulations that are unlikely benefit from knowledge retrieval from Wikipedia.
Performance of the Critic C. We evaluate the accuracy of reward predictions by splitting GPT-4 generated feedback into training, development, and test sets. The accuracy of the reward model is follows. Table shows the model performance of predicting GPT-4 judgments. As you see,
17

---

Published as a conference paper at ICLR 2024
base LM
Retrieve
ISSCP
ISREL
ISUSE
Llama2-7B FLAN-3B
93.8
93.5
80.2
73.5
85.6
73.1
82.0
72.1
Figure 4: Reward prediction accuracy using GPT-4 predictions as ground-truth predictions.
overall our fine-tuned reward model shows high prediction matching with GPT-4 predicted feedback. While our final model uses Llama2-7B as a base LM,, we also train and compare FLAN-33 (Wei et al. 2022) model on the same data, to investigate the effectiveness of different data sizes affect final reward predictions. In most aspects, our reward model shows higher than 80% accuracy, indicating the powerful ability of fine-tuned specialized LMs to evaluate text. While both models show relatively lower performance on ISUSE] this is because both models often confuse between the two highest cases (5 and 4), where human annotators can also disagree.
Details of M data creation. Here, provide detailed data creation procedures. Algorithm summarizes the process. Here we set y/ to y for simplification. Once we train the critic model, we first it input data from the aforementioned datasets, to predict whether retrieval is needed not. For the instances where the critic predicts [Retrieve we only predict the ISUsE: given input and output. For the instances where the critic predicts Retrieve -Yes, the input and the entire output queries, to find passages that are relevant to the entire output. We then split output sentences using Spacy For each sentence, we run C to predict whether the retrieval is necessary not, given the input, preceding segments, and the initial retrieved passage. If predicts Noo then not insert any paragraph at the /th segment. If C predicts [Yer,, then we use the original input and the /th segment as a retrieval query to find relevant passages for the t-th segment. For each retrieved passage, we predict [SREL and ISSuP] If there is any passage and continuation with [SREL] =Relevant and [sSUe] =Fully Supported/ [sSuP] =Partial marriially Supported, then we sample it as the continuation, while we discard part of those examples on some cases (see details below). If there is more than one passage satisfying this criterion,
first retrieve passages using
the one with the highest retrieval score. there are only ISREL
=Irre -Irrelevant levant or ISSUP =No
Support rt passages, randomly sample passage.
To avoid the dominance of certain reflection tokens in training data, we down-sample training instances. In particular, down-sample and discard 50% of the instances without any retrieval tokens, since large-scale instruction-following datasets (e.g., Alpaca), include many queries that do not require retrieval (e.g., simple and easy facts not knowledge intensive). We also notice that in Open-domain QA, there are many relevant and fully supported passages, and when we always prioritize such cases, ISREL =Rel =Relevant and IsSuP =Fully Supported will be overly represented and there's s risk that a model learns to simply output the same reflection tokens. We, therefore, up-sample some instances with the ISREL
=I rrre evant token for the QA dataset.
Algorithm MMgenn Data creation
1: Input Input-output data D X for (x. (X,Y) do
2:
3: Given (x, y) predicts Retrieve
4: if' Retrieve] predicted then
5: Retrieve relevant passages D using R given (x,
> Retrieve passages
6: for d E do
7:
C predicts IsRKI.
for each d
D. Predict relevance of passages & Predict supports of outputs
8:
predicts
IsSuP
each (y,, d)
9: @predicts
ISUSE for each d
> Predict overall utility (t = T only)
Sample d
11: else if
Retrieve is not predicted then predicts
given v, y Add augmented (x. y, d, r) to D
IsUsK
gen
https://spacy.itt
18

---

Published as a conference paper at ICLR 2024
Training examples.
T several training examples used for M training.
A.4 SELF-RAG INFERENCE
Details of beam-search score calculations.
We first compute scores for each critique type by taking the normalized probabilities of desirable tokens. For [SREL] we compute the score follows:
p ISREL
RELEVANT)
ISREL
ISREL
RElEVANT) + p ISREL
IRRELEVANT)
For
ISSUP
we compute the score as follows:
ISSUP
FULLYY
ISSUP
PARTIALLY)
ISREL
+ 0.5
S
S
where S =
Σ EE
No) P ISSUP
t). For ISUSE where we have five-scale score,
compute the weighted sum of the scores. We assigns weighted scores of w {-1, 0.5, 0.5, 13
to the tokens
ISUSE
2,, 4, 5}, and compute the final scores as follows:
5
P
ISUSE
ISUSE
W
S
where S (1E(1.2,,45)) P
ISUSE
Details of adaptive retrieval. following condition is satisfied:
For retrieval based on soft constraints, we trigger retrieval if the
Retrieve
YES)
No) > 6.
pl
YES)
p(p(
Retrieve
B EXPERIMENTAL DETAILS
B. MORE DETAILS OF TRAINING
More details of training and computations. We use Nvidia A100 with 80GB memory train our models. All models are trained for 3 epochs with a batch size of 128, peak learning rate of 2e-5 with 3% warmup steps, and linear decay afterward. We set the maximum token length to be 2,048 for the 7B model, and 1, 1,524 for the 13B model due to the memory constraint. We use Deepspeed stage (Rajbhandari et al., (020) to conduct multi-GPU distributed training, with training precision 1 enabled. FlashAttention (Dao et al., (2022) is used to make the long-context training more efficient. We run inference of our trained models using 1-2 Quadro RTX 6000 GPUs with 24GB memory.
B.2 MORE DETAILS OF EVALUATIONS
Retrieval setup details. By default, we use Contriever-MS MARCO to retrieve the top five documents from Wikipedia, and official Wikipedia embeddings based 2018 English Wikipedia. On PopQA, where question and answer pairs are created based on WikiData in 2022, we found that the 2018 Wikipedia sometimes lacks articles about some entities that have been more recently added to Wikipedia. Therefore, for PopQA, we used the December 2020 preprocessed Wikipedia corpus provided by [Iacaar et al. (022bb and generated document embeddings | The issues of performance variance from different Wikipedia dumps have been reported by prior work (Asai et al. 2020 Izacard al., (002b). Yet, we observe limited effectiveness of such off- -he-hhh- retrieval models trained primarily on nnwwledge-intensiig tasks for open-ended generation (e.g., instruction following). Recent or concurrent work studies instruction-tuning
retrieval systems (Asai et al.) 2023b) joint training of retrieval and LM components (Lin et al. 2023). while we leave exploring the effectivess of such appraoches for future work. For bio generation and open-domain QA tasks, additionally retrieve five documents using Google Programmable Search and search documents from English Wikipedia. As this API only provides snippets, we retrieve Wikipedia introductory paragraphs for the corresponding entities.
19

---

Published as a conference paper at ICLR 2024
Detailed experimental settings for individual datasets. For OpenQA datasets, set the max- imum new token number to 100 tokens. For closed-set tasks (PubHealth and ARC-C), we set the maximum new token length to 50 for all baselines. For SELF-RAG inference on PubHealth and ARC-C, instead of determining the output with the highest score in other tasks, we aggregate the scores for each option and select the answer option with the highest score. We found in zero-shot settings of fact checking, some LLMs can generate capitalized class labels (c.g.. True) while our gold labels are lower-cased. Therefore, across different LMs, for fact checking, we lowercase the predictions. In multiple choice tasks, we found some models generate answers in slightly different ways (e.g.. . (A) instead of A). We slightly modify instructions for each LLM to avoid such format violations, and further conduct string matching between each candidate and model predictions if format violations still remain. After that processing, closed set tasks, model predictions match one of the gold classes in almost all cases. For AL.C, we found that 1lama2-chat tend to generate significantly lower outputs than other models (c.g., on average, their output is nearly 100 token, while ChatGPT generates 40 tokens average), resulting in inflated str-em scores. We limit the maximum generation length to 100 tokens for all baselines to avoid this issue, rather than the original 300 tokens in the ALCE paper. Consequently, all of the baseline output length is within 30-60 tokens. For FactScore, we set the maximum new token length to 500 for baselines and 200 for SELF-RAG at each segment level.
Task-specific instructions. Table shows the list of the instructions used during evaluations. For Open-domain QA, we do not provide explicit instructions.
C RESULTS
C.I ANALYSIS
Reliance on parametric- and non-parametric memories. We analyze how frequently model answers come from retrieved passages nion-parametric memories) or their parametric memories. On two open-domain QA datasets, TriviaQA and PopQA, we conduct the following analysis: I) sample query models successfully answer correctly, 2) for each query in this group, check whether the matched ground-truth answer is sub-string of the retrieved passage or not. We evaluate SELF-RAG 7B, Alpaca 7B, Alpaca 13B, and Llama2-Char 13B. We found that SELF-RAG significantly less frequently generates answers that are not included in the provided evidence; in particular, in Alpaca 30B, 20% of the correct predictions are not included in the provided passages, followed by Ilama2- chat 13B (18%) and Alpaca (15%), while it is only 2% in SEGF-RAG. When retrieved passages are not relevant, SELF-RAG generates [SREL] =Irrelevant, indicating that the following answers may not be factually grounded, while those instruction-tuned models continue to generate plausible answers.
Effects of training data size. We analyze how the data scale affects the model's performance. In particular, we randomly sample 5k, 10k, 20k, and 50k instances from our original 150k training instances, and fine-tune four SELF-RAG 78 variants performance on PopQA, PubHealth, and ASQA (citation precision) with our final SEL.F- AG trained on the full 150k instances. We also evaluate Figures 5a 5b]and 5c shows the models' performance trained on different amount of data. Across all datasets, increasing data size often shows upward trajectories and the improvements are significantly larger in PopQA and ASQA, while we do not observe such significant improvements on Llama2-FT; when increasing the training data from 50k 150k. These results also indicate that further expanding the training data of SELF-RAS -ARG may lead to further improvements, although in this work we limit our training data size to 150k.
those subsets. Then, we compare the model
Reflection token prediction performance. We evaluate the accuracy of the Critic and Generator LMs in predicting reflection tokens. For the Critic LM, we evaluate its agreement against GPT-4 predictions on a validation set of the initially collected GPT-4 predictions. Table shows the model performance of predicting GPT-4 judgments. As you can see, overall our fine-tuned reward model shows high prediction matching with GPT-4 predicted feedback. In most aspects, our reward model shows higher than 80% accuracy, indicating the powerful ability of fine-tuned specialized LMs to evaluate text. While both models show relatively lower performance on ISUSE] this is because both models often confuse between the two highest cases (5 and 4), where human annotators also disagree.
20

---

Published as a conference paper at ICLR 2024
Type
73
Perommance
Retrieve
93.8
72
93.5
40
IsREL
80.2
35
100
150
100
40
100
ISUSE
73.5
training
of training
training
(d) Reflection pre-
(a) PopQA
(b) PubHealth
(c) ASQA (prec)
diction accuracy.
Figure 5: Training scale and Human analysis: (a) (b) (c) Training scale analysis shows the effect of the training data scale on PopQA, PubHealth and ASQA (citation precision), respectively. (d) Reflection token prediction accuracy of the Critic LM.
C.2 HUMAN EVALUATION
Evaluations supportiveness and plausibility. We conduct small human evaluations on SELF- RaG outputs, as well as the reliability of predicted reflection tokens. particular, we sampled 50 samples from PopQA and Bio results. Following Menick et al. (2022), human annotators evaluate S&P, which indicates whether the model output plausible (i.e., the output is a reasonable and on-topic response to the question as if it were occurring in conversation) and supported (i.e., the provided evidence sufficient to verify the validity of the answer). For S&P, we do not consider the instances where SELF- RAG predicts irrelevant whether the model-predicted reflection tokens about [SREL and ISSUP match their inspections (e-g., whether the fully supported output is supported by the cited evidence). Human annotators find SELF-RAG answers are often plausible and supported by relevant passages with higher S&P scores on short-form PopQA, which is consistent with Menick al. (2022). Human annotators also find IsSUP reflection token predictions are mostly aligned with their assessments. Appendix Table 6 shows several annotated examples and explanations on assessments.
no support. We then ask our annotators
IsREL and
Examples of human evaluations. Table 6 shows examples with human evaluations on S&P and ISSUP reflection tokens.
correctness of ISREL and
C.3 QUALITATIVE EXAMPLES
Table 7 shows several examples predicted by our SELF-RAG (13B). The first example the model output to an ASQA question. The first reference states that Emperor Constantine made Sunday day of rest from labor, and further the second citation supports the fact that the official adoption of Sunday as a day of rest by Constantine in AD 321. In the second example. the model predicts Cont radi ctory to the first output as the output says the person has served as the CEO since 2010, while the passage says he stepped down as CEO in 2015. Indicating those factual contradictions as reflection tokens enables to enforcement hard control and also verification model outputs easily. In the third example, while the generation is mostly correct, SELF-RAG predicts Part i ally Support to the statement listing the name of the songs, as they were not explicitly mentioned.
D FULL LIST OF INSTRUCTIONS AND DEMONSTRATIONS FOR GPT-4
Here, we show the instructions and demonstrations used to prompt GPT-4 to collect reflection tokens. Table 8 shows the instructions and demonstrations for the initial retrieval token. Table g shows the instruction and demonstrations used to collect the three-way output tokens for |Retrieee given instruction, preceding sentences, and previously retrieved passages. Due the longer demonstration and test input, we only use a single demonstration. Table [0 shows an instruction and demonstrations used collect the three-way output tokens for [SREL strations used to collect the three-way output tokens for [SREL demonstrations used to collect the five- five-way output tokens for
Table II shows an instruction and demon-
Table [2 shows an instruction and
ISUSE
21

---

Published as a conference paper at ICLR 2024
Input Identify the two types of computer memory. (from gpt-4aa Output Yeetiiew>>eaa/<>SCommpp
memory Computer memory In computing, memory refers to the computer hardware integrated circuits that store information for immediate use in a computer, it is synonymous with the term "primary storage". Computer memory operates at high speed. for example, random-access memory (RAM), as a distinction from storage that provides slow-to-access information but offers higher capacities. needed, contents of the computer memory can he transferred to secondary storage; a very common way of doing this is through memory management technique called "'vrrtual memory". An archaic synonym for memory is store. The term "memory", meaning "primary storage" or "main memory", is<p>> [sREL =Relevant] Jariaaa memory, also known main memory, is the memory that the computer can directly access and manipulate. [ [[SUU =Fully Supported]
memory- offten associated with addressable semiconductor memory, i.e. integrated circuits consisting of silicon-based transistors, used for example as primary storage but also other purposes in computers and other digital electronic devices. There are main kinds of semiconductor memory, volatile and non-volatile. Examples of non-volatile memory are flash memory (used as secondary memory) and ROM, PROM, EPROM and EEPROM memory (used for storing firmware such as BIOS). Examples of volatile memory are primary storage, which is typically dynamic random-access memory (DRAM), and fast CPU cache memory, which is typically static random-access memory (SRAM) that is fast but energy energy-consuming, offering lower memory<pa <pr ISREL =Re levant: memory can be accessed quickly. but it is volatile, meaning that the data is lost if the power is turned off.
IsSup
=Partially Supported], ed]...] ISUSE :5]
Input Give example of a situation where you have to delegate tasks (from gpt 4 1..4..appaae 4.alpaca)
Output |Reerieee NNc |The manager realizes that there is an extensive list of tasks that need to be completed in [uccess.[[Retrive order to make the event success. [Retrieve #Yes] >>>>s>>>DDD>>> |<p>>< Melegatioo-eeegaatt bility or authority to another person (normally from a manager to subordinate) carry out specific activities. It is one of the core concepts management leadership. However, the person who delegated the work remains accountable for the outcome of the delegated work. Delegation empowers a subordinate to make decisions, i.e. it is shifting of decision decision-making authority from one organizational level to lower one. Delegation. if properly done, is not fabrication. The opposite of effective delegation is micromanagement, where manager provides too much =nput,<p>1 ISREL =Re
is the assignment of any responsi-
] evant includes tasks such as creating a guest list, designing and sending in- vitations, booking a venue, coordinating with vendors, planning the eventá program, and monitoring the budget. [sstr] i 1 supported]l. Retrieve NNo [So]eince it is not possible for the manager to handle everything alone, they must delegate tasks to their team members. Retrieee ITThe manager would then assign cach team member specific responsibilities based on their individual skillsets and strengths.
application design Subject Matter Experts: These are the business users, the IS professionals. and the outside experts that will be needed for a successful workshop. This group is the backbone of the meeting<p>1 ISREL =Irrelevant |For instance, the tcamá graphic designer might be tasked with designing the invitations, while the event coordinator would be in charge of planning the program and working with vendors... (Rets.l[[ |N delegating tasks, the manager can ensure that all necessary tasks are completed in a timely manner, while also empowering their team members to take on responsibilities and contribute to the success of the event. =5]
ISUSE
Input Synthesize poem based on this prompt:"The sum is setting" (from stan stanford.alpaca
Output [Reerieve]NNooTee stars begin twinkle bright the trees, scattered throughout the night: [Nttrieee-NoJTh ]Thh stars begin to twinkle bright the trees, scattered throughout the night [ They flicker like a silent dance, A peaceful moment its trance. [ (Retrieve]=No) They flicker like a silent dance, A peaceful moment in its trance. [Nc A chorus of crickets sing a lullaby, To every ereature that roams
the sky IsUSE: =5]
Input when did walking dead season come out (from nq)
Walking Dead (season 7)-The Walking Dead (season 7) The seventh season of "The Walking Dead", an American post- ppocaapooi horror television series on AMC, premiered on October 23, 2016, and concluded on April 2, 2017, consisting of 16 episodes. Developed for television by Frank Darabont, the series is based on the eponymous series of comic books by Robert Kirkman, Tony Moore, and Charlie Adlard.
Output Yest
... <.<pp>[
ISREL
=Re 1 aevant 23, 2016
ISSUP
=Pul 1y Supported]]
IsUsE =5]
Input Given a chat history separated by new lines, generates an informative, knowledgeable and engaging response: Hello, I love German shepherd dogs and you? My favorite dog. they are officially known as German Shepherd Dogs?- Yes, have German shepherd in my house and she is pregnant (from wow)
Output hey also arc casily trained, obedient, and intelligent| [ealligent[[Reereewee=:= =Yess <<p>anmmnn Shepherd - a simple difference of opinion, it the fundamental issue of the breeds essential conformation and movement. ... They are known for being casy to train and good for performing tasks and following instructions. They are especially well known for their police work, being used for tracking criminals, patrolling troubled areas and detecting and =Relevant eRelevanttBBccusee of this, they are often used for disability assistance and for police work. [ ISSUP =Partially Supported]]
holding suspects. <p>[
ISREL.
IsUSE =2]
Table 4: List of the training examples.
22

---

Published as a conference paper at ICLR 2024
Dataset
Instruction
ARC-C
Given four answer candidates, A, B. C and D, choose the best answer choice. Please answer with the capitalized alphahet only, without adding any extra phrase or period. the following statement correct or not? Say true if it's correct, otherwise, say false. Don't capitalize add periods, just say "true" or "false".
PubHealth
Bio Generation
Tell me bio about [Person Name]
ASQA (baseline)
Instruction: Write an accurate, engaging. and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use unbiased and journalistic tone. Always cite for any factual claim. When citing several search results, use [[2[3]) Cite least document and most three documents in each sentence. If multiple documents support sentence, only a minimum sufficient subset of the documents.
ASQA (ours)
Answer following question. The question may ambiguous and have multiple correct answers, and in that case, you have to provide a long-form answer including all correct answers.
Table Full list of instructions used during zero-shot evaluations. For open-domain QA, we don't use any task specific instruction and simply use the original questions as input query.
23

---

Published as a conference paper at ICLR 2024
Imput What genre is A Question and Answer Guide to Astronomy? (FopQA)
Output [ |Retrieve =Yes] <p> Reference []<p>[ IsRei. =Relevanee A Question and Answer Guide As-
tronomy is a non-fiction book that falls under the genre of Astronomy and Cosmology.
ISSUP =Fu1 1y
Supported][ ISUSE =5]
Reference [1] A Question and Answer Guide to Astronomy is book about astronomy and cosmology and is intended for a general audience. The book was written by Pierre- Yves Bely, Carol Christian, and Jean-Rent Roy, and published in English by Cambridge University Press in 2010. It was originally written in French. The content within book written using a question and answer format. contains some 250 questions, which The Science Teacher states each are answered with a concise and well-formulated essay that is informative and readable. The Science Teacher review goes on to state that many of the answers given in the book are little gems of science writing. The Science Teacher summarizes by stating that each question is likely to be thought of by student and that the answers informative, well-constructed, and thorough. The book covers information about the planets, the Earth, the Universe, practical astronomy, history, and awkward questions such as astronomy in the Bible, UFOs, and aliens. Also covered are subjects such as the Big Bang, comprehension of large numbers, and the Moon illusion.
S&P Supported and Plausible.
ISSUP are correct, as the Reference [1] provides detailed information of the book including the genre, and the model output is fully supported by the reference.
Correctness of ISREL and ISSUP]: : Both IsREL
and
Input Who is the author of The Lic? (PopΩA)
Output Retrieve- =Yes] ro>Reference []]<p>[
ISREL =Relevant] J The author of The Lie is Sam Harri..
1SSUP
=Fully Supported] ISUSE =5]
Reference [1] Lying is a 2011 long- form essay book by American author and neuroscience expert Sam Harris.
Harris argues that can radically simplify our lives and improve society by merely telling the truth in situations
where others often lic.
S&P Plausible not supported.
Correctness of [SREL and ISSUP Both
ISREL
and
IsSUe are incorrect, the Reference [1] is about
different book whose title similar to The Lie.
Input Tell me a bio about Ronaldinho (Bio)
Output [ Beetrieve--Ves <p> <p>Reference []]<p>[ ISREL
=Relevant] Ronaldinho is a retired professional
footballer from Brazil.[ [sSuP =Fully Supported CCont inu]HHe played
attacking
midfielder and was known for his technical skills, creativity, dribhling ability, and accuracy from free
kicks|
inueel won two FIFA World Player the Year awards and a Ballon d' d'Or.[ [sUSE] =5]
Reference [1] Ronaldo Assis Moreira (born March 1980), commonly known as Ronaldinho Gaúcho or simply Ronaldinho, is a Brazilian retired professional footballer who played mostly as an attacking midfielder, but was also deployed as winger. Considered one of the best players of his generation and regarded by many as one of the greatest of all time, Ronaldinho won two FIFA World Player of the Year awards and Ballon d' OrOr. A glohal icon of the sport, he was renowned for his technical skills, creativity, dribhling ability and accuracy from free-kicks, as well as his use of tricks, feints, no-look passes and overhead...
S&P Plausible and fully supported.
Correctness of ISREL and
ISSUP Both
ISREL and
ISSUP are correct.
Input Tell me a bio about Sergei Bodrov (Bio)
Output || S <p>Reference [1]<>>[ ISREL
=Relevant] Sergei Vladimirovich Bodrov
(born June 28, 1948) is a Russian film director, screenwriter, and producer: ISSUP
=Fully
[npported]][Retiicn Coont irrontinm He was homn in Moscow, Russia. Retrieve iContinue]]eet He the presi- dent of the Russian Guild of Film Producers. [ISUSE] =5]
Reference [1] Sergei Vladimirovich Bodrov (born June 28, 1948) is a Russian film director, screenwriter, and producer. In 2003 he was the President of the Jury at the 25th Moscow International Film Festival.
S&P Plausible but only partially supported. The the Reference [1] does not explicitly mention the person is born in Moscow and there is no description about the Russian Guild of Film Producers.
Correctness of ISREL and IsSUP
ISREL
and
ISSUP are correct as the reference document is highly relevant
and the first sentence is fully supported.
Table 6: Examples of human evaluation results.
24

---

Published as a conference paper at ICLR 2024
Input When did sunday hecame the seventh day of the week (asqa)
Output [ [eetrreee <p> Reference [[]<p> ISREL =Rolevant] In 321 AD, the Roman Emperor Con stantine the Great made Sunday day of rest from labor, stating: "On the venerable Day of the Sun let the magistrates and people residing in cities rest, and let all workshops he closed" [sStP] =Fully Supported]
Retrrieee =Yes] <p>RRefreent Reference [2]<p>[ ISREL. =Relevant] The official adoption of Sunday as a day of rest by Constantine the Great in AD 321 made Sunday the seventh day of the week.[2][
ISSEP =Fully
supported][ ISUSE] =5]
Reference [1] Emperor Constantine then enacted the first Sunday Laws, for "the venerable Day of the Sun" in 321 A.D. On March 321, the Roman emperor Constantine issued decree making Sunday day of rest from labor, stating: We find good description of this sources such as the book Rest Days which states: Early Christian observance of both the spiritual seventh-day sabbath and a Lords Day assembly is evidenced in Ignatiuss letter to the Magnesians ca. 1 10.
Reference [2] Despite the official adoption of Sunday as a day of rest by Constantine, the seven-day week and the nundial cycle continued to be used side-by-sid until at least the Calendar of 354 and probably later. The fact that the canon had to be issued all is an indication that adoption of Constantine's decree of 321 was still not universal... ...
Input Tell me a bio about Rory Tapner. (bio generation)
Output <p> Re-Reeernn [1]<>>| IsRai =kel evant] | Rory Tapner is British businessman who has served as the Chief Executive Officer of Coutts. private banking and wealth management company, since 2010.[ [SSUP] #Contradictozyy) Reference [2]<p>| ISREL =Relevant] Rory Tapner born on 30 September 1959
in London, England.[ ISSUP
=Fully Supported]
Reference [1] Rory Tapner Rory Tapner a UK businessman who, September 2010 was appointed CEO of Coutts, the Wealth Management division of Royal Bank of Scotland Group. In February 2015 he stepped down as CEO when he was replaced in the role by Alison Rose
Reference [2] Holdings (from August 2016); and the Council of the University of Buckingham (from July 2017). Rory Tapner Rory Tapner (born 30 September 1959) is a UK businessman...
Input Tell me a bio about G. Venugopa. (bio generation) Output (ranked 1) [[eerieve]=Yes] <p>Reference [1]<p>[ ISREL
=Relevants G. Venugopal is popu- lar playback singer in the Malayalam film industry. Retrleve Cont Contiinuel He has sung over 300 songs in Malayalam, Tamil, and Hindi movies.[ [SSUP] =Partially Supported] [[etrieve]Yee]]] <p>Reference [2]<p>G. Venugopal was born on December 10, 1960 in India. IsSup =Fully Supported]
Reference [1] G. Venugopal G Venugopal (born 10 December 1960) is an Indian playback singer known for his work in Malayalam films. He started his singing career in the film "Odaruthammaava Aalariyaam" (1984). Since then he has sung in more than 300 films: and has over 500 private albums to his credit. The many awards has won include the coveted Best Playback Singer (Kerala State) which he won thrice, the Film Critics award and the Kerala Kaumudi Gallop Poll award (twice). Hc won The Kerala State award for Best Playback singer (2004) third time for the song
from the film "Ullam" Venugopal"
Reference [2] Kerala State Film Awards: Kerala Film Critics Award Kerala State Government Award (Profes- sional Drama): Asianet Film Awards: Kerala Kaumadi Gallup Poll Award: Academic G. Venugopal G Venugopal (born 10 December 1960) an Indian playback singer known for his work Malayalam films.
Table 7: Examples of outputs.
25

---

Published as a conference paper at ICLR 2024
Instructions
Given an instruction, please make a judgment on whether finding some external documents from the web (e.g., Wikipedia) helps to generate better response. Please answer [Yes] or [No] and write an explanation.
Demonstrations
Instruction Give three tips for staying healthy. Need retrieval? [Yes]
Explanation There might be some online sources listing three tips for staying healthy or some reliable sources to explain the effects of different behaviors health. So retrieving documents is helpful to improve the response to this query.
Instruction Describe time when you had to make difficult decision. Need retrieval? [No]
Explanation This instruction is asking about some personal experience and thus it does not require one to find some external documents.
Instruction Write a short story third person narration about protagonist who has to make an important career decision. Need retrieval? [No]
Explanation This instruction asks to write short story, which does not require external evidence to verify.
Instruction What the capital of France? Need retrieval? [Yes] Explanation While the instruction simply asks us to answer the capital of France, which is widely known fact, retrieving web documents for this question can still help.
Instruction Find the area circle given its radius. Radius Need retrieval? [No]
Explanation This math question and although we may be able to find some documents
describing formula, it is unlikely to find document exactly mentioning the answer.
Instruction Arrange the words in the given sentence form grammatically cor- rect sentence. quickly the brown fox jumped Need retrieval? [No] Explanation This task doesn't i require any external evidence, as it is a simple grammatical question.
Instruction Explain the process of cellular respiration in plants. Need retrieval? Yes]]
Explanation This instruction asks for a detailed description scientific concept, and is highly likely that we can find reliable and usefuu document support the response.
Table Instructions and demonstrations for Retrieve aspect given the input only.
26

---

Published as a conference paper at ICLR 2024
Instructions
You will be provided with an instruction, evidence, output sentence, and preceding sentences (optional). If the preceding sentence is given, the output should be the sentence that follows those preceding sentences. Your task is to determine whether the information in the output sentence can fully verified by the evidence if it requires further external verification. There are three cases:
If the output sentence can be verified solely with the evidence, then respond with [Continue to Use Evidence].
the sentence doesn't 't require any factual verification (c.g., subjective sentence or sentence about common sense), then respond with [No Retrieval].
If additional information is needed to verify the output sentence, respond with [Retrieval]. Please provide explanations for your judgments.
Instruction Explain the use of word embeddings in Natural Language Processing. Preceding sentences Word embeddings one of the most powerful tools available for Natural Language Processing (NLP). They are mathematical representations of words or phrases in vector space, allowing similarities between words and the context in which they are used to be measured.
Evidence: Word embedding
Word embedding is the collective name for set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary mapped to vectors of real numbers. Conceptually it involves mathematical embedding from space with one dimension per word to a continuous vector space with much lower dimension. Output: Word embeddings are useful for tasks such sentiment analysis, text classification, predicting the next word in a sequence, and understanding synonyms and analogies.
Rating [Retrieval]
Explanation The output discusses the applications of word embeddings, while the evidence only discusses the definitions word embeddings and how they work. Therefore, we need to retrieve other evidence verify whether the output is correct or not.
Table Instructions and demonstrations for Retrieve aspect given the input, preceding generations, and retrieved passages.
27

---

Published as a conference paper at ICLR 2024
Instructions
You'll 'll be provided with instruction, along with evidence and possibly some preceding sentences. When there are preceding sentences, your focus should be on the sentence that comes after them. Your job is to determine if the evidence is relevant to the initial instruction and the preceding context, and provides useful information to complete the task described in the instruction. If the evidence meets this requirement, respond with [Relevant]; otherwise, generate [Irrelevant].
Instruction Given four answer options, A, B, and D, choose the best answer. Input Earth's rotating causes
A: the cycling of AM and PM
B: the creation of volcanic eruptions
C: the cycling of the tides
D: the creation of gravity
Evidence Rotation causes day-night cycle which also creates a corresponding cycle of temperature and humidity creates corresponding cycle of temperature and humidity. Sea level rises and falls twice day as the earth rotates. Rating [Relevant]
Explanation The evidence explicitly mentions that the rotation causes a day-night cycle, as described in the answer option A.
Instruction age run for US House of Representatives
Evidence The Constitution sets three qualifications for service in the U.S. Senate: age (at least thirty years of age); U.S. citizenship least nine years); and residency in the state senator represents at the time of election.
Rating [Irrelevant]
Explanation The evidence only discusses the ages to run for the US Senate, not for the House of Representatives.
Table 10: Instructions and demonstrations for ISREL aspect given the input only.
28

---

Published as a conference paper at ICLR 2024
Instructions
You will receive an instruction, evidence, and output, and optional preceding sentences. If the preceding sentence is given, the output should be the sentence that follows those preceding sentences. Your task is to evaluate if the output is fully supported by the information provided in the evidence.
Use the following entailment scale to generate score:
[Fully supported]- All information in output supported by the evidence, or extractions from the evidence. This is only applicable when the output and part of the evidence almost identical.
[Partially supported] The output is supported by the evidence to extent, but there major information the output that not discussed in the evidence. For example, an instruction asks about concepts and the evidence only discusses either of them, it should be considered a [Partially supported].
[No support Contradictory] - The output completely ignores evidence, is unrelated the evidence, or contradicts the evidence. This can also happen if the evidence is irrelevant to the instruction.
Make sure to not use any external information/knowlldge to judge whether the out- is true not. Only check whether the output is supported by the evidence, and not whether the output follows the instructions or not.
Instruction Explain the use of word embeddings in Natural Language Processing. Preceding sentences Word embeddings are one of the most powerful tools available for Natural Language Processing (NLP). They are mathematical representations of words or phrases in a vector space, allowing similarities between words and the context in which they used to be measured.
Output Word embeddings are useful for tasks such as sentiment analysis, text classification, predicting the next word in sequence, and understanding synonyms and analogies. Evidence Word embedding
Word embedding the collective name for set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped vectors of real numbers. Conceptually involves mathematical embedding from space with one dimension per word to a continuous vector space with much lower dimension. Methods to generate this mapping include neural networks, dimensionality reduction on the word co-occurrence matrix, probabilistic models, explainable knowledge base method, and explicit representation in terms the context which words appear. Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance NLP tasks such as syntactic parsing, sentiment analysis, next token predictions well and analogy detection. Score [Fully supported]
Explanation The output sentence discusses the application of word embeddings, and the evidence mentions all of the applications syntactic parsing, sentiment analysis, next token predictions as well as analogy detection the applications. Therefore, the score should be [Fully supported].
Table 11: Instructions and demonstrations for tokens.
29

---

Published as a conference paper at ICLR 2024
Instructions
Given an instruction and an output, rate whether the response appears to be helpful and informative answer to the query, from 1 (lowest) - (highest). We call this score perceived utility. The detailed criterion is as follows: 5: The response provides complete, highly detailed. and informative response to the query, fully satisfying the information needs. 4: The response mostly fulfills the need in the query, while there can be some minor improvements such discussing more detailed information, having better structure of the response, im- proving coherence. 3: The response acceptable, but some major additions or improvements needed to satisfy users' needs. 2: The response still addresses the main request, but is not complete or not relevant to the query. 1: The response is barely on-topic or completely irrelevant.
Instruction Who is the current prime minister of the UK as of 2023? Output Boris Johnson was the prime minister of the UK from 2019 2022.
Perceived utility 2
Explanation While the output provides factually correct statement about the UK prime minister from 2019 2022, this instruction asks who the prime minister is of 2023, doesn't answer the instruction. Therefore, the utility 2.
Instruction Given a description of a travel destination, recommend 10 tourist attrac- tions with detailed descriptions of each. The travel destination Tokyo, Japan. Output Tokyo is a vibrant city full of exciting tourist attractions. Some must-see sights include the Tokyo Skytree, Tokyo Disneyland, Sensoji Temple, Meiji Shrine, Tsukiji Fish Market, Harajuku, and Shinjuku Gyoen. Perceived utility
Explanation This output doesn't 't provide descriptions of each attraction and the number of the attractions is also less than 10. While this output partially answers the instructions, doesn't 't match the instructions strictly.
Table 12: Instructions and demonstrations for ISUSE
tokens.
30


Reviews:
[{"review_id": "YY8sNoSAvQ", "reviewer": "Reviewer_quEF", "summary": "The authors propose a new framework called self-reflective retrieval augment generation (Self-RAG) to improve upon the vanilla RAG approach that always incorporates a fixed number of retrieved passages. They first train a LM with an extended vocabulary that includes reflection tokens. These tokens, along with a reflective retrieval/decoding algorithm, are then used at inference time to generate responses to the queries that are better informed by relevant passages.\n\nTo this end, they first generate data for and train a critic model, whose role is to evaluate and generate data for the generation model. The data for doing so is created using GPT-4 via specifically designed prompts. The reason being that using GPT-4 everywhere would be very costly and not reproducible. This model also provides the (usually human generated) signal for instruction tuning the generator.\n\nFor evaluation, the model is compared against 3 set of datasets (closed-set for fact verification, short form generation from open-domain QA datasets, and long form generation). The model is then compared with a variety of open source models, as well as cloud-based models, such as ChatGPT and Perplexity.", "questions": "- The authors claim in the abstract that \"Self-RAG outperforms ChatGPT ... on Open-domain QA ... tasks\". However, the results in Table 2 are dominated by ChatGPT and Ret-ChatGPT. They only get outperformed in PopQA. Given that, I'm surprised that the authors would make this claim\n\n- The authors claim that \"GPT-4 reflection token predictions show high agreement with human evaluations\". I find that hard to believe, given personal experience and reported results, e.g. PandaLM which for their task reports an F1 accuracy of 66% when comparing the quality of different passages. I don't deny that GPT-4 evaluation predictions have a strong positive bias towards the truth, but I still would call it high agreement. Do you have data to support this? Or do you think the gap in observations is due to comparison on slightly different evaluation tasks?\n\n- The repetitive self-reflection methods of Self-RAG will likely have an outsized effect on the computation requirements and the latency of the model. As such, I would love to see some numbers and comparisons here. Specially because it's known that given more time/tokens/compute, LLMs can improve their results.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The RAG method, which the authors set out to improve, has become highly used in the industry while having obvious limitations. This is a hard problem and any improvements on this problem will have great significance\n- The framework tries to integrate two very useful LLM approaches, RAG and self-reflection in an optimized manner", "weaknesses": "- Even though there are a lot of benchmarks and ablations, I still find that many of my questions are not answered by these evaluations\n    - Specifically, I'd like to separate out the contribution from the main two parts of the approach, the self-reflection, and RAG. I don't see much ablation or comparison on the self-reflection side. Only the `IsSup` token is ablated and that it.\n    - The RAG also has many details that I'm not certain have been compared against, other than simply changing the underlying model, which is not that interesting IMO. I believe simple changes to prompt, number of topk to retrieve, and order of retrieval (top first or top last) can produce highly nuanced results.\n    - I would have also liked to see at least an attempt at trying to combine self reflection with RAG without any pre-training\n- I find some of the numbers reported not convincing and in need of more investigation\n    - Ret-ChatGPT in Table 2: in all but the Long-form generation tasks, ChatGPT performs better than the retrieval-augmented version. This is curious and tells me either the model has seen the data (which makes the dataset slightly not a good representative of the RAG task) or that RAG is not being done correctly\n    - I have a similar (but less important) observation for the Alpaca models (specifically 13B). I can understand the reduced MAUVE score, but accuracy on PubMed has also been reduced which tells a similar story\n    - Same is true in Figure 3.a (ablations) where Self-RAG without retrieval is already better than all the other baselines in Table 2. Is this because there has been a data leak? If yes, that completely invalidates these results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nt4aOkdwey", "reviewer": "Reviewer_3ScW", "summary": "This paper introduces a novel framework known as Self-Reflective Retrieval-augmented Generation (SELF-RAG), designed to enhance the generation quality and factuality of Large Language Models (LLMs). During next token prediction, the SELF-RAG framework enables the decoding of reflection tokens from the LLM, allowing for control over the retrieval and self-reflection processes. To create training data for the reflection tokens, a critic model is trained using data generated by GPT-4. The authors also demonstrate how this framework can be used to control retrieval frequencies and guide generation towards specific critique types. Experimental results on multiple benchmarks show that the proposed SELF-RAG performs the best among non-proprietary LLMs on almost all tasks.  The ablation study shows the importance of each component.", "questions": "- Did the authors investigate the influence of reflection token distributions on the performance of the critic model? \n- Is there a possibility that the critic model may exhibit certain biases? Additionally, have they examined the distribution of retrieval tokens and its correlation with the retrieval threshold?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is novel in a way that it integrates the critic model information at training time so that the LLMs could reuse the output signals to guide the next step.\n- The proposed method can be easily adapted to generate responses with certain properties.\n- SELF-RAG significantly outperforms baselines in most cases.", "weaknesses": "- The critic model plays a pivotal role within the framework, and the authors have reported its accuracy in the Appendix. However, the paper does not include an evaluation of the LLM's accuracy in predicting reflection tokens.\n- The retrieval threshold is predetermined, but the authors have not provided an analysis of how variations in the retrieval threshold might affect downstream task performance.\n- In the ablation study, the paper only investigates 'Retrieve top1,' while SELF-RAG utilizes top 5 or top 10. Furthermore, the study exclusively focuses on 'Remove[IsSup],' neglecting an examination of the other criticize tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FtbG62823x", "reviewer": "Reviewer_4MCy", "summary": "This paper proposes a new framework, named Self-RAG, for training LMs to retrieve passages, generate text, and evaluate its own generation on-demand. Specifically, the proposed framework first uses a critique model C, distilled from prompted GPT4, to offline augment existing instruction-finetuning data with control tokens (retrieval token or critique tokens) as well as retrieved passages. Then the generator is trained on the augmented data, so that it also generates the target response as well as control tokens. At inference time, the decoding algorithm is modified to take actions when a control token is decoded. The authors implemented the framework on top of llama2 7B and 13B, and evaluated the models across various text generation tasks. Results show that the model can significant improve llama2's factuality.", "questions": "- How does Self-RAG's training affect the model in terms of aspects other than factuality?  e.g., instruction following, creativity, or reasoning.  \n- How good is the model at deciding when to retrieve? It would be nice to show the triggering rates of \"retrieval\" token on different types of tasks. \n- How much training data is needed? Does the training data needs to be carefully sampled, e.g., focusing on fact-seeking slices?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and nicely presented.\n- The paper proposes a novel framework that enables dynamic, un-demand retrieval and self-reflection in LM decoding. It address a key limitation in most existing RAG framework, that the model does not know when to retrieve. \n- The proposed method provides an interesting alternative to RLHF for using critique models. I really like its controllability and interpretability that RLHF doesn't have. \n- Experiments and ablations are sound and convincing.", "weaknesses": "- There are some related search that prompts the model to decide when retrieval is needed and how to use retrieval during decoding. E.g., \"Active retrieval augmented generation.\" by Jiang et al [1]. It would be nice to study what benefits the offline data augmentation & training approach can bring comparing to these prompting-based frameworks. \n- Since most experiments uses automated evaluation focusing on accuracy and factuality, it is still unclear to me if such a training approach could hurt some aspects of the model, e.g., instruction following, creativity, or reasoning.  \n- One limitation of the proposed Self-RAG is that the model sees each passage independent from the others. The generator cannot synthesize multiple passages. \n\n[1] Jiang, Zhengbao, et al. \"Active retrieval augmented generation.\" arXiv preprint arXiv:2305.06983 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qptNCoCYSC", "reviewer": "Reviewer_uRop", "summary": "The paper studies retrieval-augmented generation, where it aims to train a language model (LM) that learns to retrieve external documents on demand and generate a better response. Especially, the model learns to output some reflection tokens that serve different purposes. First, the model would generate a retrieval token to indicate that the current continuation needs external documents. Then with each of the retrieved documents, the model would generate a special token to indicate whether the document is relevant. After generating some text, the model would further generate the critique tokens to indicate whether the generated text is grounded by the document and whether it is helpful for the overall generation. These tokens would then enable controllable text generation during inference. Experiments show that the proposed model outperforms several other competitive baselines that are augmented with a retrieval mechanism but do not have the self-reflection step.", "questions": "1.\tIn section 3.2.2, the authors mention that during the training time, they mask out the retrieved text chunks. What is the purpose of this masking step?\n2.\tOther than just providing post hoc feedback on how good the current generation is, do the reflection tokens also affect/guide the generation process somehow?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tThe proposed method addresses two key problems including relevance and grounding by simply adding special tokens learned through fine-tuning.\n2.\tThe fine-tuning does not rely heavily on human annotation. Rather, it makes use of GPT-4 to provide the training data.\n3.\tThe reflection tokens allow users to have more control over the generation process to customize the expected response.", "weaknesses": "1.\tThe reflection tokens might be useful to select more promising generations during the decoding time. But it seems that they do not affect (or guide) the generation process from the beginning and might not help if none of the candidates is good. \n2.\tIf GPT-4 somehow can decently provide the labeled data required for each reflection step, it seems intuitive to just instruct GPT-4 to obtain the ideal response that is grounded by the input. Also, GPT-4 might make the decisions on whether to retrieve or not differently from the small LMs as GPT-4 memorizes a lot more world knowledge. Therefore, the annotation given by GPT-4 might not be suitable for small LMs. It would be great to have some discussion or clarification on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)