Active Retrieval Augmented Generation
Authors: Zhengbao Jiang
Frank F. Xu
Luyu Gao
Zhiqing Sun
Qian Liu
Jane Dwivedi-Yu
Yiming Yang
Jamie Callan
Graham Neubig
Venue: EMNLP 2023 Main (2023)
Keywords: Retrieval Augmented Language Model
Long-form Generation
Active Retrieval
TL;DR: Large language models (LLMs) tend to hallucinate, especially when generating long outputs. We present active retrieval augmented generation, in which an LLM actively decides when and what to retrieve throughout the generation process.

Abstract: Despite the remarkable ability of large language models (LMs) to comprehend and generate language, they have a tendency to hallucinate and create factually inaccurate output. Augmenting LMs by retrieving information from external knowledge resources is one promising solution. Most existing retrieval augmented LMs employ a retrieve-and-generate setup that only retrieves information once based on the input. This is limiting, however, in more general scenarios involving generation of long texts, where continually gathering information throughout generation is essential. In this work, we provide a generalized view of active retrieval augmented generation, methods that actively decide when and what to retrieve across the course of the generation. We propose Forward-Looking Active REtrieval augmented generation (FLARE), a generic method which iteratively uses a prediction of the upcoming sentence to anticipate future content, which is then utilized as a query to retrieve relevant documents to regenerate the sentence if it contains low-confidence tokens. We test FLARE along with baselines comprehensively over 4 long-form knowledge-intensive generation tasks/datasets. FLARE achieves superior or competitive performance on all tasks, demonstrating the effectiveness of our method.

Active Retrieval Augmented Generation
Zhengbao Jiang
Frank F. Xu' Luyu Gao Zhiqing Sun Qian Liu
Jane Dwivedi- Yu'
Yiming Yang' Jamie Callan' Graham Neubig'
Languagee Technologies Institute, Carnegie Mellon University
Sea Al Lab SFAIR, Meta
(zhengbaj, fangzhex, luyug, gnnggingg,gneubigeee cmu edu
Abstract
hallucinate and create imaginary content (Maynez et al., 2020; Zhou al., 2021). Augmenting LMS with retrieval components that look up relevant in- formation from external knowledge resources is a promising direction to address hallucination (Khan- delwal et al., 2020; Izacard et al., 2022).
Despite the remarkable ability of large lan- guage models (LMs) to comprehend and gen- erate language, they have tendency hal- lucinate and create factually inaccurate out- put. Augmenting LMs by retrieving informa- tion from external knowledge resources is one promising solution. Most existing retrieval aug- mented LMs employ retrieve-and -ngeeerrte setup that only retrieves information once hased on the input. This is limiting. however, in general scenarios involving generation of long texts, where continually gathering in- formation throughout generation essential. In this work, provide generalized view of ac- tive retrieval augmented generation, methods that actively decide when and what to retrieve across the course of the generation. We propose Forward-Looking Looking Active REtrieval augmented generation (FLARE), a generic method which iteratively uses prediction of the upcoming sentence to anticipate future content, which is then utilized query retrieve relevant doc- uments to regenerate the sentence if it contains low-confidence tokens. test FLARE along with baselines comprehensively over 4 long- form knowledge-intensive generation tasks/- datasets. FLARE achieves superior or compet- itive performance all tasks, demonstrating the effectiveness our method."
Retrieval augmented LMs commonly use a retrieve-and generrateeeeeeeeee setup where they retrieve doc- uments based the user's input, and then generate complete answer conditioning on the retrieved documents (Chen et al., 2017: Guu et al., 2020; Lewis al., 2020; Izacard and Grave, 2021; Sachan et al., 2021; Lee et al., 2021; Jiang et al., 2022; Izacard et al., 2022; Nakano et al., 2021; Qian et al., 2023; Lazaridou et al., 2022; Shi et al., 2023), These single-time retrieval augmented LMs outper- form purely parametric LMs, particularly for short- form mnowledge-intensin as factoid question answering (QA) (Kwiatkowski et al., 2019; Joshi et al., 2017), where the informa- tion needs are clear in the user's inpul, and it is sufficient to retrieve relevant knowledge once solely based on the input.
generation tasks such
Increasingly powerful large LMs have also demonstrated abilities in more complex tasks that involve generating long-form output, such as long- form QA (Fan et al., 2019; Stelmakh et al., 2022), open-domain summarization (Cohen al., 2021; Hayashi al., 2021: Giorgi al., 2022), and tf-tuuht CoT) reasoning (Wei al., 2022; Ho et al., 2020; Geva et al., 2021: Hendrycks et al., 2020). In contrast to short-form generation, long-form generation presents complex informa- tion needs that are not always evident from the in- put alone. Similar to how humans gradually gather information as create content such papers, essays, books, long-form generation with LMs would require gathering multiple pieces of knowl- edge throughout the generation process. For ex- ample, to generate a summary about a particular topic, the initial retrieval based on the topic name
1 Introduction
Generative language models (LMs) (Brown et al., 2020; Ouyang et al., 2022; OpenAI, 2023; Chowd- hery et al., 2022; Zhang et al., 2022; Touvron et al., 2023: Zhao et al., 2023) have become founda- tional component in natural language processing (NLP) systems with their remarkable abilitics. Al- though LMs have memorized some world knowl- edge during training (Petroni et al., 2019; Roberts et al., 2020; Jiang et al., 2020), they still tend to
'Lead contributors.
'Code and datascts are available at tttpps//ggituu com/ jzbjyb/FLARE.

---

Search results: D
LM
Retriever
[1]:
Search results: D 92
Retrieved
[2]:
[1] Search results:
Dax
documents
[2]:
[1]:
[2]:
Input
Generate summary about Joe Biden.
Step1
81
Joe Biden (born November 20, 1942) is the 46th president of the United States.
51
Biden (born November 20, 1942) 46th president of United States.
Step 2
Biden attended the University of Pennsylvanio, where he earned low
Generation
degree.
$2
graduated from the University of Delaware 1965 with Bachelor of Arts history and political science.
43
Step 3
8
Joe Biden announced his candidacy for the 2020 presidential election on August
18, 2019.
$3
Joe Biden announced his candidacy for the 2020 presidential election April
25, 2019.
Figure I: An illustration of forward-looke llokkng active retrieval augmented generation (FLARE). Starting with the user input æ and initial retrieval results Da. FLARE iteratively generates temporary next sentence (shown in gray italic) and check whether it contains low-probability tokens (indicated with underline). If (step 2 and 3), the system retrieves relevant documents and regenerates the sentence.
(e.g., Joe Biden) may not cover all aspects and de- tails. It is crucial retrieve extra information as needed during generation, such as when generat- ing a certain aspect (e.g., Joe Biden's education history) or specific detail (c.g., the date of Joe Biden's presidential campaign announcement).
they lack the required knowledge to avoid unneces- sary or inappropriate retrieval that occurs passive retrieval augmented LMs (Khandelwal et al., 2020; Borgeaud et al., 2022; Ram et al., 2023; Trivedi et al., 2022). Given the observation that large LMs tend to be well-calibrated and low probability/con- fidence often indicates lack of knowledge (Ka- davath al., 2022), we adopt active retrieval strategy that only retrieves when LMs generate low- probability tokens. When deciding what retrieve, is important to consider what LMs intend to gen- erate in the future, as the goal of active retrieval to benefit future generations. Therefore, we propose anticipating the future by generating temporary next sentence, using it as a query to retrieve rel- evant documents, and then regenerating the next sentence conditioning the retrieved documents. Combining the two aspects, we propose Forward- Looking Active REtrieval augmented generation FLARE), as illustrated in Figure 1. FLARE iter- atively generates a temporary next sentence, use as the query to retrieve relevant documents if it contains low-probability tokens and regenerate the next sentence until reaches the end.
Several attempts have been made to retrieve mul- tiple times throughout generation. These attempts include methods that passively use the past context to retrieve additional information at fixed interval (Khandelwal al., 2020; Borgeaud al., 2022; Ram al., 2023; Trivedi al., 2022) which might not accurately reflect what LMs intend to gener- ate in the future or retrieve at inappropriate points. Some works in multihop QA decompose the full question into sub-questions, each of which is used to retrieve extra information (Press et al., 2022; Yao et al., 2022; Khot et al., 2022; Khattab et al., 2022).
We ask the following question: can we create simple and generic retrieval augmented LM that ac- tively decides when and what retrieve throughout the generation process, and applicable va- riety of long-form generation tasks? We provide generalized view of active retrieval augmented gen- cration. Our hypothesis regarding when to retrieve is that LMs should retrieve information only when
FLARE is applicable to any existing LMs at inference time without additional training. Con-

---

sidering the impressive performance achieved by GPT-3.5 (Ouyang al.. 2022) on variety of tasks, we examine the effectiveness of our meth- ods on text-davinci-00.. We evaluate FLARE on diverse tasks/datasets involving generating long outputs, including multihop QA (2WikiMul- tihopQA), commonsense reasoning (StrategyQA), long-form QA (ASQA), and open-domain summa- rization (WikiAsp) (Ho al., 2020; Geva al., 2021; Stelmakh et al., 2022; Hayashi al., 2021). Over all tasks, FLARE achieves superior or com- petitive performance compared to single-time and multi-time retrieval baselines, demonstrating the effectiveness and generalizability of our method,
resulting in the interleaving of retrieval and genera- tion. Formally, at step t(t 2 1), the retrieval query I is formulated based on both the user input a and previously generated output y<r - [yo..... yt-1]:
q: qry( y<1),
where qry(-) is the query formulation function. At the beginning (t = 1), the previous generation is empty (y<1 = 0), and the user input is used as the initial query (91 = æ). Given retrieved documents Dq. LMs continually generate the answer until the retrieval is triggered or reaches the end:
LM qc 2, y<ce),,
where Ye represents the generated tokens the cur- rent step t, and the input LMs is the concatena- tion of the retrieved documents Qą. the user input æ, and the previous generation Y<t- We discard previously retrieved documents Up and only use the retrieved documents from the current step to condition the next generation to prevent reaching the input length limit of LMs.
2 Retrieval Augmented Generation
We formally define single-time retrieval augmented generation and propose the framework of active retrieval augmented generation.
2.1 Notations and Definitions
Given user input and a document corpus D { [P] (such as all Wikipedia articles), the goal of retrieval augmented LMs is to generate the answer
3 FLARE: Forward-Looking Active REtrieval Augmented Generation
y [81, 82, m [wi, u2,.... Wn] containing m sentences or n tokens leveraging information retrieved from the corpus.
Our intuition is that (1) LMs should only retrieve information when they do not have the necessary knowledge to avoid unnecessary or inappropriate retrieval, and (2) the retrieval queries should reflect the intents of future generations. We propose two forward-lookin active retrieval augmented gener- ation (FLARE) methods to implement the active retrieval augmented generation framework. The first method prompts the LM to generate retrieval queries when necessary while generating the an- swer using retrieval-encouraaingg instructions, de- noted as FLARE The second method directly uses the LM's generation search queries, denoted as FLAREdinech which iteratively generates the next sentence gain insight into the future topic, and if uncertain tokens are present, retrieves relevant documents to regenerate the next sentence.
In retrieval augmented LM, the LM typically pairs with a retriever that can retrieve a list of documents Dg = ret(q) for a query qi the LM conditions on both the user input æ and retrieved documents Dy to generate the answer. Since we focus on examining various methods of determin- ing when and what to retrieve, we follow exist- ing methods (Ram al., 2023; Trivedi et al., 2022) prepend the retrieved documents before the user input to aid future generation for both baselines and our method for fair comparisons: y = ( q. æI), where - is concatenation fol- lowing the specified order.
2.2 Single-time Retrieval Augmented Generation
3.1 FLARE with Retrieval Instructions
The most common choice is to directly use the user input the query for retrieval and generate the complete answer once y LM([D.,, æz))
Inspired by Toolformer (Schick et al., 2023), a straightforward way of expressing information needs for retrieval is generate "[Scarch(query)]" when additional information is needed (Schick et al., 2023), e.g.. "The colors on the flag of Ghana have the following meanings. Red is for [Scarch(Ghana flag red meaning)] the blood of mar- tyrs, When working with GPT-3.5 models that
2.3 Active Retrieval Augmented Generation
To aid long-form generation with retrieval, we pro- pose active retrieval augmented generation. It is a generic framework that actively decides when and what to retrieve through the generation process,

---

Search results: x
retrieval instructions might not be reliable. There- fore, we propose a more direct way of forward- looking active retrieval that uses the next sentence to decide when and what to retrieve.
[1]: Search results: 44
Retriever
[2]:
[1]: Search results:
Dss
[2]
[1]:
[2]
Input
3.2.1 Confidence-based Active Retrieval
Generate a summary about Joe Biden.
As shown Figure at step 1, we first generate a temporary next sentence 8, = LM yea)) with- out conditioning on retrieved documents. Then we decide whether to trigger retrieval and formulate queries based on $,. If the LM is confident about $,, accept it without retrieving additional informa- tion; if not, q retrieve relevant documents, and then regen- erate the next sentence 8t. The reason we utilize sentences as the basis of our iteration due to their significance as semantic units that are neither too short nor too lengthy like phrases and paragraphs. However, our approach can also utilize phrases or paragraphs as the basis.
Y1 Joe Biden attended
Generation
42 [Search(Joe Biden University]]
42
Y2 the University of Pennsylvania, where he earned
q3 Search(Joe Biden degree)]
43
a law degree.
8, formulate search queries
An illustration of forward-looking active re- trieval augmented generation with retrieval instructions (FLARBiistaa). It iteratively generates search queries (shown in gray italic) to retrieve relevant information to aid future generations.
Figure 2:
offer only API access, we elicit such behavior by few-shot prompting (Brown et al., 2020).
Since LMs tend to be well-calibrated that low probabblity/cooffdeece often indicates a lack of knowledge (Jiang et al., 2021; Kadavath al., 2022; Varshney et al., 2022), we actively trigger retrieval any token &, has a probability lower than a threshold 0 E [0, 1]. = 0 means retrieval is never triggered, while 0 = triggers retrieval every sentence.
Specifically, for a downstream task, we place the search-related instruction and exemplars at the beginning as skill 1, followed by the instruction and exemplars of the downstream task as skill 2. Given test case, we ask LMs to combine skills and 2 to generate search queries while performing the task. The structure of the prompt is shown in Prompt and full details can be found in Prompt D.3.
if all tokens of &, have probs St LM[[Dq. x. y<el)
y1 =
Prompt 3.1: retrieval instructions
otherwise
Skill 1. An instruction to guide LMs to generate search queries.
where the query qi is formulated based on š,.
Several search-related exemplars.
3.2.2 Confidence- Query Formulation One way to perform retrieval is to directly use the next sentence š; as the query q- This shares a sim- ilar spirit with methods that use generated hypo- thetical titles or paragraphs from LMs retrieval queries evidences (Gao et al., 2022; Sun et al., 2022: Yu et al., 2022: Mao et al., 2021). We gen- eralize such techniques to long-form generation where active information access is essential.
Skill 2. An instruction to guide LMs to perform a specific downstream task (e.g.. multihop QA).
Several relatee exemplars.
An instruction to guide LMs to combine skills 1 and 2 for the test case. The input of the test case.
As shown in Figure 2, when the LM generates "[Search(query)]" (shown in gray italic), we stop the generation and use the query terms to retrieve relevant documents, which are prepended before the user input to aid future generation until the next search query is generated or reaches the end. Additional implementation details are included in Appendix A.
We found retrieving with the next sentence achieves significantly better results than with the previous context, as shown later in subsection 6.2. However, it has risk of perpetuating errors tained in it. For example, if the LM produces the sentence "Joe Biden attended the University of Pennsylvania" instead of the correct fact that he attended the University of Delaware, using this er- roneous sentence as a query might retrieve mislead-
3.2 Direct FLARE
Since we cannot fine-tune black-box LMs, we found queries generated by FLAREmstnn through

---

Joe Biden attended the University of Peansylvania, where he earned law degree.
queries qt are formulated based on 8, follows:
Implicit query by masking
explicit query by question generotion
if all tokens of & have probs > 0
qi
mask(&;) qgen(&;) otherwise
Biden
Ask question which "the University Pennsylvania" Ask question which the answer is "a law degree"
3.3 Implementation Details
Base LM We validate our method on one of the most advanced GPT-3.5 LMs fext-davinci-00 by iteratively querying their API.
LM such as ChatGPT
What university did Joe Biden attend? What degree did Joe Biden earn?
Document corpus and retrievers. Since we fo- cus on the integration of retrieval and generation, we use off-the-shelf retrievers that take queries as inputs and return a list of relevant documents. For datasets that mainly rely on knowledge from Wikipedia, we use the Wikipedia dump from Karpukhin et al. (2020) and employ BM25 (Robert- son and Zaragoza, 2009) the retriever. For datasets that rely on knowledge from the open web, we use the Bing search engine as our retriever."
Figure 3: Implicit and explicit query formulation. To- kens with low probabilities are marked with underlines.
ing information. We propose two simple methods to overcome this issue as illustrated in Figure 3.
Masked sentences as implicit queries. The first method masks out low-confidence tokens in & with probabilities below a threshold B [0, 1), where higher results in more aggressive masking. This removes potential distractions from the sentence to improve retricval accuracy.
Retrieved document formatting. Multiple re- trieved documents are linearized according to their ranking and then added to the beginning of the user input using Prompt D.I.
Other implementation details such as sentence kenization and efficiency are included Appendix A.
Generated questions as explicit queries. An- other method is to generate explicit questions that target the low-confident span in &, For example, if the LM is uncertain about "the University of Penn- sylvania", question like "Which university did Joe Biden attend?" can help retrieve relevant in- formation. Self-ask (Press et al., 2022) achieved this by manually inserting follow-up questions into downstream task exemplars as shown later in Prompt D.2, which requires task-specific annota- tion efforts. Instead, we developed a universal ap- proach that generates questions for low-confidence spans without additional annotation. Specifically, We first extract all spans from 8, with probabilities below B. For each extracted span z, we prompt 5-turbo 5-t-bo to generate a question qt,x that can be answered with the span:
Multi-time Retrieval Baselines
Existing passive multi-time retrieval augmented LMs can also be formulated using our framework (subsection 2.3). In this section, we formally in- troduce three baseline categories based on when and what retrieve. These baselines are not exact reproductions of the corresponding paper because many design choices differ which makes direct comparisons impossible. We implemented them using the same settings, with the only variation being when and what to retrieve.
Previous-window approaches trigger retrieval every / tokens, where / represents the window size. Generated tokens from the previous window are used as the query:
Prompt 3.2: zero-shot question generation
4: Vt-1 (1 2),
User input at
Generated output so far yst-
y/: [1 (00-1)++1.... we).
Given the above passage, ask a question to which the answer is the term/entity/phrase "z"
Some existing methods this category are RETRO (Borgeaud et al., 2022), IC-RALM (Ram et al.,
We retrieve using each generated question and interleave the returned documents into a single ranking list to aid future generations. summary,
April 23. https///wwwww.coooffto.o usteeennnss/biin http:///p//u/bbgg//ps////s/s//// bing-web-sent web-search-api
http:://app..opnaaa.co///////

---

2023), which retrieve every few tokens, and KNN- LM (Khandelwal et al., 2020), which retrieves ev- ery token.* + We follow Ram et al. (2023) to use a window size of - 16.
questions sourced from Wikipedia articles that re- quire composition, comparison, or inference, c.g.. "Why did the founder of Versus die?" We follow Wang et al. (2022) to generate both the chain-of thought and the final answer. Experimental setting details are included in Appendix B.
Previous-sentence approaches trigger retrieval every sentence and use the previous sentence the query, and IRCoT (Trivedi et al., 2022) belongs to this category:
We use regular expressions to extract the final answer from the output and compare it with the ref- erence answer using exact match (EM), and token- level F1, precision, and recall.
9: yt-1
(t 1 2),
yt St.
Commonsense reasoning Commonsense reason- ing requires world and commonsense knowledge to generate answers. We use StrategyQA (Geva et al., 2021) which is collection of crowdsourced yes/no questions, e.g., "Would a pear sink in wa- ter?" We follow Wei et al. (2022) to generate both the chain-of-thoughe and the final yes/no answer. Details are included in Appendix B.
Question decomposition approaches manually annotated task-specific exemplars to guide LMs to generate decomposed sub-questions while pro- ducing outputs. For example, self-ask (Press et al., 2022), a method in this category, manually inserts sub-questions in exemplars using Prompt D.2. For the test case, retrieval is triggered dynamically whenever the model generates sub-question.
We extract the final answer and match it against the gold answer using exact match.
The aforementioned approaches can retrieve ad- ditional information while generating. However, they have notable drawbacks: (1) Using previously generated tokens as queries might not reflect what LMs intend to generate in the future. (2) Retriev- ing information at a fixed interval can be inefficient because it might occur at inappropriate points. (3) Question decomposition approaches require task- specific prompt engineering, which restricts their generalizability in new tasks.
Long-form QA Long-form QA aims to generate comprehensive answers to questions sceking com- plex information (Fan et al., 2019; Stelmakh et al., 2022). We use ASQA (Stelmakh et al., 2022) our testbed where inputs are ambiguous questions with multiple interpretations, and outputs should cover all of them. For example, "Where do the Philadel- phia Eagles play their home games?" could be asking about the city, sports complex, or stadium. We found in many cases it is challenging even for humans to identify which aspect of the question is ambiguous. Therefore, we created another set- ting (ASQA-hin) where we provide brief hint to guide LMs stay on track when generating an- swers. The hint for the above case is "This question ambiguous in terms which specific location or venue is being referred to." Experimental setting details are included Appendix B.
5 Experimental Setup
We evaluate the effectiveness of FLARE on 4 di- verse knowledge-intensivee tasks using few-shot in- context learning (Radford et al., 2019; Brown et al., 2020; Liu et al., 2023). We follow previous works (Trivedi et al., 2022) to sub-sample at most 500 examples from each dataset due the cost of run- ning experiments. Datasets, metries, and settings are summarized in Table 7 of Appendix B. The hyperparameters of FLARE are selected based on the development set and listed in Table 9. FLARE refers to FLAREairect if not specifically stated.
We use metrics from Stelmakh et al. (2022), in- cluding EM, RoBERTa-based QA score (Disambig- F1), ROUGE (Lin, 2004), and overall score com- bining Disambig-Fi and ROUGE (DR).
Open-domain summarization The goal of open- domain summarization is to generate comprehen- sive summary about a topic by gathering informa- tion from open web (Giorgi et al., 2022). We use WikiAsp (Hayashi et al., 2021) which aims gen- erate aspect-based summaries about entities from 20 domains in Wikipedia, e.g., "Generate a sum- mary about Echo School (Oregon) including the
Multihop QA The goal of multihop QA to answer complex questions through information re- trieval and reasoning. We use 2WikiMullihopQA (Ho et al., 2020) which contains 2-hop complex
"Since KNN-LM uses the contextualized representation corresponding to the current decoding position to retrieve rel- evant information which encodes all previous tokens. Strietly speaking. q. should he yet.

---

No ret.
Single-fing himlee-tmmeeee ret.
Previcus-window ret.
Forward-Lond Lookad-Looking Active REtrieval augmented generation (FLARE)
72.9
68.8
51.0
$2:451
91.3
43.241
16.2
28.2
ZWKiMMulihhpQA
StrategyQA
ASQA
ASQA-hint
WikiAsp
Figure 4: Comparision between FLARE and baselines across all tasks/datasets. We report the primary metric for each dataset: EM for 2WikiMultihopQA, Strateey StrategyQA, and ASQA, and UniEval for WikiAsp.
following aspects: academics, history." Experimen- tal setting details are included in Appendix B. Metrics include ROUGE, named entity-basee based F and UniEval (Zhong et al., 2022) which measures factual consistency.
Methods
F1
Prec.
Rec.
No retrieval
28.2
36.8
36.5
38.6
Single-time retrieval
39.4
48.8
48.6
51.5
Multi-time retrieval
Previous-window -window
43.2
52.3
51.7
54.5
Previous-senience
39.0
49.2
48.9
51.8
Question decomposition
47.8
56.4
56.1
58.6
6 Experimental Results
FLAREasnas (ours)
42.4
49.8
49.1
52.5
FLAREa (ours)
51.0
59.7
59.1
62.6
We first report overall results across 4 tasks/datasets and compare the performance FLARE with all the baselines introduced in section 4. We then run ablation experiments to study the efficacy of various design choices of our method.
Table FLARE and baselines on 2WikiMultihopAA Previous Previous-window (Borgeaud al., 2022; Ram al.. 2023), previous- (Trivedi et al., 2022), and ques- tion decomposition (Press al., 2022; Yao et al., 2022) methods reimplemented for fair comparisons.
6.1 Comparison with Baselines
Overall results. The overall performance of FLARE and baseline across all tasks/datasets are reported in Figure 4. FLARE outperforms all base- line on all tasks/datasets. indicating that FLARE is generic method that can effectively retrieve additional information throughout the generation.
margins. The improvement of retrieving using the previous sentence is relatively small which we hy- pothesize is mainly because the previous sentence often describes entities or relations different from those in the next sentence 2WikiMultihopAA While the previous-window approach might use the first half of a sentence to retrieve information potentially helpful for generating the second half. Among all basclines, the question decomposition approach (Press al., 2022) achieves the best per- formance. which is not surprising since the in- context exemplars manually annotated with decom- posed sub-questions (Prompt D.2) guide LMs to generate sub-questions that align with the topic/in- tent of future generations. FLARE outperforms this baseline, indicating that manual exemplar an- notation is not necessary for effective future-aware retrieval. The gap between FLARE and ques- tion decomposition is large, indicating that teaching LMs to generate search queries using task ees-ggeeric retrieval instructions and exemplars challenging.
Among various tasks, multihop QA shows the most significant improvement. This largely due to the task's ' clear definition and specific objective producing the final answer through a 2-hop rea- soming process, which makes it easier for LMs to generate on-topic output. In contrast, ASQA and WikiAsp are more open-ended, which increases the difficulty of both generation and evaluation. The improvement on ASQA-hint larger than that of ASQA because identifying ambiguous aspects is challenging even for humans in many cases, and providing a generic hint helps LMs to stay on topic.
Thorough comparisons with baselines. The per- formance of all basclines 2WikiMultihopQA are reported in Table 1. FLARE outperforms all baselines by large margin, which confirms that borward-looking active retrieval highly effective. Most multi-time retrieval augmented approaches outperform single-time time retrieval but with different
We report all metrics for the other datasets Table 2. FLARE outperforms baselines with re- spect to all metrics. Retrieval using the previ-

---

Datasets Metrics
Strategy QA EM
ASQA
ASQA-hint
WikiAsp
EM D-F, R-L DR
EM D-F, R-L DR
UniEval E-F, R-L
No retrieval
72.9
33.8 24.2 33.3 28.4
40.1 32.5 36.4 34.4
47.1
14.1 26.4
Single-time retrieval
68.6
40.0 27.1 34.0 30.4
43.2 34.8 37.4 36.0
52.4
17.4 26.9
Multi-time retrieval
Previous-windee window
71.2
39.9 27.0 34.3 30.4
43.7 35.7 37.5 36.6
51.8
18.1
27.3
Previous-sentence
71.0
39.9
27.9 34.3 30.9
44.7 35.9
37.5 36.7
52.6
17.8
27.2
FLARE (ours)
77.3
41.3 28.2 34.3 31.1
46.2 36.7 37.7 37.2
53.4
18.9
27.6
Table 2: Comparison between FLARE and baselines on StrategyQA, ASQA, ASQA-hint, and WikiAsp. D-F, is Disambig-Fi F,, R-L is ROUGE-L, and E-F, is named entity-based Ft.
2WikiMultihoPAA
ASQA-hint
2WikiMullihaPAA
StrategyOA
EM F, Prec. Rec.
EM D-F, R-L DR
80.0
60.0
72.9
73.8
77.3
76.9.74.7-73.4
73.4
Previous
39.0 49.2
48.9
51.8
42.5
34.1
36.9 35.5
Next
48.8 57.6
57.1 60.5
45.9
35.7
37.5 36.6
40.0
44.6
47.6
47.2
51.0
51.0
20.0
28.2
Table head-to-head comparison between using the previous sentence and the next sentence for retrieval.
0.0
25.0
50.0
75.0
100.0
%ssees//eeneences wilh retrieval
#Tokens
EM
Prec.
Rec.
Figure 5: Performance (EM) of FLARE with respect to the percentage steps/sentences with retrieval on 2WikiMulthhoPQA and StrategyQA.
16
43.2
52.3
51.7
54.5
32
43.6
52.4
52.0
55.0
48
40.0
49.3
49.0
52.0
All
39.0
48.5
48.2
51.1
the past hurts the performance, further confirming our hypothesis that previous context might not be relevant to intent of future generations.
Table 4: Previous window approaches using different numbers of tokens queries.
Importance of active retrieval. Next, we inves- tigate how active retrieval threshold affects per- formance. To alter our method from not retrieving to retrieving every sentence, adjust the confi- dence threshold that determines when to trigger retrieval from portion of steps/sentences where retrieval is acti- vated, and present the performance based on it. As shown in Figure 5, on 2WikiMulthhopAA, the per- formance plateaus when the retrieval percentage exceeds 60%, indicating that retrieval when LMs confident is not necessary. On Strategy QA, the performance drops when the retrieval percentage exceeds 50%, indicating that unnecessary retrieval can introduce noise and impede the original gen- eration process. We found triggering retrieval for 40%-80% of sentences usually leads to a good per- formance across tasks/datasets.
ous window underperforms single-time retrieval on ASQA, which we hypothesize is because the previous window does not accurately reflect future intent. Since we focus on evaluating factuality, met- rics with emphasis on factual content (such as EM, Disambig-Fj, UniEval) are more reliable than metrics computed over all tokens (ROUGE-L).
We then calculate the pro-
6.2 Ablation Study
Importance of forward-looking retrieval. We first validate that forward-looking retrieval is more effective than cest-context-based retrieval. We run ablation experiments on 2WikiMultihopAA and ASQA-hint comparing retrieval using the previ- ous versus the next sentence. Specifically, both methods retrieve every sentence and directly use the complete previous/next sentence as queries. As shown in Table 3, using the next sentence to retrieve is clearly better than using the previous sentence, confirming hypothesis.
Effectiveness of different query formulation methods We study implicit query formation by masking and explicit query formulation through question generation. In Table 5, we compare the performance FLARE with different masking
We also run previous-ver mevious-window approaches using different numbers of past tokens as queries. As shown Table 4, using too many tokens (> 32) in

---

8
EM
Prec.
8 Conclusion
0.0
0.488
0.576
0.571
0.605
To aid long-form generation with retrieval aug- mentation, we propose an active retrieval aug- mented generation framework that decides when and what to retrieve during generation. We imple- ment this framework with forward-looking active retrieval that iteratively uses the upcoming sentence to retrieve relevant information if contains low- confidence tokens and regenerates the next sen- tence. Experimental results on tasks/datasets demonstrate the effectiveness of our methods. Fu- ture directions include better strategies for active retrieval and developing efficient LM architectures for active information integration.
0.2
0.498
0.588
0.582
0.616
0.4
0.510
0.597
0.591
0.627
0.6
0.506
0.593
0.586
0.622
Table 5: Performance of FLARE with respect to the masking threshold / on 2WikiMultihoPO..
ASQA-hint EM D-F, R-L DR
WikiAsp UniEval E-F, R-L
Implicit 45.7 36.9 37.7 37.3
53.4
18.8
27.7
Explicit 46.2 36.7 37.7 37.2
53.4
18.9
27.6
Table 6: comparison between implicit and explicit query formulation methods in FLARE.
Limitations
thresholds 3. Retrieving directly with the complete sentence (3- 0) is worse than masking tokens with low probabilities, confirming our hypothesis that low-confidence erroneous tokens can distract retrievers. We compare implicit and explicit query formulation methods in Table 6. Performances of both methods are similar, indicating that both meth- ods can effectively reflect information needs.
We also conduct experiments on Wizard of Wikipedia (Dinan et al., 2019) and ELI5 (Fan et al., 2019), and found that FLARE did not provide sig- nificant gains. Wizard of Wikipedia is knowledge- intensive dialogue generation dataset where the out- put is relatively short (~20 tokens on average) retrieving multiple disparate pieces of information might not be necessary. ELI5 (Fan al., 2019) a long-form QA dataset requiring in-depth an- swers to open open-ended questions. Due to issues men- tioned in Krishna et al. (2021) such as difficulties of grounding generation in retrieval and evalua- tion, both single-time retrieval and FLARE did not provide significant gains over not using retrieval. From an engineering perspective, interleaving gen- cration and retrieval with a naive implementation increases both overheads and the cost of generation. LMs need to be activated multiple times (once for each retrieval) and a caching-free implementation also requires recomputing the previous activation each time after retrieval. This issue can be poten- tially alleviated with special architectural designs that encode the retrieved documents Dz, and the input/generation (w/y independently.
7 Related Work
We refer to subsection 2.2 and section for ex- tensively discussion on single-time and multi-time time retrieval augmented LMs, which is the most rele- vant area to this paper.
Iterative and adaptive retrieval Iterative re- trieval and refinement has been studied in both text and code generation tasks (Peng et al., 2023; Zhang et al., 2023; Zemlyanskiy al., 2022; Yu al., 2023). FLARE differs from these methods in the granularity of generation and retrieval strategies. Adaptive retrieval has been studied in single-time time retrieval scenarios based on either question pop- ularity or generation probabilities (Mallen et al., 2022; Li et al., 2023), while we focus on long-form generation requiring active information access.
A Acknowledgements
This work was supported in part by a grant from the Singapore Defence Science and Technology Agency and the IBM PhD Fellowship. We thank Chunting Zhou, Amanda Bertsch, Uri Alon, Hi- roaki Hayashi, Harsh Trivedi, Patrick Lewis, Timo Schick, Kaixin Ma, Shuyan Zhou, and songwei Ge for their insightful discussions and help with the experiments.
Browser-enhanced LMs WebGPT (Nakano et al., 2021) and WebCPM (Qin et al., 2023) train LMs to interact with browser to enhance factuality using reinforcement learning or supervised train- ing where multiple queries can be triggered before generation. FLARE is built on bext- retrievers but can be combined with a browser to potentially improve retrieval quality.

---

References
Nachshon Cohen, Oren Kalinsky, Yftah Ziser, and Alessandro Moschitti. 2021. Wikisum: Coherent summarization dataset for efficient human-evaluation. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/JCCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 212- 219. Association for Computational Linguistics.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, lean-Baptise Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang. Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Si- monyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bul- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston, 2019. Wizard of wikipedia: Knowledge-poowe powered conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net,
Angela Fan, Yacine Jernite, Ethan Perez. David Grang- ier, Jason Weston, and Michael Auli. 2019. ELIS: long form question answering. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3558-3567 Association for Computational Linguis- tics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Luyu Gao. Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without rele- vance labels. CORR, abs/2212. 10496.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, 2021. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL. 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 1870-1879. Association for Computational Linguistics.
John M. Giorgi, Luca Soldaini, Bo Wang, Gary D. Bader, Kyle Lo. Lucy Lu Wang, and Arman Co- han. 2022. Exploring the challenges of open domain multi-document summarization. CoRR, abs/2212. 10526.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes. Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, 0bs/22244002111
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming- Wei Chang. 2020. REALM: retrieval- augmented language model pre-training. CoRR, 0bs/2002.08909
Hiroaki Hayashi. Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021. Wikiasp: A dataset for multi-domain aspect- based summarization. Trans, Assoc. Comput. Lin- guistics, 1-222.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. CORR, abs/2009.03300
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A multi-hop QA dataset for comprehensive evaluation of reason- ing steps. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13,

---

2020, pages 6609-6625, International Committee on Computational Linguistics.
Urvashi Khandelwal. Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Gautier Izacard and Edouard Grave. 2021. Leveraging
passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, EACL 2021, Online, April 19- 23, 2021, pages 874- 880. Association for Computational Linguisties.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. bemonstrate-searchh Composing retrieval and language models for knowledge-intensiee NLP. CoRR, abs/2212.14024.
Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. CoRR, abs/2208.03299.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. CoRR, abs/2210.02444
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know When language models know? on the calibration of language mod- els for question answering. Trans. Assoc. Comput. Linguistics, 9:962-977 977.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021. Hurdles to progress in long-form question answering. In North American Association for Computational Linguistics.
Zhengbao Jiang, Luyu Gao, Jun Araki, Haibo Ding, Zhiruo Wang. Jamie Callan, and Graham Neubig, 2022. Retrieval as attention: End-to-ent learning of retrieval and reading within a single transformer. CoRR, 02/2221200222..
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Callins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Woi Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452- 466.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How models know. Trans. Assoc. Comput. Linguistics, 8:423-438.
know what language
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 1601-1611. Association for Computational Linguistics.
Angeliki Lazaridou. Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot prompting for open-domain question answering. CoRR, abs/2203.05115.
Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paran- jape, Christopher D. Manning, and Kyoung-Gu Woo. 2021. You only need one model for open-domain question answering. CORR, abs/2112.07381.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer. Zac Hatfield-Dodde Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jack- son Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse. Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. CoRR, 2ttp2/2077.22211.1
Patrick S. H. Lewis. Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel. Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmente
generation for knowledge-intensive NI.P tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12 12, 2020, virtual.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jingyuan Wang, Jian-Yun Nie, and Ji-Rong Wen. 2023. The web can be your oyster for improving large language models. CoRR, abs/2305.10990
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020. pages 6769-6781. Associa- tion for Computational Linguistics.
Chin-Yeu Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

---

Association for Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Comput. Surv. 1509):955:1199:33.
EMNLP-IJCNLP 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 1663-2433. Association for Computational Linguistics.
Ofir Press, Muru Zhang. Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2022. Mcasuring and narrowing the compositionality gap in language models. arXiv preprini
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Investigating effectiveness and limitations of parametric and non- parametric memories. CoRR, 101/2222.11511..
Hongjing Qian, Yutao Zhu, Zhicheng Dou, Haoqi Gu. Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Webbrain: Learning to generate factually correct articles for queries by grounding on large web corpus. CoRR, ahs/2304.0433
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao. Jiawei Han, and Weizhu Chen. 2021. Generation-augne augmented retrieval for open- domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Compu- national Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACL/I- JCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4089-4100. Association for Computational Linguistics.
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang. Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023. Webepm: In- teractive web search for chinese long-form question answering. CoRR, abs/2305.06849.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1915 On- line. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Blog, 1(8).
Ori Ram, Yoav Levine. Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Brown, and Yoav Shoham. 2023. In-context surrievll-augmente lan- guage models. arXiv preprint arXhv:2302.00083
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser- assisted question question-answerinne answering with human feedback. CoRR, abs/2112.09332
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages Computational Linguistics.
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155.
Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and be- yond. Found. Trends Inf. Retr. 3(4):333-389
Devendra Singh Sachan, Siva Reddy, William L.. Hamil- ton, Chris Dyer, and Dani Yogatama. 2021. End-to- end training of multi-document reader and retriever for open -pmn-domain question answering. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 25968-25983
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feed- back. CoRR, 2iN/2302.12813.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Seialom. 2023. Toolformer: Language models can teach themselves to use tools.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language mod- els as knowledge bases? In Proceedings of the Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing,
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. REPLUG: retrieval-augmenntdd black-box language models. CoRR, 12t//3301.12552
2019 Conference

---

Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Wei Chang. 2022. ASQA: factoid questions meet long-form answers. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arub Emirates, December 7-11, 2022, pages 87788.88. Association for Computational Linguistics.
Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. CoRR, abs/2303.12570
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. ArXiv, 0hss2200.01066.
Zhiqing Sun, Xuezhi Wang. Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation -uugmmetee language models. CoRR, abs/22 10.6
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yia Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, 1is/2333.1822..
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleav- ing retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. CoRR, abs/2212.10509.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2023-2038. Association for Computa- tional Linguisties.
Neeraj Varshney, Man Luo, and Chitta Baral. 2022. Can open-domain QA reader utilize external knowledge efficiently like humans? CoRR, abs/2211. 1227077
Xuezhi Wang, Jason Wei, Dale Schuurmans. Quoe Le, Ed H. Chi, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203.1l 11171.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Find- ings of the Association for Computational Linguis- rics: ACL-IJCNLP 2021, pages 1393-1404, Online. Association for Computational Linguisties.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning large language models. CoRR, abs/2201.11903.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. CoRR, abs/2210.03629.
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang, 2022. Generate rather than retrieve: Large language models strong context generators. CoRR, abs/2209. 10063.
Wenhao Yu. Zhihan Zhang, Zhenwen Liang. Meng Jiang, and Ashish Sabharwal. 2023. Improving lan- guage models via plug-and-play retrieval feedback. CoRR, abs/2305. 14002..
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, and Fei Sha. 2022. Generate-and--etrieve Use your predictions improve retrieval for seman- tie parsing. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 4946 4951. International Committee on Computational Linguistics.

---

A FLARE Implementation Details
Multihop QA For "Why did the founder of Ver- sus die?", the output we aim to generate is "The founder of Versus was Gianni Versace. Gianni Ver- sace was shot and killed on the steps of his Miami Beach mansion on July 13, 1997. So the answer is shot." We use 8 exemplars from Trivedi et al. (2022) listed in Prompt D.4 for in-context learn- ing, BM25 as the retriever, and Wikipedia articles as the retrieval corpus. Similar to the observation in Trivedi retrieval results for exemplars improves the per- formance, we use the input æ of each exemplar to retrieve several documents and then add them using the format in Prompt D.I. We found increasing the number of retrieval documents often increases per- formance. Therefore, we use the maximum number of documents that fit within the input length limit of text-davinci-003, which is 2 for 2Wiki- MultihopQA.
FLARE implementation details We found that LMs can effectively combine retrieval and downstream task-related skills and generate meaningful search queries while performing the task. However, there two issues: (1) LMs tend generate fewer search queries than necessary. (2) Generating excessive search queries can disrupt answer generation and adversely affect performance. We address these issues using two methods respectively. First, we increase the logit of the token "T" by 2.0 to improve the chances of LMs generating "[Search(query)]". Second, whenever LMs generate search query, use it retrieve relevant information, promptly remove it from the generation, and generate the next few tokens while forbidding "T" by adding large negative value to the logit of "[".
(2022), we found incorporating
The initial query of FLARE. FLARE starts with the user input a as the initial query to re- trieve documents to generate the first sentence $ LM([2, æ)) to bootstrap the iterative gener- ation process. For the following steps, the tempo- rary forward-looking sentence is generated without retrieved documents.
Commonsense Reasoning For "Would pear sink in water?", the output aim to generate is "The density of pear about 0.6g/cm², which is less than water. Objects less dense than water float, Thus, a pear would float. So the final answer is no." We use exemplars from Wei et al. (2022) listed Prompt D.5, BM25 on the Wikipedia corpus, and 3 retrieved documents to run experiments.
Sentence tokenization. For each step we gen- crate 64 tokens which are longer than most sen- tences, and NLTK sentence tokenizer tract the first sentence and discard the rest.
Long-form QA For "Where do the Philadelphia Eagles play their home games?", the output we aim to generate is "We need to consider the dif- ferent possible locations or venues that could be considered the home field of the Philadelphia Ea- gles. These include the city, the sports complex, or the stadium. Therefore, this question has 3 in- terpretations and the answers are: (1) The city is Philadelphia. (2) The sports complex is the South Philadelphia Sports Complex. (3) The stadium is the Lincoln Financial Field stadium." For both the original setting (ASQA) and the setting with hints (ASQA-hine we manually annotate 8 exemplars (Prompt D.6 and D.8), use BM25 on the Wikipedia corpus, and 3 retrieved documents to experi- ments.
Efficiency As shown in subsection 6.2, aver- age retrieval is triggered for 30% 60% of sen- tences depending on downstream tasks. In compar- ision, KNN-LM (Khandelwal et al., 2020) retrieves every token, RETRO or IC-RALM (Borgeaud et al., 2022; Ram et al., 2023) retrievers every 4~32 to- kens, and IRCoT (Trivedi et al., 2022) retrieves every sentence. Compared to single-time retrieval, however, interleaving retrieval and generation with naive implementation indeed increases overheads. which we discuss in the limitation section (sec- tion 9).
Open-domain Summarization The original WikiAsp dataset is designed for multi-document summarization and provides list of references systems. We converted it into the open-domain setting by removing the associated references and instead gathering information from the open web. For "Generate a summary about Echo School (Ore-
B Datasets and Settings
Datasets, metrics, and experimental settings are summarized in Table 7.
Phttps://wwal tk. www//pi/g///iioooooo /n/l/ tokenize. FunktSenneeneeTokeiierrrcc

---

gon) including the following aspects: academics, history.", the oulput we aim to generate "# Aca- demics. 2008, % of the school's s seniors re- ceived their high school diploma... # History. The class of 2008 was the 100th class in the school's history." where # is used to indicate aspects. We manually annotate 4 exemplars (Prompt D.10), and use the Bing search engine to retrieve 5 documents from the open web. To avoid leaking, we exclude several Wikipedia-relan related domains listed in Table from Bing's search results.
Hyperparameters
Hyperparameters of FLARE on different datasets are listed in Table 9.
D Prompts and Few-shot exemplars
The prompt used to linearize multiple documents is shown in Prompt D.1. The prompt used in self- ask (Press et al., 2022) is shown in Prompt D.2. Prompts and exemplars of different tasks/datasets are shown in Prompt D.3, D.4, D.5, D.6, D.8, and D.10, respectively.
Prompt D.I: document formatting
Search results:
Document
2]
Document 2
The user input æ:
Prompt D.2: multihop QA with self-ask
Who lived longer. Theodor Haecker Harry Vaughan Watkins?
Question:
Are follow up questions needed here: Yes.
Follow up: How old Theodor Haccker when he died? Intermediate answer: Theodor Haccker years old when he died.
Follow up: How old was Harry Vaughan Watkins when died?
Intermediate answer: Harry Vaughan Watkins was 69 years when he the final answer is: Harry Vaughan Watkins.

---

Settings
2WikiMulthopQA (Ho et al., 2020)
Strategy (Geva al., 2021)
ASQA (Stelmakh al., 2022)
WikiAsp (Hayashi et al., 2021)
Dataset statistics
Task
multihop QA
commonsense QA
long-form QA
open-domain summarization 500
#Examples
500
229
500
Evaluation settings
Metrics
EM. Fi, Prec.. Rec.
EM
EM. Disambig-Fi Fi. ROUGE, DR UniEval, enity-Fi, ROUGE
Retrieval settings
Corpus
Wikipedia BM25
Wikipedia BM25
Wikipedia BM25
open web Bing 5
Retriever
Top-k
3
Prompt format
#Exemplars
8
4
Ret. for exemplars
/
X
X
Table 7: Dataset statistics and experimental settings different tasks.
wikipedia.org, wikiwand.com, wiki2.org, wikimedia.org
Table 8: Wikipedia--elatee domains excluded from Bing's search results.
Dataset
0
3
Query formulation Combine single- & multi-time retrieval
2WikiMultihopOA
0.8
0.4
implicit
X
StrategyQA
0.4
0.4
implicit
X
ASQA & ASQA-hint
0.8
0.4
explicit
WikiAsp
0.8
0.4
explicit
Table 9: Hyperparameters of FLARE on different datasets.

---

Prompt D.3: retrieval instructions for 2WikiMultihoPAA
Skill Use the Search look up relevant information by writing "[Search(term)]" where "term" search term you want to look up. For example:
Question: But what are risks during production nanomaterials?
Answer (with Search): "Scarch(nanomaterill production risks)] Some nanomaterials may give rise to various kinds lung damage.
The colors on the flag of Ghana have the following meanings. Answer (with Search); Red is for (Ghanc flag red meaning)] the blood of martyrs, green for forests, and gold for mineral wealth.
Question:
Question:
Metformin is the first-line drug for what?
Answer (with Search): [Search(Mettfrrmin first-line drug)) patients with type 2 diabetes and obesity.
Answer questions by thinking step-by-step. First, write out the reasoning steps, then draw the conclu- sion. For example:
Skill 2.
Question: When did the director of film Hypnerite (Film) die? Answer (with step-by-step The film Hypocrite was directed by Miguel Morayta. Miguel Morayta died on 19 June 2013. So the answer is 19 June 2013.
Question: Are both Kurram Garhi and Trojkisti located in the same country? Answer (with step-hy-step): Kurram Garhi is located in the country of Pakistan. Trojkrsti is located in the country of Republic of Macedonia. Thus, they not in the same country. So the answer is no.
Question: Do director of film Coolie No. (1995 Film) and director of film The Sensational Trial have the same nationality?
Answer (with step step): Coolie No. (1995 film) directed by David Dhawan. The Sensational Trial directed by Karl Freund. David Dhawan's 's nationality is India. Karl Freund's nationality Germany. Thus, they do not have the same nationality. So the answer no.
Question: Who is Boragchin (Wife Of Ögedei)'s father-in-law? Answer (with step-by-step): Boraqehin is married Ógedei Khan. Ögedei Khan's father Genghis Khan, Thus, Boraqchin's father-in-law Genghis Khan. So the answer Genghis Khan.
Question: Who was first of Martin Hodge and Ivania Martinich? Answer (with step-by-stepi Martin Hodge was born on February 1959, Ivania Martinich was born on 25 July 1995, Thus, Martin Hodge was born first.
answer is Martin Hodge.
Question: When did director of film Laughter Hell die? Answer (with step-by-step): The film Laughter In Hell was directed by Edward Cahn, Edward L. Cahn died on August 25, 1963. So the answer August 25, 1963.
Question: Which film has the director died later, The Gal Who Took the West or Twenty Plus Two? Answer (with step-by-see byp-by-step): The film Twenty Plus Two was directed by Joseph M. Newman. The Gal Who Took the West was directed by Frederick de Cordova. Joseph M. Newman died on January 23, 2006. Fred de Cordova died on September 15, 2001. Thus, the person to die later from the two is Twenty Plus Two. So the answer Twenty Plus Two.
Question: Who is the grandchild of Krishna Shah (Nepalese Royaly? Answer (with step-by-step): Krishna Shah has a child named Rudra Shah. Rudra Shah has a child named Prithvipati Shah. Thus, Krishna Shah has a grandchild named Prithvipati Shah. So the answer is Prithvipari Shah.
Now, combine the aforementioned two skills. First, write out the reasoning steps, then draw the conclusion, where the reasoning steps should also utilize the Search API "[Search(term]" whenever possible.
Question: Where did Minbyauk Thihapate's wife die?
Answer (with step-by-step & Search):

---

Prompt D.4: exemplars of 2WikiMultihoPAA
Question:
When did the director of film Hypocrite (Film) die?
Answer: The film Hypocrite was directed by Miguel Morayta. Miguel Morayta died 19 June 2013. 3. So the answer is 19 June 2013.
Are both Kurram Garhi and Trojkrsti located in the same country? Answer: Kurram Garhi is located the country of Pakistan. Trojkrsti located in the country of Republic of Macedonia. Thus, they not in the same country. So the answer is no.
Question:
Question: Do director of film Coolie No. (1995 Film) and director of film The Sensational Trial have the nationality?
Answer: Coolie No. 1 (1995 film) directed David Dhawan. The Sensational Trial directed by Karl Freund. David Dhawan's nationality India. Karl Freund's nationality Germany. Thus, they do not have the same nationality. is
Question: Who Boragchin (Wife Ógedei)'s father-in-lave
Answer: Boraqchin is married to Ögedei Khan. Ögedei Khan's 's father Genghis Khan, Thus, Boragehin's father-in-law
Genghis Khan. answer is Genghis Khan.
Question: Who was first of Martin Hodge and Ivania Martinich?
Answer: Martin Hodge was born on February 1959. Ivania Martinich was born on 25 July 1995. Thus, Martin Hodge was born first. So answer is Martin Hodge.
Question: When did director of film Laughter Hell die?
Answer: The film Laughter Hell was directed by Edward L. Cahn. Edward L, Cahn died on August 25, 1963. So the answer is August 25, 1963.
Question: Which film has the director died later, The Gal Who Took the West or Twenty Plus Two?
Answer: The film Twenty Plus Two was directed by Joseph M. Newman, The Gal Who Took the West was directed by Frederick de Cordova. Joseph M. Newman died on January 23, 2006. Fred de Cordova died on September 15, 2001, Thus, the person to die later from the two is Twenty Plus Two. So the answer is Twenty Plus Two.
Question: Who is the grandchild of Krishna Shah (Nepalese Royaly? Answer: Krishna Shah has a child named Rudra Shah. Rudra Shah has a child named Prithvipati Shah. Thus, Krishna Shah has grandchild named Prithvipati Shah. So the answer is Prithvipati Shah.
Question: Answer:
Which country the director of film Citizen Mavzik is from?

---

Prompt D.5: exemplars of StrategyQA
Generate yes answer the following question.
Question: Do hamsters provide food for any animals?
Answer: Hamsters are prey animals. Prcy are food for predators. Thus, hamsters provide food for some animals. So the final is yes.
Generate yes
the following question.
Question: Could Brooke Shields succeed University of Pennsylvania?
Answer: Brooke Shields to Princeton University. Princeton University about academically rigorous the Uni- versity of Pennsylvania. Thus, Brooke Shields could also succeed at the University of Pennsylvania. So the final answer yes.
the following question. Question: Hydrogen's atomic number squared excceds numher Spice Girls?
Generate yes
Answer: Hydrogen has atomic number of I. squared There Spice Girls. Thus, Hydrogen's 's atomic number is
squared is less than So the final
Generate a yes or no answer the following question.
Question: common see frost during some college commenccmennts?
Answer: College commencement ceremonies can happen December, May, and June. December the winter, so there can frost. Thus, there could he frost at some commencements. So the tinal answer is yes.
Generate a yes or no answer the following question.
Question: Could a llaaa birth twice during War in Vietnam (1945-46)?
Answer: The War in Vietnam was 6 months. The gestation period for a Ilama is 11 months, which is more than months. Thus, a llama could not give birth twice during the War in Vietnam. So the final answer is no.
Generate a yes or no answer to the following question.
Question: Would a pear sink in water?
Answer: The density of a pear is ubout 0.6g/cm², which less than water. Objects less dense than water float. Thus, a pear would float. So the final answer is no.
Generate a yes or no answer to the following question. Question: Did Spiderman fight against Falcon in the MCU?
Answer:

---

Prompt D.6: exemplars of ASQA
Given an ambiguous question, figure out its interpretations and answer them one by one. Question: Who played bonnie in gone with the wind?
Answer: This question is ambiguous in terms of which version or adaptation of Gone with the Wind is being referred to. In order to figure out its interpretations, we need to consider different versions or adaptations of Gone with the Wind. Gone with the Wind has two versions or adaptations: the 1939 film Gone with the Wind or the 2008 musical Gone with the Wind. Therefore, this question has 2 interpretations: (1) Who played Bonnie in the 1939 film Gone with the Wind? (2) Who played Bonnie in the 2008 musical Gone with the Wind? The answers to all interpretations are: (1) The 1939 film Gone with the Wind's 's character Bonnie was played by Eleanore Cammack "Cammie" King. (2) The 2008 musical Gone with the Wind's character Bonnie was played by Leilah de Meza.
Given ambiguous question, figure out its interpretations and answer them one by one. Question: What is the second largest city in the usa?
Answer: This question is ambiguous in terms of the criteria being used to determine the second largest city in the USA. In order to figure out its interpretations, measured by two criteria: population area. Therefore, this question has 2 interpretations: (I) What is the second largest city in the USA by population? (2) What is the second largest city the USA by area? The answers to all interpretations (1) The second largest city in the USA by population is Los Angeles, California. (2) The second largest city the USA by area is Juneau, Alaska.
need to consider different criteria to determine a city's 's size. City size can be
Given ambiguous question, figure its interpretations and answer them one by one. Question: When was bohemian rhapsody released single?
Answer: This question is ambiguous terms of which country's release the single being referred to. In order to figure out interpretations, we need consider different countries where Bohemian Rhapsody released. Bohemian Rhapsody was released in the United Kingdom and in the United States on different dates. Therefore, this question has 2 interpretations: (I) When was Bohemian Rhapsody released as single in the United Kingdom? (2) When was Bohemian Rhapsody released as single the United States? The answers to interpretations are: (1) Bohemian Rhapsody was released as a single in United Kingdom on 31 October 1975. (2) Bohemian Rhapsody was released as single in the United States on December 1975.
Given an ambiguous question, figure out its interpretations and answer them one by one. Question: Where do the philadelphia eagles play their home games?
Answer: This question is ambiguous in terms of which specific location or venue is being referred to. In order to figure out its interpretations, we need to consider the different possible locations or venues that could be considered the home field of the Philadelphia Eagles. These include the city, the sports complex, or the stadium. Therefore, this question has 3 interpretations: (l) What city do the Philadelphia Eagles play their home games? (2) In what sports complex do the Philadelphia Eagles play their home games? (3) What stadium do the Philadelphia Eagles play their home games? The answers to all interpretations are: (1) Philadelphia Eagles play their home games in the city Philadelphia. (2) Philadelphia Eagles play their home games in the South Philadelphia Sports Complex. (3) Philadelphia Eagles play their home games in the Lincoln Financial Field stadium.
Given an ambiguous question. figure out its interpretations and answer them one by one. Question: When did xhox one come out in australia?
Answer: This question is ambiguous in terms of which specific version of the Xbox One is being referred to. In order to figure out its interpretations, we need to consider the diffcrent versions of the Xbox One that have been released. Xhox One has two versions: the Xbox One video game console or the Xbox One X high-end model. Therefore, this question has 2 interpretations: (1) When did the Xbox One release in Australia? (2) When did the Xbox One X release in Australia? The answers to all interpretations are: (1) The Xbox One video game console was released in Australia on November 22, 2013. (2) The Xbox One X video game console was released in Australia on November 7, 2017.
Given ambiguous question, figure out its interpretations and answer them one by one. Question: When does the movie summer of 84 come out?
Answer: This question is ambiguous in terms of which release of the movie is being referred to. In order to figure out its interpretations, we need to consider different releases of the movic Summer of '84, The movie Summer of '84 is first released at the Sundance Festival hefore it's released throughout the US. Therefore, this question has 2 interpretations: (I) When did the movie Summer of '84 first release at the Sundance Festival? (2) When did the movie Summer of '84 first release throughout the US? The answers to all interpretations are: (I) Summer of '84 was released at the Sundance Festival January 22, 2018. (2) Summer of '84 released throughout the US on August 10, 2018.

---

Prompt D.7: exemplars of ASQA (cont.)
Given an ambiguuus question, figure out its interpretations and answer them one by one. Question: What was roy orbison's first number one hit?
Answer: This question is ambiguous in terms of which specific chart or region is being referred to. In order to figure out its interpretations, we need to consider the different charts and regions where Roy Orhison's 's music was popular. Roy Orbison is popular in both the US Hot 100 and Canada, and the UK and Ireland. Therefore, this question has 2 interpretations: (1) What was Roy Orbison's first number one hit in the US Hot 100 and Canada? (2) What was Roy Orhison's 's first number one hit in the UK and Ireland? The answers to all interpretations are: (I) Running Scared was the first number one hit for Roy Orbison in the US Hot 100 and Canada. (2) Only the Lonely (Know the Way Feel) was the first number one hit for Roy Orbison in the UK and Ireland.
Given ambiguous question, figure out its interpretations and answer them one by one. Question: What is the criminal's ''s name in the breakfast club?
Answer: This question is ambiguous in terms of which specific name is being referred to the character's name or the actor's
name. In order to figure out its interpretations, we need to consider hoth possibilities: the character's name the actor's
name. Therefore, this question has interpretations: (I) What is the criminal's 's character name in The Breakfast Club? (2)
What is the the name of the actor who played the criminal in The Breakfast Club? The answers to all interpretations are: (1) John Bender was the name of the criminal's 'ss character in The Breakfast Club. (2) Judd Nelson was the actor of the criminal in The Breakfast Club.
Given ambiguous question, figure its interpretations and answer them one.
Question: How many state parks there virginia? Answer:

---

Prompt D.8: exemplars of ASQA-hint
Given an ambiguous question and a hint on which aspect of the question is ambiguous, figure out its interpretations and answer them one by one.
Question: Who played bonnie in gone with the wind?
Hint: This question is ambiguous in terms of which version or adaptation of Gone with the Wind is being referred to.
Answer: In order to figure out its interpretations, we need to consider different versions or adaptations of Gone with the Wind. Gone with the Wind has two versions or adaptations: the 1939 film Gone with the Wind or the 2008 musical Gone with the Wind. Therefore, this question has interpretations: (1) Who played Bonnic in the 1939 film Gone with the Wind? (2) Who played Bonnie in the 2008 musical Gone with the Wind? The answers to all interpretations are: (1) The 1939 film Gone with the Wind's 's character Bonnie was played by Elcanore Cammuck "Cammic" King. (2) The 2008 musical Gone with the Wind's character Bonnic was played by Leilah de Meza.
Given an ambiguous question and a hint on which aspect of the question is ambiguous, figure out its interpreta- tions and answer them one by one.
Question: What is the second largest city in the usa? Hint:
This question is ambiguous in terms of the criteria being used to determine the second largest city in the USA. Answer: In order to figure out its interpretations, we need to consider different criteria to deternine a city's ' size. City size can be measured hy two criteria: population area. Therefore, this question has 2 interpretations: (1) What is the second largest (2) What is the second largest city in the USA by urea? The answers to all interpretations are: (I) The second largest city in the USA by population is Los Angeles, California. (2) The second largest city in the USA by area is Juneau, Alaska.
city in the USA by population?
Given an ambiguous question and a hint on which aspect of the question is ambiguous, figure out its interpreta- tions and answer them one by one.
Question: When was bohemian rhapsody released as a single?
Hint: This question is ambiguous in terms of which country's release of the single is being referred to.
Answer: order figure out its interpretations, we need to consider different countries where Bohemian Rhapsody is released. Bohemian Rhapsody was released in the United Kingdom and in the United States on different dates. Therefore, this question has 2 interprctations: (1) When was Bohemian Rhapsndy released a single the United Kingdom? (2) When was Bohemian Rhapsody released as single in the United States? The answers to interpretations are: (1) Bohemian Rhapsody was released as a single in the United Kingdom on 31 October 1975. (2) Bohemian Rhapsody was released as single in the United States on December 1975.
Given an ambiguous question and a hint on which aspect of the question is ambiguous, figure out its interpreta- tions and answer them one by one.
Question: Where do the philadelphia eagles play their home games?
Hint: This question is ambiguous in terms of which specific location or venue is being referred to.
Answer: In order to figure out its interpretations, we need to consider the different possible locations or venues that could he considered the home field of the Philadelphia Eagles. These include the city, the sports complex, or the stadium. Therefore, this question has 3 interpretations: (1) What city do the Philadelphia Eagles play their home games? (2) In what sports complex do the Philadelphia Eagles play their home games? (3) What stadium do the Philadelphia Eagles play their home games? The answers to all interpretations are: (1) Philadelphia Eagles play their home games in the city Philadelphia. (2) Philadelphia Eagles play their home games in the South Philadelphia Sports Complex. (3) Philadelphia Eagles play their home games in the Lincoln Financial Field stadium.
Given an ambiguous question and a hint on which aspect of the question is ambiguous, figure out its interpreta- tions and answer them one by one.
Question: When did xhox one come out in australia?
Hint: This question is ambiguous in terms of which specific version of the Xbox One is being referred to.
Answer: In order to figure out its interpretations, we need to consider the different versions of the Xbox One that have been released. Xbox One has two versions: the Xbox One video game console or the Xbox One X high-end model. Therefore. this question has 2 interpretations: (1) When did the Xbox One releuse in Australia? (2) When did the Xbox One X release in Australia? The answers to all interpretations are: (1) The Xbox One video game console was released in Australia on November 22, 2013. (2) The Xbox One X video gume console was released in Australia on November 7, 2017.
Given an ambiguous question and a hint on which aspect of the question is ambiguous, figure out its interpreta- tions and answer them one by one.
Question: When does the movie summer of 84 come out?
This question is ambiguous in terms of which release of the movie is being referred to. Answer: In order to figure out its interpretations, we need consider different releases of the movie Summer of '84. The movie Summer of '84 is first released the Sundance Festival before it's released throughout the US. Therefore, this question has 2 interpretations: (1) When did the movie Summer of '84 first release at the Sundance Festival? (2) When did the movie Summer of 84 first release throughout the US? The answers to all interpretations are: (1) Summer of '84 was released at the Sundance Festival on January 22, 2018. (2) Summer of '84 was released throughout the US August 10, 2018
Hint:

---

Prompt D.9: exemplars of ASQA-hiQnn hint (cont.)
Given ambiguous question and hint which aspect of the question is ambiguous, figure its interpretations and answer them one by onc.
Question: What was roy orbison's 's first number hit?
Hint:
This question ambiguous in terms of which specific chart region being referred
need consider the different charts and regions where Roy Orbison's music popular. Roy Orbison is popular in both the US Hot 100 and Canada, and the UK and Ireland. Therefore, this question has interprctations:
Answer: In order figure out its interpretations,
What Roy Orhison's first number hit the US Hot 100 and Canada? (2) What was Roy Orbison's 's first number hit in the UK and Ireland? The answers all interpretations was the first number hit for Roy Orbison the US 100 Canada. (2) Only the Lonely (Know the Way Feel) first number one hit Roy Orbison the UK and Ireland.
(1) Running Scared
Given an ambiguous question and a hint on which aspect of question ambiguous, figure out interpreta- tions and answer them one by one.
Question: What the criminal's name the breakfast club?
Hint: This question is ambiguous terms of which specific name is being referred to the character's name or actor's name.
Answer: In order figure out interpretations. we need to consider both possibilities: character's 's name or actor's
name. Therefore, this question has interprctations: (I) What is the criminal's 's character name The Breakfast Club? (2)
What is the the name the actor who played the criminal The Breakfast Club? The answers to all interpretations are: (1)
John Bender was the name of the criminal's 'ss character in The Breakfast Club. (2) Judd Nelson was the actor of the criminal in The Breakfast Club.
Given an ambiguous question and a hint on which aspect of the question is umbiguous. figure out its interpreta- tions and answer them one by one.
Question: How many state parks are there in virginia? Hint: This question is ambiguous in terms of the time frame or period heing referred to. Answer:

---

Prompt D.10: exemplars of WikiAsp
Generate a summary about Aslanhane Mosque including the following aspects: location, history with one aspect per line.
Location
The mosque is in the old quarter of ankara next to ankara castle. With an altitude of 947 metres (3, 107 fi) it overlooks ankara at 39"56'12 32"51'55"E.
# History
The mosque is one of the oldest mosques in Turkey still standing. It was built during the reign of Mesud II of the Anatolian Seljuks in 1290. Its architect was Ebubekir Mehmet. It was commissioned by two Ahi leaders named Hüsamettin and Hasaneddin. However, in 1330, it was repaired by another Ahi leader named Scrafettin after whom the mosque was named. After several minor repairs the mosque was restored by the directorate general of foundations in 2010-2013 term.
Generate summary about Untold Legends: The Warrio's Code including the following aspects: reception, gameplay, development with one aspect per line. Reception
The game received "mixed or average reviews" according video game review aggregator Metacritic.
Gameplay
The warrior's code is hack n' slash action role-playing game, which concentrates on action-oriented combat.
Development
a pre-order bonus, the game was shipped with a small action figure of the Guardian class.
Generate a summary about Raid on St. Augustine including the following aspects: aftermath, background with one aspect per line.
Aftermath
Once the English had gone Menéndez and the rest of the Spanish settlers returned to find smoldering ruins and very little left. He soon and begged for help from viceroy of Cuba and the settlement took a while to build itself back up. The destroyed fort was replaced with the present day Castillo de San Marcos. Background
War had already been unofficially declared by Philip offered her support to the rehellious Protestant Dutch rehels. The Queen through Francis Walsingham ordered Sir Francis Drake to lead an expedition to attack the Spanish New World in a kind of preemptive strike. Sailing from Plymouth, England, he struck first at Santiago in November 1585 then across the Atlantic at the Spanish new world city of Santo Domingo of which was captured and ransomed on 1 January 1586 and following that successfully attacked the important city of Cartagena on 19 February. Drake wanted to strike at another Spanish city on the Main before finally visiting and replenishing Sir Walter Raleigh's 's new colony of Roanoke Colony on the American East Coast. Then after this he hoped to make the Transatlantic crossing back to England. The fleet headed north, and in late April Drake put into the Spanish Cuban mainland and his men dug wells in search of fresh water and gathered supplics to help counter an outhreak of dysentery after which he moved on. The flect traveled north within sight of land on the Florida peninsula sailing pust the West coast. On 27 May 1586 as they approached further north a small fort was spotted on the shore, with small inlet close by. This was the location of St Augustine, the most northerly town in Spain's 's New World Empire, and the oldest permanent colonial settlement in North America. Drake knew of the place and was also aware of the fact that the spanish under Pedro Menéndez de Avilés had ordered all of the French Huguenot colonists that had tried to settle in the area executed. Drake decided on one final opportunity to raid and plunder, and chance to avenge his fellow Protestants.
Spain after the Treaty of Nonsuch in which Elizabeth had
Generate a summary about Lakewood (Livingston, Alabama) including the following aspects: architecture. his- tory with one aspect per line. # Architecture
The house has plan that is relatively rare in carly Alahama architecture. The plan features brick ground floor that is topped by one-and-a and-a-half--ae stories of wood frame construction. The ground floor originally contained domestic spaces, with the formal rooms the principle floor and bedrooms the upper floor. A central hallway present on all levels. The facade is five bays wide, with central entrance doors on the ground and principle floors. The bays are divided by two-story Doric pilasters, with the middle third of the facade occupied by a two-tiered tetrastyle Dorie portico. Two curved wrought iron staircases uscend from ground level the front center of the upper portico, leading the formal entrance.
History
Lakewood built for Joseph lake, native of North Carolina, by Hiram W. Bardwell, master builder. Construction was completed 1840. Located adjacent the University West Alabama, Julia Strudwick Tutwiler, Lake relative, periodically resided the house from 1881 to 1910 while she served president of the university. It was then known as Livingston Normal College. The house was extensively photographed by Alex Bush for the Historic American Buildings Survey November and December 1936. Lakewood has continued current day. The house and its surviving 10 acres (4.0 ha) grounds were listed on the Places Peril in 2012 due to immediate threat of its acquisition by developers.
owned by descendants of Lake family to
Generate a summary about Carlos Moedas including the following aspects: biography. early life. political career with one aspect per line.


Reviews:
[{"review_id": "MC4TUfGjJr", "reviewer": "Reviewer_PjRU", "summary": "", "questions": "A. Some descriptions in section 3.2 are not clear. In line 320, what does \"extract all spans from s_t with probabilities below beta\" mean? How do you define a \"span\" here? What is \"the probability of a span\"? What role is the hyper-parameter beta here?\n\nB. In section 5, you mentioned the experiments are conducted in the manner of few-shot in-context learning. Do the few-shot examples have retrieved documents with them? Or do they only have questions and answers?\n\nC. I didn't fully understand the setting in section 6.2 about the importance of forward-looking retrieval. When you retrieve with the next sentence in this experiment, did you \"first generate the next sentence, then use the complete next sentence as the query, then re-generate the next sentence\"?\n\nD. In table 4, why don't you test beta=1? From my understanding, even when beta=1, it just means masking all low-confidence tokens instead of the whole sentence, right?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback, and we are happy that you appreciated the idea!\n\n> Q1 (reason to reject 1): why FLARE-instruct is forward-looking retrieval augmentation?\n\nWe use task-agnostic instructions and 3 exemplars (in Prompt E.1) to prompt the LM to generate search terms that reflect *future* information needs, thus can be forward-looking. For example, in the second exemplar, the search term “Ghana flag red meaning” is generated *before* explaining the meaning.\n\n> Q2 (reason to reject 1): FLARE-instruct as a baseline instead of part of the proposal.\n\nWe indeed view FLARE-direct as our major technical contribution and view FLARE-instruct as an exploratory variation/ablation of active retrieval augmented generation. We will be sure to reflect this in the revised paper. Part of the reason why FLARE-instruct performs poorly is because it is hard to teach text-davinci-003 to generate proper search terms with just 3 exemplars. A promising future direction is to directly finetune LMs (such as LLaMA) to generate search terms while answering questions with high-quality training data.\n\n> Q3 (reason to reject 2): direct evidence on using information in the past is less useful.\n\nThanks for the suggestion! We will add an analysis measuring the similarity (e.g., ROUGE) between documents retrieved based on the past/next sentence and the gold next sentence.\nWe hypothesize that retrieval with the next sentence can find information more similar to the gold next sentence, therefore increasing the downstream performance, as reflected in table 3.\n\n> Q4 (reason to reject 2): a baseline that uses all the past information.\n\nIn our preliminary experiments, we tried using different numbers of past tokens (as much as 128) as retrieval queries, and we found using 16 tokens (used in the paper) generally achieved the best overall accuracy. We hypothesize that tokens far in the past might not be relevant to the current context, and using all past tokens as queries might distract the model. We will add this ablation experiment in the revised version.\n\n> Q5 (reason to reject 3): why aren't the two baselines (previous-sentence and question decomposition) tested on the other 3 datasets?\n\nWe do have the performance of the previous-sentence method on the other 3 dataset and it is slightly worse than the previous-window method, similar to the observation on 2WikiMultihopQA. For example, EM of the previous-sentence approach on StrategyQA/ASQA are 71.0/49.2. We will add these numbers in the revised version.\n\nThe question decomposition method proposed in self-ask (Press et al., 2022), as the name suggests, specializes in and is proposed for solving compositional questions. They are questions that require multiple hops of querying and can straightforwardly break down into simpler questions. Therefore we reuse their manually designed prompts on multi-hop QA (i.e., 2WikiMultiHopQA) as a critical baseline. This helps us demonstrate FLARE works well compared to specialized prompting strategies. On the other hand, when it comes to the other datasets such as summarization, the way to decompose becomes unclear, and to the best of our knowledge, there is no such existing work. This is also why FLARE is a more versatile method that does not require special designs.\n\n> Q6 (reason to reject 4): use the same model (text-davinci-003) for query generation.\n\nThanks for the suggestion! The reasons why we use text-davinci-003 for answer generation and chatgpt (gpt-3.5-turbo) for query generation are:\n1. Answer generation requires LMs to return probabilities while query generation does not. Chatgpt does not provide probabilities, so we used text-davinci-003 for answer generation.\n2. Chatgpt is much cheaper than text-davinci-003.\n3. Chatgpt is better at following instructions to generate reasonable queries.\nWe will consider adding a comparison experiment using text-davinci-003 to generate queries.\n\n> Q7 (question 1): what are spans used in query generation?\n\nWe first find all low-probability tokens (lower than beta) and consecutive low-probability tokens form spans. Queries are generated for each span separately. beta decides how much we trust the LM’s initial generation. Lower beta (e.g., 0.1) means we are lenient and only mask or generate queries for tokens with probabilities < 0.1.\n\n> Q8 (question 2): do the few-shot examples have retrieved documents?\n\nFor 2WikiMultihopQA, we follow previous baselines (Trivedi et al., 2022) to use retrieved documents for exemplars. For other datasets, due to the context length limit, we do not use retrieved documents for exemplars. This is mentioned in the appendix (line 1090-1095), and we will move it to the main body to make it clear.\n\n> Q9 (question 3): importance of forward-looking retrieval.\n\nYour understanding is correct, we first generate the next sentence, then use the complete next sentence as the query, then regenerate the next sentence.\n\n> Q10 (question 4): beta = 1 in table 4.\n\nSetting beta = 1 will mask all tokens in the sentence and nothing is left to be used as the retrieval query.\n\n> Q11: Typos in figure 2 and figure 4.\n\nThanks for pointing out these presentation issues! We will correct typos and reorganize figure 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is based on a novel idea: active retrieval-augmented generation. First, it leverages future information (the pseudo next sentence) to formulate the query instead of the past information. Second, it actively selects when to retrieve by targeting on the low-confidence tokens, which is more efficient than invoking retrieval in every sentence. These two perspectives as long as the proposed simple-and-effective solutions can bring insight to the field of long-form retrieval-augmented generation.\n2. The proposed method is proved effective in multiple different tasks. The authors did comprehensive ablation studies to prove that both \"using future information\" and \"using low-confidence tokens\" are effective in the retrieval approach.", "reasons_to_reject": "1. I didn't understand why section 3.1 (FLARE-instruct) is a type of \"forward-looking retrieval augmentation\". The description in section 3.1 and the prompt in E.1 does not convince me that this method leverages any anticipated information in the future. Given the poor performance of this method in table 1, I think this method should be listed as a baseline instead of part of the proposal.\n2. The authors are claiming that \"using information in the past\" is not useful for predicting future tokens throughout the whole paper. in addition to indirect evidence from downstream accuracy scores, is there any direct experimental evidence on this (e.g., analysis on the relationship between the retrieved documents and the next generated sentence, or to what extent the model uses the retrieved information)? Besides, it would be better to have a baseline to use all the past information (including all generated tokens and the original query itself) as retrieval query.\n3. Why aren't all three multi-time retrieval baselines in table 1 tested on all datasets? Especially for question decomposition, which is the strongest baseline in table 1.\n4. The involvement of the external model ChatGPT in query generation may bring the risk of using additional knowledge of ChatGPT to assist the pipeline. Therefore, a safer option is to use the same model (text-davinci-003) for query generation.", "questions_to_authors": "A. Some descriptions in section 3.2 are not clear. In line 320, what does \"extract all spans from s_t with probabilities below beta\" mean? How do you define a \"span\" here? What is \"the probability of a span\"? What role is the hyper-parameter beta here?\n\nB. In section 5, you mentioned the experiments are conducted in the manner of few-shot in-context learning. Do the few-shot examples have retrieved documents with them? Or do they only have questions and answers?\n\nC. I didn't fully understand the setting in section 6.2 about the importance of forward-looking retrieval. When you retrieve with the next sentence in this experiment, did you \"first generate the next sentence, then use the complete next sentence as the query, then re-generate the next sentence\"?\n\nD. In table 4, why don't you test beta=1? From my understanding, even when beta=1, it just means masking all low-confidence tokens instead of the whole sentence, right?", "paper_topic_and_main_contributions": "This paper proposed a novel method for long-form retrieval-augmented generation. It is mainly motivated by two weaknesses in previous retrieval-augmented methods:\n1. The past generated tokens do not necessarily reflect what information is beneficial for generating the next token. Therefore, this work proposed to first generate a \"pseudo\" next sentence and used this sentence as the retrieval query.\n2. Using the past context within a sliding window does not always yields good queries for retrieval. This work proposed an active retrieval method which focuses on the low-confidence tokens in the generated sentence.\nBesides, the authors also explored two methods of formulating the retrieval query using the next generated sentence with low-confidence tokens: (1) masking out low-confidence tokens with a probablity, or (2) re-generate a question with ChatGPT targeting on low-confidence spans.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Figure 2 caption has grammar errors: \"implicit the explicit query formulation\". Should it be \"implicit and explicit\"?\n2. Two datasets in figure 4 should be plotted separately because they are not at the same scale. The red line is almost flat in this figure."}}, {"review_id": "2S2pBZGuCJ", "reviewer": "Reviewer_uvL6", "summary": "", "questions": "1. For strategyQA data set, why only 229 data were extracted for testing?\n2. What is the length of each wikipedia documentation chunk？\n3. Is there any treatment for the results returned by bing search, such as length truncation？\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback, and we are happy that you appreciated the thorough experiments!\n\n> Q (reason to reject 1): baselines (previous-sentence and question decomposition) are not tested on the other 3 datasets.\n\nWe do have the performance of the previous-sentence method on the other 3 dataset and it is slightly worse than the previous-window method, similar to the observation on 2WikiMultihopQA. For example, EM of the previous-sentence approach on StrategyQA/ASQA are 71.0/49.2. We will add these numbers in the revised version.\n\nThe question decomposition method proposed in self-ask (Press et al., 2022), as the name suggests, specializes in and is proposed for solving compositional questions. They are questions that require multiple hops of querying and can straightforwardly break down into simpler questions. Therefore we reuse their manually designed prompts on multi-hop QA (i.e., 2WikiMultiHopQA) as a critical baseline. This helps us demonstrate FLARE works well compared to specialized prompting strategies. On the other hand, when it comes to the other datasets such as summarization, the way to decompose becomes unclear, and to the best of our knowledge, there is no such existing work. This is also why FLARE is a more versatile method that does not require special designs.\n\n> Q (question 1): 229 examples for strategyQA.\n\nWe follow previous papers in using the development set of StrategyQA (10% of the full dataset) to run experiments.\n\n> Q (question 2): length of each wikipedia documentation chunk.\n\nWe use the Wikipedia corpus released by DPR (Karpukhin et al., 2020) where each chunk has about 100 words.\n\n> Q (question 3): bing search results truncation.\n\nWe use the snippets returned from bing which are relatively short so we do not perform additional truncation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a generic retrieval augmented LLMs framework, which can actively decide when and what to retrieve across the course of the generation.\n2. The experiment is thorough. The authors verified the effectiveness of the method on different types of tasks and made detailed experimental analysis.\n3. The article is clear and easy to follow.", "reasons_to_reject": "1. On StrategyQA, ASQA, ASQA-Hint, and WikiAsp tasks, the authors only compared FLARE to simple baselines, not to other baseline methods.", "questions_to_authors": "1. For strategyQA data set, why only 229 data were extracted for testing?\n2. What is the length of each wikipedia documentation chunk？\n3. Is there any treatment for the results returned by bing search, such as length truncation？\n", "paper_topic_and_main_contributions": "This paper focuses on two important questions of retrieval augmented large language models: when and what to retrieve. Different from the previous single-time retrieval and inflexible multi-time retrieval, the authors propose an active retrieval method to trigger retrieval at the right time. They also propose a future-aware retrieval approach to use temporary next sentences as queries to aid future generations. Empirical results show that FLARE outperforms baselines on all four datasets.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eBcE8kiaY1", "reviewer": "Reviewer_6yUm", "summary": "", "questions": "1. Can you report the latency and efficiency between your approach and baselines?\n2. What's the threshold value (\\beta and \\theta) for different tasks? Do you choose it by empirical experiments on different tasks?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback!\n\n> Q1 (reason to reject 1): Novelty of FLARE-instruct and FLARE-direct.\n\nFLARE-instruct is indeed inspired by Toolformer (as mentioned in line 221) but one significant difference is that we aim to generate multiple queries in long-form generation scenarios. Regarding the novelty/contribution of this work, we view FLARE-direct as our major technical contribution and view FLARE-instruct as an exploratory variation/ablation. We discussed several existing works in the related work section (in the appendix E) and we will move this section to the main body in the revised version. As far as we know, FLARE-direct is the first method that actively decides when to retrieve with probabilities and what to retrieve with forward-looking. We are happy to cite and compare other baselines if the reviewer can provide relevant references.\n\n> Q2 (reason to reject 2): efficiency and latency of the proposed methods.\n\nThanks for the suggestion! We agree that iterative retrieval during generation increases latency compared to single-time retrieval, which we mentioned in the “efficiency” paragraph in appendix A. Generation latency is correlated with the number of retrieval triggered during generation. Since our method actively decides when to retrieve based on probability, it is faster than other multi-time retrieval baselines that trigger retrieval regularly (every sentence/few tokens). We do plan to report latency, but since we access OpenAI LMs through API calls, the measured latency might be overestimated because of the overhead of API calling. One future direction is to implement FLARE on top of LLaMA, where we have full control of the generation process and the latency measurement is more accurate.\n\n> Q3 (question 2): threshold value (beta and theta) for different tasks.\n\nWe choose these threshold values (reported in table 8 in the appendix) based on the performance on the development set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Their proposed approach FLARE demonstrate that actively decodes when and what to retrieve throughout the generation process can bring gains for multiple QA tasks.\n2. They designed two approaches that mask sentences as implicit queries and generate questions as explicit queries for iteratively retrieval to overcome the errors that might exist in the next sentence.", "reasons_to_reject": "1. The first approach FLARE with Retrieval Instruction is similar to Toolformer, while the second Direct FLARE that utilize the confidence to decide retrieve is straightforward and might have been explored. The author should do some existing literature.\n2. The author should report the efficiency and latency since the proposed approach requires multiple iterative retrievals.", "questions_to_authors": "1. Can you report the latency and efficiency between your approach and baselines?\n2. What's the threshold value (\\beta and \\theta) for different tasks? Do you choose it by empirical experiments on different tasks?\n", "paper_topic_and_main_contributions": "This paper proposes a novel approach, named Forward-Looking Active Retrieval Augmented generation (FLARE), to reduce the hallucination for long-form text generation. Specifically, they iteratively generate a temporary next sentence and check whether it contains low-probality tokens. If so, they retrieve again with the reformulated query and regenerate the sentence. FLARE can be used for any existing LMs during inference time without additional training. On four QA datasets (MultihopQA, StrategyQA, ASQA, WikiAsp), they demonstrate the effectiveness of FLARE over all baselines. However, according to the author, their approach did not provide significant gains on Wizard of Wikipedia and ELI5. Overall, this paper is well-written and brings insights to the community that forward active retrieval can be useful for QA tasks and reduce hallucination.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method, Active Retrieval, which prompts an LLM to trigger iterative retrieval while generations. Their experimental results show effectiveness on Multi-hop QA, long-form QA, and aspect-based summarization datasets. While I agree with some of the limitations shared by the reviewers (e.g., limited novelty given Toolformer, applicability to other LLMs, evaluations on focused tasks), overall the idea is exciting and the experiments are well conducted. Therefore, I recommend for acceptance.
Decision: Accept-Main