Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Authors: Zhihong Shao
Yeyun Gong
yelong shen
Minlie Huang
Nan Duan
Weizhu Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: retrieval-augmented language model
prompting
retrieval

Abstract: Retrieval-augmented generation has raise extensive attention as it is promising to address the limitations of large language models including outdated knowledge and hallucinations.
However, retrievers struggle to capture relevance, especially for queries with complex information needs.
Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation.
In this paper, we show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner:
a model's response to a task input shows what might be needed to finish the task, and thus can serve as an informative context for retrieving more relevant knowledge which in turn helps generate a better response in another iteration.
Compared with recent work which interleaves retrieval with generation when completing a single output, Iter-RetGen processes all retrieved knowledge as a whole and largely preserves the flexibility in generation without structural constraints.
We evaluate Iter-RetGen on multi-hop question answering, fact verification, and commonsense reasoning, and show that it can flexibly leverage parametric knowledge and non-parametric knowledge, and is superior to or competitive with state-of-the-art retrieval-augmented baselines while causing fewer overheads of retrieval and generation.
We can further improve performance via generation-augmented retrieval adaptation.

Enhancing Retrieval-Augmenned Large Language Models with Iterative Retrieval-Generation Synergy
Zhihong Shao', Yeyun Gong", yelong shen', Minlie Huang"; Nan Duan', Weizhu Chen
The CoAI Group, DCST, Institute for Artificial Intelligence. State Key Lab of Intelligent Technology and Systems,
Beijing National Research Center for Information Science and Technology,
Tsinghua University, Beijing 100084, China
2 Microsoft Research Asia Microsoft Azure Al
szh 199mails..... tsinghua. edu. cn aihuangêtsinghua. cnd. edu.cn
Abstract
Retrieval-augmeented LLMs, therefore, have raised widespread attention L.LM outputs can be poten-
Retrieval- augmented generation has raise exten- sive attention as it is promising to address the limitations of large language models including outdated knowledge and hallucinations. How- ever, retrievers struggle to capture relevance, especially for queries with complex informa- tion needs. Recent work has proposed to im- prove relevance modeling by having large lan- guage models actively involved in retrieval, i.e., to guide retrieval with generation. In this pa- per, we show that strong performance can be achieved by method we call (TER-RETGEN, which synergizes retrieval and generation an task
tially grounded on external knowledge.
Previous retrieval- augmented LMS (Izacard et al., 2022b; Shi al., 2023) typically adopted one-time retrieval, i.e., to retrieve knowledge us- ing only the task input (e.g.. a user question for open-domain question answering). One-time re- trieval should suffice to fulfill the information needs if they are clearly stated in the original input, which is applicable to factoid question answering (Kwiatkowski et al., 2019) and single-hop fact ver- ification (Thorne al., 2018), but not tasks with complex information needs, e.g.. multi-hop rea- soning (Yang et al., 2018) and long-form question answering (Fan et al., 2019).
iterative manner: model's response input shows what might be needed to finish the task, and thus can serve as an informative context for retrieving which in turn helps generate better response another iteration. Compared with recent work which interleaves retrieval with gener- ation when completing a single output, ITER- RETGEN processes all retrieved knowledge as a whole and largely preserves the flexibility in generation without structural constraints. We evaluate TEER-RETGEN on multi-hop question answering. fact verification, and commonsense reasoning, and show that it can flexibly lever- age parametric knowledge and non parametric knowledge, and is superior to or competitive with state-of-the-art retrieval-auge -ugmentee base- lines while causing fewer overheads of retrieval and generation. We can further improve per- formance via generation-augmented retrieval adaptation.
relevant knowledge
To fulfill complex information needs, recent work proposes to gather required knowledge multi- ple times throughout the generation process, using partial generation (Trivedi et al., 2022a; Press et al., 2022)) or forward-looking sentence(s) (Jiang et al., 2023) as search queries. However, such structured workflows of interleaving retrieval with generation have the following limitations: (1) as intermediate generation is conditioned on knowledge retrieved before, with no awareness of knowledge retrieved afterwards, they fail to process all retrieved knowl- edge as a whole during the generation process; (2) they require multi-round retrieval to gather a com- prehensive set of knowledge, and may frequently change the prompts by updating newly retrieved knowledge, thus increasing the overheads of both retrieval and generation.
1 Introduction
In this paper, we find it simple but effective to enhance retrieval-augmeeeee LLMs through itera- tive vetrieval-genencc Fig 1). ITER-RETGEN iterates retrieval-augmeented generation and generation-auugentie retrieval: Retrieval-augmented generation outputs a response to a task input based on all retrieved knowledge (initially using the task input as the query). This
Generative Large Language Models (LLMs) have powered numerous applications, with well- perceived utility. Despite being powerful, LLMs lack knowledge that is under-represented in their training data, and are prone to hallucinations, es- pecially in open-domain settings (OpenAI, 2023).
synergy (ITER-RETGEN
*Corresponding author: Minlie Huang.

---

output shows what might be needed fulfill the task, and thus can serve as an informative context to retrieve more relevant knowledge, i.e., generation- augmented retrieval. The newly retrieved knowl- edge can benefit another iteration of retrieval- augmented generation. We can also leverage model generations to adapt retrieval, by distilling knowl- edge from a re-ranker with access to model genera- tions dense retriever with access to task inputs only, which may be beneficial scenarios where user inputs can be easily collected, but relevant knowledge or desirable outputs are not annotated.
parametric knowledge and non-parametric knowledge effectively. ITER-RETGEN con- sistently outperforms Self-Ask on question answering tasks, regardless whether in- context non-parametric knowledge mentions the answers or not.
2 Related Work
In recent months, there has been a surge in LLM- powered applications, such as ChatGPT, Bing Chat, and CoPilot (Chen et al., 2021). While showing an unprecedented level of performance, LLMs are sub- ject to the following limitations: (1) Due to high demand for compute and data, it remains an open research question to continually update LLMs both efficiently and effectively (Scialom et al., 2022); (2) LLMs also tend to hallucinate (OpenAI, 2023), i.e., generating plausible but non-factual texts. To alleviate these issues, there growing trend of augmenting LLMS with tools (Mialon et al., 2023; Gou et al., 2023), c.g., a code interpreter (Gao et al., 2022b; Shao et al., 2023) or a search engine (Nakano et al., 2021). in an attempt to offload sub- tasks to more qualified experts, or to enrich the input context for LLMs by providing more relevant information.
We evaluate our method on three tasks, includ- ing multi-hop question answering, fact verification, and commonsense reasoning. Our method prompts LLM to produce chain of reasoning steps fol- lowed by the final answer under few-shot set- ting. For in-context demonstrations, we focus on problem-solving and follow Wei et al. (2022) to annotate chains of thoughts, without explicitly con- sidering how generation-augmentte retrieval might be affected, which makes it conceptually simple and easy to implement. Our method achieves up to 8.6% absolute gains over previous state-of- tate--ffthhe- art aatrivval-augmente methods on four out of six datasets while being competitive the remaining two. According our experiments, generation gen- erally benefits from more iterations, with two itera- tions giving the most performance gains. One may customize the performance cost tradeoffs by choos- ing an appropriate number of iterations. We can further improve performance and also reduce itera- tions via the aforementioned generation-augment retrieval adaptation.
Retrieval augmentation is a mainstream direc- tion to connect L.MMs to the external world. Previ- ous aerreval--uugeentee LMs (Izacard and Grave, 2021; Shao and Huang, 2022) typically receive re- trieved knowledge in passive way: knowledge is retrieved based the task inputs without LMs' intervention. As it difficult for a retriever cap- ture relevance, especially in the zero-shot setting, recent work shows a shift towards having LLMs actively involved in retrieval to improve relevance modeling, e.g., to provide specific context for retrieval with model generations (c.g., generated search queries (Nakano et al., 2021; Press et al., 2022; Yao et al., 2022), partial generation (Trivedi et al., 2022a), forward-looking sentences (Jiang et al., 2023)). Khattab al. (2022) proposed a DSP programming framework that supports vari- ous retrieval- aaggeeedde methods.
We summarize our findings as follows:
Automatic metrics such as exact match can significantly underestimate the performance of LLMs in question answering tasks. More- over, improvements in exact match do not always reflect improvements in generations. Evaluation using LLMs may be more reliable.
ITER TEER-RETGEN is superior to competi- tive with state-of-the-art retrieval-augmenntr methods, while being simpler and causing fewer overheads of retrieval and generation. With cnneration-augmentte retrieval adapta- tion, we can further improve performance and also reduce overheads (by reducing itera- tions).
Recent work interleaves retrieval with generation when completing single output. Such structured workflow may reduce the flexibility in generation (Yao et al., 2022). ITER-RETGEN avoids interrupt- ing generation with retrieval, but iterates retrieval and generation, i.c., to leverage the complete gen- eration from the previous iteration to retrieve more
It is desirable for an LLM to leverage both

---

Question:
Question: 9
What the heighr the player who won the 2015 AFI. Rising award!
What height the player who won the 2015 AFL Rising Star award?
Retrieval: Do
Generation-Angmentt Retrieval:
Paile
Tidl: 2015 AFL Rising Star Context: The NABAFL AFL Rising Star award given annually to stand out young player in Australian Football League. Hogan of Melbourne
Title: Jesse Hugan Context: Jesse Hogan playing for the Mellourne Foothall Club. A key forward, Hogan 1.95
tall made AFL debut in the 2015 scason and won the Ron Evans Medal as the AFL Rising Star
Retrreval-Augmennee Generation: P/
Eetrieval-Aummenne Generation: S'ull ll 1 30
The 2015 AFL Rising Star award was won by Jesse Hogan of Melbourne. Jesse Hogan professional Australian rules footballer. He is 198 em tall. So the
The 2015 AFL Rising Star award was won by Jesse Hogan of Melbourne. Jesse Hogan is 1.95 m tall.
198
So the answer 1,95
Iteration 1
Iteration
Figure VEER-RETGEN iterates retrieval and generation. each iteration, ITER-RETGEN leverages model output from the previous iteration specific context help retrieve more relevant knowledge, which may help improve model generation (c.g., correcting the height of Hesse Hogan in this figure). We only show two iterations in this figure for brevity. Solid arrows connect queries to the retrieved knowledge, and dashed arrows denote retrieval-augmented generation.
relevant information which helps improve genera- tion the next iteration. TTER-RET RETGEN also has the advantage of processing all retrieved knowl- edge as whole during the generation process, and is conceptually simpler and nasier-t-iimpleentt while being empirically strong in multi-hop ques- tion answering, fact verification, and commonsense reasoning.
top-k paragraphs, and then (2) prompt an LLM M to produce an output y/, with both the retrieved paragraphs (denoted as D D 1g) and & integrated into the prompt. Therefore, each iteration can be formulated as follows:
y MM wMMMil(((lpotiiie (ppplpopopptppaalllrro 4)), VISIST (1)
The last output yr will be produced as the final response.
closely related work called GAR (Mao et al., 2021) augments queries with generated background information. HyDE (Gao et al., 2022a) also shares a similar spirit, but focuses on zero-shot informa- tion retrieval, and proposes to first prompt an LLM to produce "hypothetical" paragraphs that cover the information needed to answer given question, and then use the generated paragraphs to retrieve the real ones. RepoCoder (Zhang et al., 2023) focuses on repository-level code completion, and proposes a 2-iteration retrieval-generation paradigm where the second iteration leverages the intermediate code completion for retrieval. By contrast, we propose to synergize retrieval and generation with ITER- RETGEN on various natural language tasks, and explore how we can further adapt retrieval with model generations.
3.2 Generation-Augmente Retrieval
There are many natural language tasks with com- plex information needs. For example, in open- domain multi-hop question answering, specific in- formation needs may manifest themselves only after correctly answering some prerequisite sub- questions. In other words, there may exist semantic gaps between the original question q and its sup- porting knowledge, which can be effectively addressed by retriever with a representation bot- tleneck. In the first iteration, we can retrieve knowl- later iterations,
edge with only the question the L.M output from the previous iteration, though having no guarantee of correctness, shows what might be needed to answer the question, and thus can be leveraged to bridge the semantic gaps; with improved retrieval, LLM potentially pro- duce better output.
3 Iterative Retrieval-Generation Synergy
3.1 Overview
3.3 Retrieval-Augmentee Generation
Given question 9 and retrieval corpus D {d} where d is a paragraph, ITER-RETGEN repeats retriavnl-generation for T' iterations; iteration t, we (1) leverage the generation Vt-1 from the previous iteration, concatenated with g, to retrieve
In each iteration, generate an output using Chain-of-Thoughe prompting except that we also prepend retrieved knowledge to the question q. Though there may exist more advanced prompting

---

variants, e.g., incorporating previous generations into the prompt enable direct refinements, leave the explorations for future work, and focus on investigating the synergy between retrieval and generation in straightforward manner.
et al., 2022b), and Bamboogle (Press et al., 2022). On MuSiQue, we followed Press et al. (2022) use only 2-hop questions; (2) Fact Verification, including Feverous (Aly et al., 2021); (3) Com- monsense reasoning. including StrategyQA (Geva et al., 2021). Examples are presented in Table 1.
3.4 -neeration-Aummentedd Retrieval Adaptation
We used the October 2017 (Yang et al., 2018) and the December 2018 (Karpukhin et al., 2020) Wikipedia dump as the retrieval corpus for Hot- PotQA and 2WikiMultHHoPQA respectively, and used the December 2021 Wikipedia dump (Izacard et al., 2022b) for the other datasets.
Model generations not only provide specific con- texts for retrieval, but can also be leveraged to op- timize the retriever, that information needs in question can be better captured by the retriever.
Dense Retriever We adopted dense retrieval in our experiments. Given a dense retriever parametrized by - (.q. fa where fg and fa de- note parameters of the query encoder and the para- graph encoder, respectively, the similarity score between a query and paragraph is calculated as the inner product of their encoded vectors:
4.2 Evaluation Settings
We conducted evaluations on all 125 questions from Bamboogle, the first 500 questions from the train set of StrategyQA, and the first 500 questions from the development sets of the other datasets. All methods evaluated under the 3-shot setting, sharing the same questions in demonstrations.
so df (E(g; 0,), E(d; Oa))
(2)
Evaluation metrics are exact match (EM) and F1 for multi-hop question answering datasets, and ac- curacy for both fact verification and commonsense reasoning datasets. For more robust evaluation, also evaluate the correctness of model outputs using text-davinci-001. the resulting metric de- noted as Acc The prompt used for evaluation is as follows, where (question), (model output]. and { (answer) are placeholders.
Re-ranker A re-ranker, parametrized by o, out- puts the probability of paragraph being relevant a query; we denote the probability as (g d).
Distillation A re-ranker is typically better at cap- turing relevance between a query and paragraph than retriever. Therefore, we distill knowledge from re-ranker to retriever. To help the retriever better address the semantic gaps between a ques- tion and its supporting knowledge, allow access to y1 for the re-ranker (where y1 the LLM out- put from the first iteration). We optimize only the query encoder of the retriever using the following training objective:
Prompt for Evaluating the Correctness of Model Output
In the following task, you are given Question, a model Prediction for the Question, and Ground-truth Answer to the Question. You should decide whether the model Prediction implies the Ground-truth Answer.
0 arg min KL(Po(-ly1.g) Po(ll))
exp(so(v1 d)/r)
Pa(d)n Ever vi exp(so(v1))) (.'// (3)
Po(dlg) Erep iv exp(=o(g, d'/T)
exp(sa(q,d)/1
Question
[question]
where KL(---) denotes the KL divergence between two probabilistic distributions.
Prediction
(model output)
4
Experiments
Ground-truth Answer (answer)
4.1 Datasets
We experimented on six datasets of three reason- ing tasks: (1) Multi-hop question answering. including HotPotQA (Yang et al., 2018), 2Wiki- MultiHopQA (Ho et al., 2020), MuSiQue (Trivedi
Does the Prediction imply the Ground-truth Answer? Output Yes or No:

---

Datasets
Example
HotPoIQA
What the name of this American musician, singer, actor, comedian, and songwriter, who worked with Modern Records and born December 5, 1932?
2WikiMultiHopQA
Which film came out first, Blind Shaft or The Mask Of Fu Manchu?
MuSiQue
In which year did the publisher of In Cold Blood form?
Bamboogle
When did the first prime minister of the Russian Empire come into office?
Feverous
Is it true that Based on the same platform the Chevrolet Sail, the Baojun 310 was launched on 2017 Beijing Auto Show where the price ranges from 36.800 yuan to 60.800 yuan?
StrategyQA
Is it common to see frost during some college commencements?
Table I: Example questions from six datasets.
4.3 Baselines
marize retrieve knowledge for subsequent use. the prediction stage, DSP generates the answer us- ing CoT based on the summarized knowledge and retrieved documents.
Direct Prompting (Brown et al., 2020) prompts an LLM directly generate the final answer without an explanation. When augmenting Direct prompt- ing with retrieval, we used the question to retrieve knowledge which will be placed before the ques- tion in the prompt.
4.4 Implementation Details
We used text-davinci-003 version of Instruct- GPT (Ouyang et al., 2022) as the backend LLM. We also present experiments using the open-source Llama-2 models (Touvron et al., 2023) in Ap- pendix A. All experiments used greedy decoding. Contriever-MSMARR
CoT Prompting (Wei et al., 2022) prompts an LLM to generate natural language reasoning steps followed by the final answer.
ReAct (Yao et al., 2022) interleaves reasoning, ac- tion, and observation steps, until reaching the ac- tion of finalizing an answer. An action can be either generating query to search for informa- tion finalizing answer. An observation is the concatenation of retrieved paragraphs.
(Izacard et al., 2022a) was used for retrieval. We retrieved top-5 paragraphs for each query. We allowed most interactions with retrieval for ReAct and Self-Ask. We adapted the implementation of DSP use the same gen- eration model and retrieval systems as the other methods.
Self-Ask (Press et al., 2022) interleaves (i) follow- up question generation, (ii) retrieval using the follow-up, and (iii) answering the follow-up condi- tioned on the retrieved knowledge, until no more follow-up questions are generated and the LLM gives an answer to the original question. We fol- lowed (Yoran et al., 2023) to prepend newly re- trieved paragraphs to the original question. On our evaluated tasks, Self-Ask is conceptually similar to ReAct, with the main difference being that Self- Ask accumulates retrieved knowledge before the original question in the prompt, while ReAct places retrieved knowledge right after its query. Self-Ask and IRCoT (Trivedi al., 2022a) also share the spirit of synergizing reasoning and retricval.
Note that the first iteration of ITER-RETGEN is CoT prompting with retrieval augmentation. There- fore, ITER-RETGER RETGEN and CoT prompting share the same annotated in-context demonstrations. All prompts are presented in the Appendix.
4.5 Main Results
As shown by Table 2, ITER-RETGEN (T 2 2) achieve significantly higher Acc! than retrieval- augmented baselines HotPotQA, 2WikiMulti- HopQA, Bamboogle, and StrategyQA, while being competitive with the best method (i.e., Self-Ask) on MuSiQue and Feverous.
When increasing the number of iterations for TTER-RETGELL performance generally improves, with the second iteration giving the greatest boost.
DSP (Khattab et al., 2022) comprises multi-hop retrieval stage and answer prediction stage. For each hop within the retrieval stage, the model is prompted to generate search queries and to sum-

---

HotPotQA
2WikiMultiHoPQA
MuSiQue
Bamboogle
Feverous
StrategyQA
Method
EM
Acc
EM
F1
Acc
EM
FI
Acc
EM
F1
Acc
Acc
Acc
Acc
Ace*
Without Retrieval
Direct
21.9
36.8
44.8
21.3
29.2
33.9
7.0 18.7 15.8 11.2
24.4
28.0
60.1
60.1
66.5
66.7
CoT
30.0
44.1
50.0
30.0
39.6
44.0
19.4 30.9
28.6
43.2
51.1
60.0
59.8
59.8
71.0
71.0
With Retrieval
Direct
31.6
44.7
53.3
27.3
35.4
43.6
13.9
28.2
26.5
17.6
31.8
43.2
69.8
69.8
65.6
65.6
ReAct
24.9
44.7
61.1
28.0
38.5
45.9
23.4
37.0
37.9
21.8
31.0
40.3
66.4
66.4
66.9
66.9
Self-Ask
36.8
55.2
64.8
37.3
48.8
55.9
27.6
41.5
42.9
31.5
41.2
54.8
70.7
70.7
70.2
70.2
DSP
43.8
55.0
60.8
ITER- RETGEN
39.2
53.9
65.5
33.7
45.2
55.4
24.2
38.6
38.1
36.8
47.7
57.6
67.0
67.0
72.0
72.0
2 44.1
58.6
71.2
34,9
47.0
58.1
26.4
41.1
41.0
38.4
48.7
59.2
68.8
68.8
73.0
73.0
TTER-RETGEN3 45.2
59.9
71.4
34.8
47.8
58.3
25.7
41.4
40.8
37.6
47.0
59.2
69.0
69.0
72.3
72.3
VTER-RETGEN 4 45.8
61.1
73.4
36.0
47.4
58.5
26.7
41.8
40.8
38.4
49.6
60.0
71.5
71.5
73.8
73.8
TEERREETEN 5 45,2
60.3
72.8
35.5
47.5
58.8
25.7
40.7
39.6
39.2
49.7
60.8
70.3
70.3
73.2
73.2
ITE-REEGEEN& 6 45.9
61.0
73.3
35.5
48.1
59.4
25.9
40.5
39.8
40.0
50.0
59.2
70.9
70.9
72.4
72.4
ITER TTER-RETGEN 7 45.1 60.4
72.9
35.5
47.4
58.4
26.1
42.0
41.0
40.0
50.7
60.8
70.5
70.5
74.1
74.1
Table 2: Evaluation results multi-hop question answering, fact verification, and commonsense reasoning datasets. Acct is the accuracy of model outputs evaluated with text-davinci-O00 For ITER- REER-R we evaluated LLM outputs in different iterations (up to iterations). Underlined metric values are higher than those of Self-Ask.
HotPoIQA
2WikiMuliHHopQA
MuSiQue
Bamboogle
Feverous
Strategy QA
Method
#APII # Doc # API
API Doc
API
Doc
#
# Doc
# API Doc
ReAct
14.3
3.0
15.0
2.9
14.4
2.8
14.1
2.1
10.6
2.8
14.2
Self-Ask
3.2
16.0
3.2
15.9
3.0
14.8
3.0
14.9
2.3
11.3
3.0
15.1
Table 3: Average numbers of API calls to text-dave -ext-davinci-003 and retrieved paragraphs for ReAct and Self-Ask. Note that ITER-RETGEN (T' = 2) achieves significantly higher or competitive Acc! with fewer API calls (i.e., 2) and fewer retrieved paragraphs per iteration, 10 total).
It is worth noting that, as shown by Table 3, ITER-
DSP significantly. We manually investigate 10 ran- dom questions where DSP fails but STER-RETGER provides correct answers. On 40% of them, DSP fails to retrieve documents that cover the correct answers, while 50% of them, the summarized knowledge is misleading, e.g., for the question "What occupation Chris Menges and Aram Avakian share?". DSP generates a wrong sum- mary "Chris Menges and Aram Avakian are both members of the American and British Societies of Cinematographers.. while the retrieved documents mention that Aram Avakian is film editor and director, and only Chris Menges with the American and British Societies of Cine- matographers.
RETGEN (T = 2) is superior to or competitive with ReAct and Self-Ask using fewer API calls to the LLM (i.c.,3 2) and fewer retrieved paragraphs (i.e., 5 per iteration, 10 in total). ITER-RETGEN is also conceptually simple, which to iterate retrieval- augmented CoT, without complex processing.
We also compared ITER-RETGEN with DSP which also generates the answer using CoT based on retrieved knowledge but differs information collection and processing. In each iteration, ITER- RETGEN retrieves knowledge based on (1) the ques- tion and (2) the previous model output which shows what may be needed to answer the question. With tend to
the number of iterations increasing, obtain more comprehensive and relevant set of knowledge. Besides, unlike DSP, we do not summa- rize the retrieved documents for answer generation, and thus will not introduce summarization errors. As shown in Table 2, -ERTEEN outperforms
Acct is Reliable Metric To investigate how liable Acc is, we focused on model outputs where EM and Acct disagree, and manually checked which metric gives more correct labels. On cach of the four multi-hop question answering datasets,

---

Dutaset
HotPotQA
Feverous
Retriever
Original Distilled w/o yi Distilled yi Original
Distilled w/o yi
Distilied y/
IEER-RETGEN 1
65.5
67.1
67.7 75.7
67.0
67.3 68.1
70.7
HEER-RETGNN 2
71.2
75.2
68.8
69.5
Table 4: Effect of using LLM generation y1 on optimizing dense retriever. We evaluated (TER-RETGEN on HotPorQA and Feverous in terms Acc!.
Subset
CoT V
CoT X
w/ Answer Retrieved
w/o Answer Retrieved
Method
Self-Ask ITER-RETGEN Self-Ask ITER-RETGEN Self-Ask
ITER-RETGEN
Self-Ask
ITER-RETGEN 2
HotPorQA
77.5
88.0
52.0
54.4
78.1
86.9
29.9
40.8
2WikMuuliHoPAA
68.8
78.2
46.2
42.0
73.1
77.2
30.1
42.3
MuSiQue
68.5
66.9
32.6
30.7
72.9
78.9
12.2
22.9
Bamboogle 73.0
77.3
28.0
32.0
76.2
82.2
32.8
46.2
Table Comparisons between Self-Ask and TEER-RETGEN = 2) on different subsets, in terms of Acc CoT v is the subset of questions which CoT answers correctly without retrieval; CoT X is the complement. w/ Answer TTER-RETGEN successfully retrieves
Retrieved is the subset of questions for which a method (Self-Ask paragraphs that mention the answers; w/o Answer Retrieved is the complement. WEER-RETGEN tends to be much better at preserving the LLM's performance questions that can be solved using CoT without retrieval, and is consistently more accurate regardless of whether retrieved knowledge mentions the answers or not.
we randomly sampled 20 model outputs from the second iteration of REEGEE, resulting in 80 samples in total. For 98.75% of samples, EM is 0 and Acc! 1, while Acc! gives the correct la- bels 97.5% of the time, indicating that EM severely underestimates model performance. We also car- ried out the same evaluation for Self-Ask, and Acct gives the correct labels 98. 75% of the time when it is inconsistent with EM.
Generation Benefits Retrieval Adaptation To investigate how LLM outputs can be leveraged for retrieval adaptation, we experimented on Hot- PotQA and Feverous. Specifically, on each dataset, we sampled 9,000 random questions from the train set for training, and 1,000 for validation. We ap- plied ITER-RETGEN for one iteration, and used the model outputs /1 for retrieval adaptation as in Sec- tion 3.4. We used TART (Asai al., 2022) the re-ranker, and distilled knowledge from TART the dense retriever for no more than 1,000 steps. Batch size was 32 and learning rate was 1e-5, We used the retriever checkpoint with the lowest distil- lation loss.
Acct offers the advantage of identifying model outputs that are semantically correct, even if their surface forms differ from the annotated an- swers. As an illustration, for the question "Whi ch country Jan Baptist Van Rensselaer's father is from?", the annotated answer is Dutch, while the model prediction is Netherlands, which is correct terms of Acct but penalized by EM. Notably, TTER-RETGEN (T 2 2) consistently demonstrate lower EM but higher Acc! than Self- Ask 2WikiMultiHopQA, suggesting that en- hancements in EM do necessarily reflect im- provements the quality of generated answers.
As shown by Table 4, retrieval adaptation en- ables ITER-RETGEN achieve significantly higher Acct with fewer iterations. We also demonstrated the benefits of using y1 for adaptation by showing its improvements over a variant which only dif- fers in that the re-ranker has no access to yı the training objective of this variant can be obtained by removing all ø1 notations in Eq. 3.
4.6
Ablation Study
Iteration
2
3
4
6
7
HotPotQA
49.5 66.1
65.7
66.5
66.7
66.7
67.1
4.6.1
Generation Augments Retrieval
2WikiMuliHopOA
29.0
45.2
46.2
46.7
45.8
45.8
46.5
Table 6 shows the answer recall of retrieval in dif- ferent iterations. The first iteration uses only the questions for retrieval and suffers from low an- swer recall. the second iteration, retrieval, aug- mented with the LLM output from the first iteration,
MuSiQue
18.6
32.3
32.3
33.7
32.7
32.9
Bamboogle
20.8 36.0 36.8
36.0
35.2
36.0
36.0
Table 6: Answer recall of retrieved paragraphs in differ- ent iterations for STER-RETGEN:

---

HotPotQA
Question: The urena where the Lewiston Mainciaes played their home gumes can seat how mant peuple?
Answer: 3,677 seated
Iteration
Retrieved Knowledge
(1) Title: Lewiston Maineiacs Context: The Lewiston Maineiacs were a junior ice hockey team The team played its home
games at the Androscoggin Bank Colisée
(2) Title: Bangor Auditorium Context: ... a 5,948 seat multi- purpise arena located in Bangor, Maine
Generation
The Lewiston Maineiacs were junior hockey team of the Quebee Major Junior Hockey League based Lewiston, Maine. The team played its home games the Androscoggin Bank Colisée. The Androscoggin Bank Colisée can seat up 5,948 people.
So the answer 5,948
Iteration 2
Retrieved Knowledge
(1) Title: Lewiston Mainciacs Context: The Lewiston Mainciacs were junior ice hockey team The team played its home
gumes at the Androscoggin Bank Colisée
Androscoggin Bank Colisée Context: The Androscoggin Bank Colisée (formerly Central Maine Civic Center and Lewiston Colisee) is a 4,000 capacity (3,677 seated) multi- purpose arena...
(2) Titl:
Generation
The Lewiston Maineiacs were junior ice hockey leam of the Quebee Major Junior Hockey League based in Lewiston, Maine,
They played their home games at Androscoggin Bank Colisée. The Androscoggin Bank Colisée has a capacity of 4,000 (3,677 seated). So the answer is 3,677
Strategy QA
Question: Can you get Raclette in YMCA headquarters city?
Answer: Yes
Iteration
Retrieved Knowledge
(I) Title: YMCA Building (Racine. Wisconsin) Context: ... Since the 50's, headquarters of the publishing house and the bookstore are located rue de Montagne Sainte-Gene Geneviève 75005 Paris
Generation
The YMCA headquarters is located in Paris, France. Raclette a traditional Swiss dish, made with melted cheese served with potatoes, pickles, and other accompaniments. It not traditional French dish, is unlikely to be found in Paris. So the answer is No
Iteration 2
Retrieved Knowledge
(I) Title: Raclette Context: Raclette is Swiss dish, also popular in France, based on heating cheese and scraping off the melted part
(2) Title: YMCA Building (Racine. Wisconsin) Context: Since the 50's, headquarters of the publishing house and the bookstore are located in 11 rue de la Montagne SuinteGGeneviève 75005 Paris... ... Generation
YMCA headquarters located Paris, France. Raclette dish native to parts of Switzerland, but also popular France. So is likely that Raciette can found in Paris. So the answer is Yes
Table 7:
Two examples demonstrating retrieval-generatine synergy. We underline generated phrases that help retrieve relevant knowledge which successfully corrects factual errors (in red) in the second iteration. Irrelevant retrieved paragraphs are not shown in the table for brevity.
achieves significantly higher recall, indicating that LLM generations can help bridge the semantic gaps between complex questions and their supporting knowledge. However, performance quickly hits a plateau afterwards.
trieval as proxy for assessing an LLM's capability to answer the question using its parametric knowl- edge. Compared with Sclf-Ask, REN tends to be significantly better at preserving the LLM's performance on questions that the LLM can solve using CoT without retrieval, while heing com- petitive on the complementary subset. This may be because the structural constraints from Self-Ast Ask makes an LLM over-sensitive to the precision and comprehensiveness of follow-up question genera- tion and answering. and Self-Ask is also incapable of processing all retrieved knowledge as a whole, thus reducing the LLM's flexibility in solving a question. Moreover, BEER-RETGEN consistently outperforms Self-Ask by large margin, regardless
4.6.2 ITER-RETGEN Leverages Parametric and Non-Parametric Knowledge Better Ideally, an LLM should flexibly utilize non- parametric knowledge parametric knowledge depending on whether in-context non-parametric knowledge relevant or not. Table presents per- formance breakdowns on different subsets of ques- tions for investigation. We considered the ability of CoT to answer a question correctly without re-

---

of whether the in-context non-parametric knowl- edge mentions the answers or not. This indicates that when the in-context non-parametric knowledge is irrelevant or incomplete, STER-RETGEN exploits parametric knowledge hetter than Self-Ask.
Limitations
In this work, we propose to enhance retrieval- augmented large language models with ITER- RETGEN which synergizes retrieval and generation in an iterative manner, and demonstrates strong performance compared to more structured prompt- ing techniques such Self-Ask. However, it's worth noting that our experiments utilized fixed black-box large language model, which may not have been equally optimized for various forms prompting. It would be intriguing to investigate the potential of prompting-speciffi (gradient-base based) op- timization in pushing the limits further. This could involve enabling large language model to leverage parametric and non-pane con-aaraettric knowledge more flexibly and effectively. By exploring this avenue. we may uncover new insights and advancements in the field. Furthermore, our experiments did not cover long-form generation which would probably benefit from more fine-grained retrieval than ITER- RETGEN does in this work. We acknowledge that this area warrants further exploration, and we leave it for future work.
4.7 Error Analysis
On HotPotQA, we manually analyzed 20 random cases where STER-RETGER (T 2) fails. 25% of predictions are false negatives. On 10% of cases, ITER-RETGEN retrieves all necessary information but fails to perform correct reasoning. The remain- ing 65% of error cases are related with retrieval, 76.9% of which, retrieval is misled by completely wrong reasoning from the first iteration, while on the other cases, reasoning in the first iteration partially correct, but the retriever fails to retrieve the missing pieces in the second iteration. We also observed that, in the first iteration, reasoning can be negatively affected by noisy and possibly distrac- tive knowledge retrieved using only the questions the queries.
5 Case Study
A Acknowledgements
Table 7 demonstrates satrivall-gnnrattin synergy with two examples from HotPoiQA and Strate- EAQ, respectively. In the first iteration, as both questions need multi-hop reasoning, the retriever fails retrieve all supporting knowledge using only the questions. Despite being affected by distrac- tive retrieved knowledge (the capacity of a different arena in the example from HotPotQA) and show- ing imperfect parametric knowledge (the generated statement that Raclette is unlikely to be found in Paris in the example from StrategyQA) in the first iteration, the LLM generates phrases that help re- trieve relevant knowledge in the second iteration, and successfully corrects its outputs.
Zhihong Shao and Minlie Huang were supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010), They were also supported by the Guoqiang In- stitute of Tsinghua University, with Grant No. 2020GQG0005.
6 Conclusion
We demonstrate the effectiveness of ITER-RETGEN in answering questions with complex information needs. Despite simple, IEER-RETGEN outper- forms retrieval-aummented methods that have more complex workflow, which we believe could serve a strong baseline for future research retrieval-augmentee generation. We also show that retrieval adaptation can fur- ther improve the performance of TTER-RETGEST while also reducing overheads.

---

References
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022b. PAL: program-aided language models. CoRR, abs/2211.
Rami Aly. Zhijiang Guo, Michael Sejr Schlichtkrull, James Thome, Andreas Vlachos, Christos Christodoulopoulos Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: fact extraction and verifi- cation over unstructured and structured information. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurlPS Datasets and Benchmurks 2021, December 2021, virtual.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use laptop? A question answering benchmark with implicit reasoning strategies. Trans, Assoc. Comput. Linguistics, 9:346-361.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang. Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing.
Akari Asai, Timo Schick, Patrick S. H. Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannanch Hajishirzi, and Wen-tau Yih. 2022. Task-aware re- trieval with instructions. CoRR, 00//22111099666
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara. and Akiko Aizawa. 2020. Constructing A multi-hop QA dataset for comprehensive evaluation of reason- ing steps. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6609-6625. International Committee on Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fiic Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are seot learners. CoRR, 141/20
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022a. Unsupervised dense in- formation retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert. Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Nla Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing large language models trained on code. CoRR, abs/2 107.03374.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 874- 880. Association for Computational Linguistics.
Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave 2022b. Few-shot learning with retrieval augmented language models. CoRR, abs/2208.03299.
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. CORR, abs/2305.06988.
Vladimir Karpukhin. Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 6769 6781. Associa- tion for Computational Linguistics.
Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, Jason Weston, and Michael Auli. 2019. ELI5: long form question answering. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3558-3567 Association for Computational Linguis- tics.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li. David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate--seacch-erde Composing retrieval and language models for knowledge-intensive NLP. CoRR, abs/2212. 14024.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022a. Precise zero-shot dense retrieval without rele- vance labels. CoRR, abs/2212.1 2.10099..

---

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Was Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: henchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452- 466.
Zhihong Shao and Minlie Huang. 2022. Answering open-domain multi-answer answer questions via a recall- then-verify framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1825-1838. Association for Computational Linguisties.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. REPLUG: retrieval-augmmented black-box language models. CoRR, 0bs/2300.12655)
Yuning Mao, Pengcheng He. Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-augmentee retrieval for open- domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 1 11th International Joint Conference on Natural Language Processing. ACL/JUNNP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4089-41140 4100. Associa- tion for Computational Linguistics.
James Thorne, Andreas Vlachos, Christos Christocdoulopoulos, and Arpit Mittal. 2018.
FEVER: large-scale dataset for fact extraction and VERífication. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 809-819, New Orleans. Louisiana. Association for Computational Linguisties.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ramakanth Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Seialom. 2023. Augmented language models: a survey. CoRR, abs/2302.078.44
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Ana Lachaux. Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang. Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Ro- driguez, Robert Stojnic. Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.0922
Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang. Karl Cobbe, Tyna Eloundou. Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser- assisted question-answerte with human feedback. CoRR, 12.09332..
OpenAl. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models follow instruc- tions with human feedback. In NeurlPS.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a. Interleav- ing retrieval with ofain-of-thought reasoning for knowledge-intensicn multi-step questions. CoRR, abs/22 22112.1..5 10509.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350.
Harsh Trivedi. Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. 2022b. MuSiQue: Multi- hop questions via single- hop question composition. Trans. Assoc. Comput. Linguistics, 10:539-554
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022. Continual-th t0: Progressively instruct- ing 50+ tasks to language models without forgetting. CoRR, abs/2205.1222 123993..........
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.
Zhihong Shao, Yeyun Gong, Yelong Shen, Min- lie Huang, Nan Duan, and Weizhu Chen. 2023. Synthetic prompting: Generating chain-of-thought demonstrations for large language models. CoRR, abs/2302.00618.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and

---

Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop hop question answer- ing. In Proceedings of the 2018 Conference Em- pirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages Linguistics.
Association for Computational
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. CoRR, abs/2210.03629.
Oni Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. 2023. Answering questions by meta-reasoning over multiple chains of thought. CoRR, abs/2304.13007.
Fengji Zhang. Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. CoRR, Abs/2303.125777

---

A Experiments Using Llama-2
To demonstrate the effectiivnees of ITER-Ree RETGEN on open-source models, we replaced the generation model text-davinci-000 in Table 2 with Llama- 2 models (Touvron et al., 2023), and re-ran the evaluation. As shown in Table 8, ITER-RETGEN consistently outperforms all baselines significantly.
B Few-Shot Prompts
In this section, we present all few-shot prompts used in our experiments. We replace retrieved paragraphs with the placeholder (Knowledge) for brevity. CoT prompting shares the same in-context demonstrations with ITER-RETGEN, except that is not augmented with retrieval.
B.1 HotPotQA
Prompts Direct Prompting, ReAct, Self-Ask, and STER-RETGEN are presented Table 9, Table 10, Table 11, and Table 12, respectively.
B.2 2WikiMultHHopQA
Prompts for Direct Prompting, ReAct, Self-Ask, and ITER-RETGEN are presented in Table 13, Table 14, Table 15, and Table 16, respectively.
B.3 MuSiQue
Prompts for Direct Prompting, ReAct, Self-Ask, and IEER-RETGEN are presented in Table 17, Table 18, Table 19, and Table 20, respectively.
B.4 Bamboogle
Prompts for Direct Prompting, ReAct, Self-Ask. and TEER-RETGEN are presented Table 21, Table 22, Table 23, and Table 24, respectively.
B.5 Feverous
Prompts for Direct Prompting, ReAct, Self-Ask, and ITER-RETGEN are presented in Table 25, Table 26, Table 27, and Table 28, respectively.
B.6 StrategyQA
Prompts for Direct Prompting, ReAct, Self-Ask, and RETGER are presented Table 29, Table 30, Table 31, and Table 32, respectively.

---

Model Dataset
Llama-2-13B
Llama-2-70B
HotPotQA 2WikiMuliHopQAA StrategyQA HotPotQA 2WikiMuliHopQA StrategyQA
Without Retrieval
Direct
36.4
31.6
60.5
47.2
39.0
72.7
CoT
43.0
33.2
63.7
55.2
46.0
72.7
With Retrieval
Direct
51.8
38.6
63.3
58.6
45.1
73.3
ReAct
36.0
27.5
61.5
42.6
36.8
69.5
Self-Ask
45.8
38.5
63.3
58.4
53.2
71.7
ITER-RETGEN 53.8
44.6
62.8
64.4
55.1
74.8
ITHR-RETGEN 57.8
48.0
67.2
67.8
57.9
76.6
Table Experiments using the open-source Llama-2 models. used Acc! as evaluation metric, i.c., to evaluate the accuracy of model outputs with text-davinci-003.
[Knowledge]
Question: What is the name of this American musician, singer, actor, comedian, and songwriter, who worked with Modern Records and born in December 5, 1932?
The unswer is Little Richard
(Knowledge]
Question: Between Chinua Achebe and Rachel Carson, who more diverse jobs?
The is Chinua Achebe
[Knowledge]
Question: Remember Me Ballin' is CD single by Indo G that features American rapper hom in what year? The answer is 1979
Table 9: 3-Shot Demonstrations for Direct Prompting HotPotQA.

---

Given following question, answer by providing follow up questions intermediate unswers. each follow up question, you given a context which the top returned Wikipedia snippets for the question. If follow up questions are necessary, answer the question directly.
#
Question: What is the name of this American musician, singer, actor, comedian, and songwriter, who worked with Modern Records and horn in December 5, 1932? Are follow up questions needed here: Yes.
Follow up: Who worked with Modern Records? ( |nowledgee
Intermediate answer: Artists worked with Modern Records include Etta James, Little Richard, Joc Houston, Ike and Tina Tureer and John Lee Hooker.
Follow up: Is Bita James un American musician, singer, actur, comedian, and songwriter, and was burn in December 5, 1932? [Knowledge]
Intermediate answer: Etta James was born January 25, 1938, not December 5, 1932, so the answer no. Follow up: Little Richard an American musician, singer, actor, comedian, songwriter, und was born in December 1932? [nnwledgee)
Intermediate answer: Yes, Little Richard, hom December 5, 1932, American musician, singer, actor, comedian and songwriter.
So the final answer is: Little Richard #
Question: Between Chinua Achehe and Rachel Carson, who had more diverse jobs? Are follow up questions needed here: Yes. Follow up: What jobs did Chinua Achehe have? [Knoleegge]
Intermediate answer: Chinua Achebe was a Nigerian (1) novelist, (2) poet, (3) professor, and (4) critic, so Chinua Achebe had 4 jobs.
Follow up: What jobs Rachel Carson have? [Knowledge]
Intermediate answer: Rachel Carson was an American (1) marine biologist, (2) author, (3) conservationist, so Rachel Curson had jobs.
Follow up: Did Chinua Achebe more jobs than Rachel Carson? (Knoledgee Intermediale
Chinua Achebe had jobs, while Rachel Carson had 3 jobs. greater than 3, yes, Chinua Achebe had
more jobs.
So the final answer is: Chinua Achehe #
Question: Remember Me Ballin' is a CD single by Indo G that features American rapper born in what year? Are follow up questions needed here: Yes.
Follow up: Which American rapper is featured by Remember Me Ballin', CD single by Indo G?
Knowledge)
Intermediate answer: Gangsta Boo
Follow up: which year was Gangsta Boo born? [nowledgee]
Intermediate answer: Gangsta Boo was born in August 7, 1979, so the answer is 1979.
final answer 1979
Table 10: 3-Shot Demonstrations for ReAct on HotPoIQA.

---

Given the following question, answer by providing follow up questions and intermediate answers. For each follow up question, you given a context which is the lup returned Wikipedia snippets for the question. If no follow up questions are necessary. answer the question directly.
#
[Knowledge]
Question: What is the name of this American musician, singer, actor, comedian, and songwriter, who worked with Modern Records and born December 5. 1932?
Are follow up questions needed here: Yes.
Follow up: Who worked with Modern Records?
Intermediate answer: Artists worked with Modern Records include Fita James, Little Richard, Joe Houston, The and Tina Tumer and John Lee Hooker.
Follow up: Etta James American musician, singer, actor, comedian, and songwriter, and born in December 5, 1932? Intermediate answer: Fita James was horn in January 25, 1938, not December 5, 1932, so the answer is
Follow up: Is Little Richard an American musician, singer, actor, comedian, and songwriter, and was born in December 5, 1932? Intermediate answer: Yes, Little Richard, born in December 5, 1932, is American musician. singer, actor, comedian and songwriter.
So the final answer is: Little Richard
#
(Knowledge]
Question: Between Chinua Achebe and Rachel Carson, who had more diverse jobs?
Are follow up questions needed here: Yes.
Follow up: What jobs Chinua Achebe have??
Intermediate answer: Chinua Achebe was a Nigerian (1) novelist, (2) poet, (3) professor, and (4) critic, so Chinua Achebe had 4 johs.
Follow up: What jobs did Rachel Carson have?
Intermediate answer: Ruchel Carson was American (1) marine biolugist, (2) author, and (3) conservationist, Rachel Carson had jobs.
Follow up: Did Chinma Achebe have more johs than Rachel Carson? Intermediate answer: Chinua Achebe had jobs, while Rachel Carson had 3 jobs. 4 is greater than 3. so yes, Chinua Achebe had more jobs.
So the final answer is: Chinua Achebe #
[Knowledge]
Question: Remember Me Ballin' is a CD single by Indo G that features an American rapper born in w what year? Are follow up questions needed here: Yes.
Follow up: Which American rapper featured by Remember Me Ballin', a CD single by Indo G? Intermediate answer: Gangsta Buo
Follow up: which year was Gangsta Boo born?
Intermediate answer: Gangsta Boo was born in August 7, 1979, the answer is 1979.
So the final answer 1979
Table 11: 3-Shot Demonstrations for Self-Ask on HotPorQA.
(Knowledge]
Question: What is name this American musician, singer, actor, comedian, and songwriter, who worked with Modern Records and born December 5, 1932?
Let's think step by step.
Artists who worked with Modern Records include Eita James, Joe Houston, Little Richard, Ike and Tina Turner and John Lee Hooker in the 1950s and 1960s, of these Little Richard, born in December 5, 1932, was American musician, singer, actor, comedian, and songwriter. So the answer is Little Richard
[Knowledge]
Question: Between Chinua Achebe and Rachel Carson, who had more diverse jobs?
Let's think step by step.
Chinua Achehe was a Nigerian novelist, poet, professor, and critic. Rachel Carson was an American marine biologist, author, and conservationist. So Chinua Achebe had 4 jobs, while Rachel Carson had 3 jobs. Chinua Achebe had more diverse jobs than Rachel Curson. So the answer Chinua Achebe
(Knowledge] Question: Remember Me Ballin' is CD single by Indo G that features American rapper bomn in what year? Ler's think step by step.
Remember Me Ballin' is the CD single by Indo G featuring Gangsta Boo, Gangsta Boo is Lola Mitchell's stage name, who was born in August 7, 1979, and is an American rapper. So the answer is 1979
Table 12: 3-Shot Demonstrations for on HotPotQA.

---

(Knowledge)
Question: Which film came out first, Blind Shaft The Musk Of Manchu?
The is The Mask Of Fu Manchu
[Knowledge]
Question: When did John V. Prince Of Anhalt-Zerbst's father die? The unswer is 12 June 1516
[Knowledge]
Question: Which film has the director who was born later, Extrano Viaje or Love In Pawn? The answer is EI Extrano Viaje
Table 13: 3-Shot Demonstrations for Direct Prompting on 2WikiMuliHHoQA.
Given the following question, answer by providing follow up questions and intermediate unswers. For each follow question, given context which the tup returned Wikipedia stippets for the question. If no follow questions are necessary, answer the question directly. #
Question: Which film came out first, Blind Shaft or The Mask Of Fu Manchu? Are follow up questions needed here: Yes. Follow up: When did Blind Shaft come out? [Knowledge]
Intermediate answer; Blind Shaft came out 2003.
Follow up: When did The Mask Of Manchu come out? Knowledge)
Intermediate
The Mask Of Manchu The Mask Of Fu Manchu
in 1932.
So the final #
Question: When did John V, Prince Of Anhalt-Zerbst's father die? Are follow up questions needed here: Yes. Follow up: Who is the father of John V, Prince Of Zerbst? (Knowledge]
Intermediate unswer: The father of John Prince Of Anhall is Emnest I, Prince of Anhall-Dessau Follow up: When did Ernest I, Prince Anhalt-Dessau Dessau die? [nowleegge]
Intermediate answer; Ernest Prince Anhalt-Dessau died on June 1516. final answer 12 June 1516
#
Question: Which film has the director who born later, Extrano Viaje Love Pawn? follow up questions needed here: Yes. Follow up: Who is the director ( Knowledge]
Extrano Viuje?
Intermediate answer: The director of El Extrano Viaje is Fernando Fernan Gomez. Follow up: Who is the director of Love in Pawn? [ Knowledge) Intermediate answer: The director of Love in Pawn is Charles Saunders. Follow up: When was Fernando Fernan Gomez born? Knowledge]
Intermediate answer: Fernando Fernan Gomez was born on 28 August 1921. Follow up: When was Charles Saunders (director) hom? [nowledge)
Intermediate answer: Charles Saunders the final
April 1904.
Extrano Viaje
Table 3-Shot Demonstrations ReAct on 2WikiMultiHopOA.

---

Given the following question, answer by providing follow up questions intermediate answers. For each follow up question, you are given a context which the top returned Wikipedia snippets for the question. If no follow up questions are necessary. unswer the question directly.
#
[Knowiedge]
Question: Which film out lirst, Blind Shafl The Mask Of Manchu? Are follow up questions needed here: Yes.
Follow up: When did Blind Shaft come out?
Intermediate unswer: Blind Shaft came out in 2003.
Follow up: When did The Mask Of Fu Manchu come out? Intermediate answer: The Mask Of Fu Manchu came out in 1932. So the final answer is: The Mask Of Fu Manchu
#
[Knowledge]
Question: When did John V. Prince Of Anhalt-7erbst's father die? Are follow up questions needed here: Yes.
Follow up: Who is father John V. Prince Anhalt-Zerbst!
Intermediate answer: The father of John Prince Of Anhalt-7erbst Emesst I, Prince of Anhalt-Dessan Follow up: When did Emnest Prince of Anhalt-Dessau die? Intermediate So the final #
Ermest Prince Anhalt-Dessee died June 1516.
12 June 1516
[Knowledge]
Question: Which film has the director who was born later, El Extramo Viaje or Love In Pawn? Are follow up questions needed here: Yes.
Follow up: Who is the director of Fl Extrano Viaje?
Intermediate unswer: The director of EI Extrano Viaje is Fernando Fernan Gomez. Follow up: Who is the director of Love Pawn?
Intermediate answer: The director of Love in Pawn is Charles Saunders. Follow up: When was Fernando Fernan Gomez born?
Intermediate answer: Fernando Fernan Gomez was born on 28 August 1921, Follow up: When Charles Saunders (director) hom? Intermediate
Charles Saunders born April 1904.
So the final answer Extrano Viaje
Table 15: 3-Shot Demonstrations for Self-Ask on 2WikiMultHHoOOA.
(Knowledge)
Question: Which film came out first, Blind Shaft or The Mask Of Fu Manchu? Let's think step by step.
Blind Shaft is a 2003 film, while Mask Of Manchu opened in New York December 2, 1932. 2003 comes after 1932. Therefore, The Mask Of Fu Manchu came earlier than Blind Shafi.
So the
The Mask Of Fu Manchu
(Knowledge] Question: When did John V. Prince Of Anhalt-Zerbst's father die? Let's think step by step. John was the second son of Ernest I, Prince of Anhalt-Dessee Ernest 1, Prince of Anhull-Dessau died 12 June 1516.
So the answer is 12 June 1516
(Knowledge]
Question: Which film has the director who was hom later, Extrano Viaje or Love Pawn?
Let's think step by step.
The director of El Extrano Viaje is Fernando Fernan Gomez, was born on August 1921. director Love In Pawn Charles Saunders, who was born on April 1904, August 1921 comes after 8 April 1904. Therefore, Fernando Fernan Gomez horn later than Charles Saunders.
So the
Extrano Viaje
Table 16: S-oho Demonstrations for RER-ERE on 2WikiuuliHHpQA.

---

[Knowledge]
Question: In which year did the puhlisher of In Cold Blood form? The answer is 2001
[Knowledge]
Question: Who was in charge the city where The Killing a Sacred Deer was filmed!? The is John Cranley
[Knowledge]
Question: Where the Avalon Peninsula is the city that Signal Hill overlooks?
The answer is eastern tip
Table 17: 3-Shot Demonstrations for Direct Prompting MuSiQue.
Given the lollowing question, answer by providing follow up questions and intermediate unswers, For each lollow up question, you given context which is the top returned Wikipedia snippets for the question. If no follow questions are necessary, answer the question directly. #
Question: In which year did the publisher of In Cold Blood form?
Are follow up questions needed here: Yes. Follow up: What business published Cold Blood? [Knowledge]
Intermediate answer: Cold Blood was published in book form by Random House, Follow up: Which year witnessed the formation Random House? |nowledgee]
Intermediate answer: Random House form 2001. the final answer 2001
#
Question: Who in charge the city where The Killing of Sacred Deer filmed?? Are follow up questions needed here: Yes. Follow up: In which city was The Killing of Sacred Deer filmed |Knolldgee)
Intermediate answer: The Killing of Sacred Deer was filmed in Cincinnati.
Follow up: Who was in charge of Cincinnati? (nowleegee]
Intermediate unswer: The present Mayor of Cincinnati is John Cranley, so John Cranley in charge. So the final answer is: John Cranley #
Question: Where the Avalon Peninsula is the city that Signal Hill overlooks?
follow up questions needed here: Yes.
Follow up: What city does Signal Hill overlonk? Knowledge)
Intermediate
Signal Hill hill which overlooks the city of St. John's. Follow up: Where the Avalon Peninsula is St. John's located?
( Knowledge]
Intermediate unswer: St. John's is located the eastern tip of the Avalon Peninsula. So the finul answer is: eastern tip
Table 18: 3-Shot Demonstrations for ReAct MuSiQue.

---

Given the following question, answer by providing follow up questions and intermediate unswers. For each follow question, you are given a context which is the top returned Wikipedia snippets for the question. If no follow questions are necessary, answer the question directly.
#
[Knowledge]
Question: which year did the publisher of In Cold Blood form?
Are follow up questions needed hcre: Yes. Follow up:
business published Cold Blood?
Intermediate answer: Cold Blood was published book form Random House. Follow up: Which year witnessed the formation Random House?
Intermediate answer: Random House form in 2001.
So the final #
2001
[Knowledge]
Question: Who was in charge of the city where The Killing of a Sacred Deer was filmed?
Are follow up questions needed here: Yes.
Follow up: In which city was The Killing of a Sacred Deer filmed Intermediate answer: The Killing of Sacred Door was filmed in Cincinnati.
Follow up: Who was in charge Cincinnati? Intermediate answer: The present Mayor of Cincinnati John Cranley, so John Cranley in charge. So final answer is: John Cranley
#
(nnowledee]
Question: Where the Avalon Peninsula the city that Signal Hill overlooks? Are follow up questions needed here: Yes. Follow up: What city dues Signal Hill overlook?
Intermediate answer: Signal Hill is hill which overluoks the city of St. John's. Follow up: Where the Avalon Peninsula is St. John's 's located? Intermediate unswer: St. John's is located the eastern tip of the Avalon Peninsula.
So the final answer is: eastern tip
Table 19: 3-Shot Demonstrations for Self-Ask on MuSiQue.
[Knowledge] Question: In which year did the publisher of In Cold Blood form? Let's think step by step. In Cold Blood was first published in book form by Random House. Random House was form in 2001. So the answer is 2001
(Knowledge]
Question: Who was in charge of the city where The Killing of a Sacred Deer was filmed? Let's think step by step.
The Killing of Sacred Deer was filmed Cincinnati. present Mayor of Cincinnati John Cranley, Therefore, John city.
Cranley charge
the John Cranley
[Knowledge]
Question: Where the Avalon Peninsula is the city that Signal Hill overlooks?
Let's think step by step.
Signal Hill is hill which overlooks the city of St. John's. St. John's is located on the eastern tip of the Avalon Peninsula. So the answer is castern tip
Table 20: 3-Shot Demonstrations for ITER-RETGEN MuSiQue.

---

[Knowledge]
Question: When did the first prime minister of the Russian Empire come into office?
The answer is 1905-11-06 00:00:00
[Knowledge]
Question: most populous city in Punjab how large (area wise)?
The is 310 square kilometers
[Knowledge]
Question: What the capital of the country where yoga originated?
The answer is New Delhi
Table 21: 3-Shot Demonstrations for Direct Prompting Bamboogle.
Given the following question, answer by providing follow up questions and intermediate unswers, For each lollow up question, you given context which is the top returned Wikipedia snippets for the question. If no follow questions are necessary, answer the question directly. #
Question: When did the first prime minister of the Russian Empire come into office?
Are follow up questions needed here: Yes. Follow up: Who is the first prime minister of the Russiun Empire? [Knowledge]
Intermediate answer: Sergei Witte
Follow up: When did Sergei Witte come into office? |nowledgee]
Intermediate
Sergei Witte was appointed November 1905.
the final answer 1905-1 1-006 00:00:00
#
Question: The most populous city in Punjab how large (area wise)?
Are follow up questions needed here: Yes.
Follow up: What is the most populous city in Punjab? (Knowlegge)
Intermediate answer: Luchiana is the most pupulous and largest city in Punjub. Follow up: How large is Ludhiana, the most populous city in Punjab? (Kowllegge]
Intermediate answer: The urea of Ludhiana over 310 km2. So the final answer is: 310 square kilometers #
Question: What
capital
country where yuga originated?
follow up questions needed here: Yes.
Follow up: Which country was yoga originated? Knowledge)
Intermediate answer: There is consensus yoga's origin. Suggested origins include India. Follow up: What is the capital of India? Knowledge)
Intermediate unswer: The current capital of India is New Delhi.
So the finul answer is: New Delhi
Table 22: 3-Shot Demonstrations for ReAct on Bamboogle.

---

Given the following question, answer by providing follow up questions and intermediate unswers, For each follow up question. you are given a context which is the top returned Wikipedia snippets for the question. If no follow up questions are necessary, answer the question directly.
#
{ (Knowledge)
Question: When did the first prime minister of Russian Empire come office?
follow up questions needed here: Yes.
Follow up: Who is the first prime minister of the Russian Empire?
Intermediate answer: Sergei Witte Follow up: When did Sergei Witte come into office?
Intermediate answer: Sergei Witte was uppointed on 6 November 1905.
So the final answer is: 1905-11 1-06 00:00:00
#
[Knowledge]
Question: The most populous city in Punjab is how large (area wise)? Are follow up questions needed here: Yes. Follow up:
is the most populous city in Punjab?
Intermediate unswer: Ludhiana is the most pupulous and largest city in Punjab. Follow up: How large is Ludhiana, Intermediate answer: The area Ludhiana the final
most pupulous city Punjab?
310 km2.
square kilomeeeess
#
]Knowledge]
Question: What the capital of the country where yoga originated? Are follow up questions needed here: Yes.
Follow up: Which country was yoga originated?
Intermediate answer: There is no consensus on yoga's origin. Suggested origins include India. Follow up: What is the capital of India?
Intermediate unswer: The current capital of India is New Delhi.
So the final amswer is: New Delhi
Table 23: 3-Shot Demonstrations for Self-Ask on Bamboogle.
(Knowledge]
Question: When did the first prime minister Russian Empire come office? Let's think step by step.
The first prime minister of the Russian Empire Count Sergei Witte, Sergei Witte appointed 6 November 190S. So the answer is 1905- 1-06 00:00:00
[Knowledge]
Question: The most populous city in Punjah is how large (area wisej? Let's think step by step.
Ludhiana is the most populous and the largest city in the Indian state of Punjab. The city has an area of over 310 km2.
So the answer 310 square kilometers
(Knowledge] Question: What the capital of the country where yoga originated? Let's think step step. Suggested origins include pre- pre-Vedic Vedie Eastern states of India. The current capital of India is New Delhi.
So the answer New Delhi
Table 24: 3-Shot Demonstrations for ITER-RETGRA Bamboogle.

---

(Knowledge]
Question: Is true that Belgrade Race is an annual men's footrace of around kilometres (5834 metres) that is held Belgrade, Serbia through history, past winners includes Brahim Lahlafi (Ist edition), Philip Mosima (3rd) and Josphat Menjo (6th)? The answer is Yes
(Knowledge]
Question: Is it true that Based on the same platform the Chevrolet Sail, the Baojun 310 was launched on 2017 Beijing Auto Show where the price ranges from 36.800 yuan to 60.800 yuan?
The answer is No
[Knowledge]
Question: Is il true that Florida International University pedestrian bridge collapse was funded with a $19.4 million Transportation Investment Generating Economic Recovery grant from the United States Department Transportation in 201 along with state agencies and the bridge cost $14.2 million to construct? The answer is No
Table 25: 3-Shot Demonstrations for Direct Prompting on Feverous.
Given the following question, answer by providing follow up questions and intermediate answers. For each follow up question, you given a context which is the top returned Wikipedia snippets for the question. If no follow up questions are necessary. unswer the question directly. The final answer should always be either Yes or No, and NOTHING ELSE. #
Question: Is true that Belgrade Race is an annual men's 's footrace of around 6 kilometres (5834 metres) that is held in Belgrade, Serhia through history, past winners includes Brahim Lahlafi sI edition), Philip Mosima (3rd) and Josphat Menjo (6th)? Are follow up questions needed here: Yes.
Follow up: What is the Belgrade Race? [Knowledge]
Intermediate answer: The Belgrade Race Through History an annual men's 's footrace around 6 kilometres (5834 metres) that held in Belgrade, Serbia.
Follow up: Has Brahim Lahlafi won Belgrade Race? Knowledge)
Intermediate answer: Yes, Brahim Lahlati was the winner in 1996. Follow up: Has Philip Mosima won Belgrude Race? |Knolldgee]
[
Intermediate unswer: Yes. Philip Mosima beat Marathon world record and won in 1998 Follow up: Has Jusphat Menjo won Belgrade Race?
[ Knowledge)
Intermediate unswer: Yes, Josphat Menjo broke the meet record and won the competition. So the final amswer is: Yes #
Question: Ts it true that Based on the same platform as the Chevrolet Sail, the Baojun 310 was launched on 2017 Beijing Auto Show where the price ranges from 36.800 yuan to 60.800 yuan? Are follow up questions needed here: Yes. Follow up: When and where was the Banjun 310 launched? Knowledgee)
Intermediate answer: The Baojun 310 was launched on 2016 Beijing Auto Show, 2017 Beijing Auto Show.
So the final answer No #
Question: Is it true that Florida International University pedestrian bridge collapse was funded with a $19.4 million Transportation Investment Generating L'conomic Recovery grant from the United States Department of Transportation in 2013, along with state agencies and the bridge cost $14.2 million to construct? Are follow up questions needed here: Yes.
Follow up: How was Florida International University pedestrian bridge collapse funded? [ [nowledge)]
Intermediate answer: Florida International University pedestrian bridge was a $14.2 22 million project funded with $19.4 4 million Transportation Investment Generating Economic Recovery (TIGER) grant from the United States Department of Transportation in 2013, along with state agencies, which consistent with facts in question.
Follow up: How much did cost to construct Florida International University pedestrian bridge? Knowledge]
Intermediate answer: The bridge cost $9 million to construct, $14.2 million. So the final answer is: No
Table 26: 3-Shot Demonstrations for ReAct on Feverous.

---

Given the following question, answer by providing follow up questions and intermediatc answers. For each follow up question, you given context which is the top returned Wikipedia snippets for the question. If no follow up questions are necessary, answer the question directly. The final answer should always he either Yes No, and NOTHING ELSF.
#
[Knowledge]
Question: Is true that Belgrade Race is an annual men's 's footrace of around 6 kilometres (5834 metres) that is held in Belgrade, Serbia through history, past winners includes Brahim Lahlafi st edition), Philip Mosima (3rd) and Josphat Menjo (6th)? Are follow up questions needed here: Yes.
Follow up: What is the Belgrade Race? Intermediate answer: The Belgrade Race Through History is an annual men't 's footrace of around 6 kilometres (5834 metres) that is held in Belgrade, Serbia.
Follow up: Has Brahim Lahlafi won Belgrade Race?
Intermediate answer: Yes. Brahim Lahlafi was winner 1996. Follow up: Has Philip Mosima won Relgrade Race?
Intermediate answer: Yes, Philip Mosima beat Marathon world record and 1998 Follow up: Has Josphat Menjo Belgrade Race?
Intermediate answer: Yes, Josphat Menjo broke the meet record and won the competition. So the final answer is: Yes #
(Knowledge)
Question: Is it true that Based on the same platform as the Chevrolet Sail, the Baojun 310 was launched on 2017 Beijing Auto Show where the price ranges from 36.800 yuan to 60.800 yuan?
Are follow up questions needed here: Yes.
Follow up: When and where was the Banjun 310 launched?
Intermediate answer: The Baojun 310 was launched on 2016 Beijing Auto Show, not 2017 Beijing Auto Show. So the final answer is: No
#
[Knowledee]
Question: Is that I'lorida International University pedestrian bridge collapse was funded with $19.4 million Transportation Investment Generating Economic Recovery grant from the United States Depariment Transportation in 2013, along with stute agencies and the bridge cost $14.2 million construct? Are follow up questions needed here: Yes.
Follow up: How was Florida International University pedestrian bridge collapse funded?
Intermediate answer: Florida International University pedestrian bridge was a $14.2 million project funded with $19.4 million Transportation Investment Generating Economic Recovery (TIGER) grant from the United States Department of Transportation in 2013, along with state agencies, which is consistent with facts in the question.
Follow up: How much did it cost to construct Florida International University pedestrian bridge?
Intermediate answer: The hridge cost 59 million to construct, not $14.2 million.
So the final answer is: No
Table 27: 3-Shot Demonstrations for Self-Ask on Feverous.

---

You are required to verify facts in the following questions. The final answer to question should always be either Yes or No, and NOTHING ELSF.
[Knowledge]
Question: Is true that Belgrade Race is an annual men's footrace of around kilometres (5834 metres) that held Belgrade, Serbia through history, past winners includes Brahim Lahlafi (Ist edition), Philip Mosima (3rd) and Josphat Menjo (6th)?
Let's think step by step.
need verify facts in the question. The Belgrade Race Through History (5834 metres) that is held in Belgrade, Serbia. In 1996 Brahim Lahlaf was the winner of the competition. Philip Mosima won the competition in 1998, and beat Marathon world record holder Paul Tergat. Jusphat Menjo also won the competition and broke the meet record. Therefore, past winners include Brahim Lahlafi, Philip Mosima and Josphat Menjo, All facts verified. So the answer is Yes
annual men's footrace of around 6 kilometres
[Knowledge]
Question: Is it true that Based on the same platform as the Chevrolet Sail, the Baojun 310 was launched on 2017 Beijing Auto Show where the price ranges from 36.800 yuan to 60.800 yuan? Let's think step by step.
need verify facts the question. The Baojun 310 was indeed based on the same platform as the Chevrolet Sail. The Baojun 310 was launched on 2016 Beijing Auto Show, not 2017 Beijing Auto Show. So the answer No
(Knowledge]
Question: Is it true that Florida International University pedestrian bridge collapse was funded with a $19.4 million Transportation Investment Generating Economic Recovery grant from the United States Department of Transportation in 2013, along with state ugencies and the bridge cost $14.2 million to construct? Let's think step by step. I need to verify facts in the question. Florida International University pedestrian bridge was a $14.2 million project funded with $19.4 million Iransportation Investment Generating L'conomic Recovery (TIGER) grant from the United States Department of Transportation in 2013, along with slate agencies. The Bridge cost $8 million to construci, not $14.2 million.
So the answer is No
Table 28: 3- Shot Demonstrations for STER-RETGES on Feverous.
(Knowledge)
Question: Is it common to see frost during some cullege commenccemmntts? The answer is Yes
[Knowledge]
Question: Could a lama hirth twice during War Vietnam (1945-46)? The answer is No
[Knowledge]
Question: Would Richard Dawkins hypothetically refuse an offering of the Last rites? The answer is Yes
Table 29: 3-Shot Demonstrations for Direct Prompting StrategyQA.

---

Given the following question, answer by providing follow questions and intermediate unswers. For each follow question. you are given a context which is the top returned Wikipedia snippets for the question. If no follow questions are necessary, answer the question directly. The final answer should always be cither Yes or No, and NOTHING ELSE. #
Question: Is it common to see frost during some college commenccmmnnts?
Are follow up questions needed here: Yes. Follow up:
seusons can you expect to see frost! (nowledge)
Intermediate answer: Frost common during the winter.
Follow up: When college commencement? [Knowledge]
Intermediate
College commencement ceremonies often happen during the months of December, May, June. Follow up: Do any of the months December, May. June occur during the Winter? ( Knowledge]
Intermediate answer: December is in the winter.
So the final answer is: Yes
#
Question: Could a Ilama birth twice during War in Vietnam (1945-46)? Are follow up questions needed here: Yes. Follow up: How long did the vietnam war (1945-1946) last? (nowledgee]
Intermediate answer: The Vietnam (1945-46) lasted around 6 months. llama gestation period?
Follow up: How long Knowledge]
Intermediale answer: The gestation period for Ilama 11.5 months. Follow up: What is 2 times 11.5? [ Knowledge]
Intermediate answer: 23, which is longer than 6. So the final answer is: No
#
Question: Would Richard Dawkins hypothetically refuse an offering of the Last rites? Are follow up questions needed here: Yes. Follow up: What are the last Rites? |nowledge]
Intermediate answer: The Last rites, in Catholicism, are the last prayers and ministrations given to an individual the faith, when possible, shortly before death. Follow up: What Richard Dawkins religious beliefs? Knowledge)
Intermediate answer: Richard Dawkins is known an outspoken atheist, well known for his criticism creationism and intelligent design.
Follow up: Would atheist participate in Catholies prayers? { Knowledge]
Intermediate answer: It is unlikely that an atheist would participate in Catholics prayers.
So the final answer is: Yes
Table 30: 3-Shot Demonstrations for ReAct on StrategyQA.

---

Given the following question, answer by providing follow up questions and intermediate answers. For each follow question, you given a context which is the lop returned Wikipedia snippets for the question. If no follow questions are necessary. answer the question directly. The final answer should always be cither Yes or No, and NOTHING ELSE.
#
[Knowledge)
Question: Is it common to see frost during some college commencements? Are follow up questions needed here: Yes.
Follow up: What seasons can you expect to see frost?
Intermediate answer: Frost common during the winter. Follow up: When college commencement?
Intermediate answer; College commencement ceremonies often happen during the months December, May, June. Follow up: Do any of the months December, May, June occur during the Winter? Intermediate answer: December is in the winter.
So the final answer is: Yes
#
[Knowledge]
Question: Could a Ilama birth twice during. War in Vietnam (1945-46)?
Are follow up questions needed here: Yes.
Follow up: How long did the vietnam war (1945-1946) last?
Intermediate answer: The War in Vietnam (1945-46) lasted around 6 months. Follow up: How long the llama gestation period?
Intermediate unswer: The gestation period for a llamna 11.5 months.
Follow up: What 2 times 11.5?
Intermediate answer: 23, which longer than So the final answer No
#
]Knowledge]
Question: Would Richard Dawkins hypothetically refuse an offering of the Last rites?
Are follow up questions needed here: Yes. Follow up: What are the last Rites? Intermediate answer: The Last rites, in Catholicism, are the last prayers and ministrations given to an individual of the faith, when possible, shortly before death.
Follow up: What are Richard Dawkins religious beliets?
Intermediate answer: Richard Dawkins is known as an outspoken atheist, well known for his criticism of creationism and intelligent design.
Follow up: Would an atheist participate Catholics prayers?
Intermediate answer: It unlikely that un atheist would participate Catholics prayers.
final answer Yes
Table 3-Shot Demonstrations for Self-Ask on StrategyQA.
You are required to answer the following questions. The final answer to a question should always be either Yes or No, and NOTHING ELSE.
(Knowledge]
Question: common to frost during some college commencements? Let's think step step.
College commencement ceremonies often happen during the months of December, May, and sometimes June. Frost isn't uncommon to see during the month of December, it is the winter. So the answer is Yes
[Knowledge]
Question: Could a Ilama birth twice during War in Vietnam (1945-46)? Let's think step by step.
The War in Vietnam (1945-46) lasted around months. The gestation period for a llama months. llaaa birth twice, the minimum time needed 2 times 11 months, which is 22 months, longer than 6 months. So answer No
[Knowledge]
Question: Would Richard Dawkins hypotheticully refuse an offering of the Last rites? Let's think step step.
Richard Dawkins is known rites, in Catholicism, are the last prayers and ministrations given individual of the faith, when pussible, shortly before death. It is unlikely that an atheist would participate in Catholics prayers. So the answer is Yes
outspoken atheist, well known for his criticism of creationism and intelligent design. The Last
Table 32: 3-Shot Demonstrations for ITER- RETGEN on StrategyQA.


Reviews:
[{"review_id": "Z2oUyPestP", "reviewer": "Reviewer_vbix", "summary": "", "questions": "1. What is the length of each candidate Wikipedia chunk?\n2. Table 1 is not necessary and can be placed in the appendix.\n3. Why ITER-RETGEN is worse than self-ask on MuSiQue? Why does CoT perform almost best on Bamboogle?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time, insightful comments, and questions. We provided our responses below.\n\n> In lines 65-70, the authors mention the limitations of previous work, but ITER-RETGEN seems to have similar problems.\n\nThe limitations we mentioned apply to the structured workflows adopted by previous work like ReAct and Self-Ask.\n\n* Limitation 1: `they fail to process all retrieved knowledge as a whole during the generation process`\n\n  * Methods like ReAct and Self-Ask interleave retrieval and generating intermediate reasoning steps, until the model generates the final answer. Each intermediate reasoning step is conditioned on knowledge retrieved before, with no awareness of knowledge retrieved afterwards.\n  * **How does Iter-RetGen alleviate this limitation?** In each iteration, Iter-RetGen retrieves knowledge based on (1) the question and (2) the previous model output which shows what may be needed to answer the question. Unlike ReAct and Self-Ask conditioning each reasoning step on partial knowledge, Iter-RetGen generates reasoning steps and the final answer based on all knowledge retrieved in an iteration. Notably, knowledge retrieved in an iteration is not complementary to that in the previous iteration, but tends to be more comprehensive and relevant. In our preliminary experiments, we have tried conditioning reasoning steps and the final answer on a merged version of knowledge from different iterations, but found no noticeable improvements.\n\n* Limitation 2: `they invoke retrieval multiple times, and may frequently change the prompts by updating newly retrieved knowledge, thus increasing the overheads of both retrieval and generation`\n\n  * To finalize a single answer, methods like ReAct and Self-Ask have to invoke multiple rounds of retrieval and generation to gather a comprehensive set of knowledge, with different rounds focusing on complementary aspects and updating the prompt accordingly.\n  * **How does Iter-RetGen alleviate this limitation?** Unlike ReAct and Self-Ask invoking multiple rounds of retrieval and generation to finalize a single answer, Iter-RetGen produces an answer in each iteration, with each iteration consisting of a single round of retrieval and generation. We exploit the synergy between retrieval and generation with multiple iterations. The number of iterations is a hyper parameter, and usually, 2 or 3 iterations suffice to give the most performance boost. As can be seen in Table 3, Iter-RetGen using 2 iterations already achieves significantly higher or competitive accuracy while causing fewer overheads of retrieval and generation than ReAct and Self-Ask.\n\n>  What is the length of each candidate Wikipedia chunk?\n\nWe used preprocessed wikipedia chunks from previous works. Chunks from the 2017, 2018, 2021 Wikipedia dumps are of size no more than 100, 100, 200 words, respectively.\n\n> Table 1 is not necessary and can be placed in the appendix.\n\nThanks for the suggestion! We will place it in the appendix.\n\n> Why ITER-RETGEN is worse than self-ask on MuSiQue?\n\nIter-RetGen is competitive with Self-Ask on MuSiQue, but with a slightly lower accuracy. We manually investigated 10 random MuSiQue questions where Iter-RetGen (T=2) fails but Self-Ask gives correct answers. For Iter-RetGen, 7 / 10 of error cases are related with retrieval, on 2 of which reasoning in the first iteration is totally correct but the retriever retrieves distractive facts in the second iteration, while on the other cases, reasoning in the first iteration is partially correct, but the retriever fails to retrieve the missing pieces in the second iteration.\n\n> Why does CoT perform almost best on Bamboogle?\n\nText-davinci-003 is knowledgeable, and can achieve almost the best accuracy using its own parametric knowledge (i.e., using CoT without retrieval) on Bamboogle. Iter-RetGen can achieve a slightly higher accuracy while both ReAct and Self-Ask perform significantly worse.\n\nOn Bamboogle, all retrieval-augmented methods evaluated in this paper benefit Text-davinci-003 no more often than introducing noisy or even distractive facts. We manually investigated 10 random Bamboogle questions where both CoT without retrieval and Iter-RetGen give wrong answers. For 60% of them, we failed to retrieve relevant information using our retrieval system even with clearly-stated queries:\n\n* The dense retriever fails to capture fine-grained information needs, e.g., for the query *When did George Washington leave office?*, the dense retriever fails to retrieve `Washington was inaugurated as the first president of the United States, and ended on March 4, 1797`, but returns `Washington decided early in 1796 that he would retire... departure from office after two terms`.\n* The retrieval corpus is not comprehensive, e.g., the 2021 Wikipedia dump seems to contain no information about *How much protein is in one boiled egg yolk?*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a simple and effective method to enhance retrieval-augmented LLMs through retrieval-generation synergy.\n2. This paper introduce a reliable evaluation metric, which is more rubust than exact match.", "reasons_to_reject": "1. In lines 65-70, the authors mention the limitations of previous work, but ITER-RETGEN seems to have similar problems.\n", "questions_to_authors": "1. What is the length of each candidate Wikipedia chunk?\n2. Table 1 is not necessary and can be placed in the appendix.\n3. Why ITER-RETGEN is worse than self-ask on MuSiQue? Why does CoT perform almost best on Bamboogle?", "paper_topic_and_main_contributions": "This paper propose ITER-RETGEN to synergize retrieval and generation in an iterative manner to enhance large language models. And the authors also adapts retrieval with model generations. Empirical results show that ITER-RETGEN performs better on three different tasks than other retrieval-augmented methods. In addition, the authors propose a reliable evaluation metric, which uses text-davinci-003 to evaluate  the correctness rather than the exact match.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uRiNIRRyS8", "reviewer": "Reviewer_yJLF", "summary": "", "questions": "Q1. Can you elaborate how the previous methods \"fail to process all the retrieved knowledge\"? And how your method can achieve it?\nQ2. Can you provide some insights on how the performance of ITER-GEn compared to Re2G? That is to say, we finetune the generation based on the retrieval outputs instead of using in-context learning?\n\nUpdated: I like to thank the authors for their feedbacks. However, the reply does not clearly address Q1, and since the novelty is not the strongest point of the paper, I like to keep my scores as it is.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time, insightful comments, and questions. We provided our responses below.\n\n> The work is motivated by the issues presented from line 063-070: (1) previous methods fail to process all the retrieved knowledge; (2) previous studies invoke retrieval multiple times. However, it is not clear to me how such issues affect the method design. For example, how ITER-GEN can \"process all the retrieved knowledge\"? In addition, theoretically, ITER-GEN also needs to invoke retrieval multiple times.\n> \n> Can you elaborate how the previous methods \"fail to process all the retrieved knowledge\"? And how your method can achieve it?\n\nThe limitations we mentioned apply to the structured workflows adopted by previous work like ReAct and Self-Ask.\n\n* Limitation 1: `they fail to process all retrieved knowledge as a whole during the generation process`\n\n  * Methods like ReAct and Self-Ask interleave retrieval and generating intermediate reasoning steps, until the model generates the final answer. Each intermediate reasoning step is conditioned on knowledge retrieved before, with no awareness of knowledge retrieved afterwards.\n  * **How does Iter-RetGen alleviate this limitation?** In each iteration, Iter-RetGen retrieves knowledge based on (1) the question and (2) the previous model output which shows what may be needed to answer the question. Unlike ReAct and Self-Ask conditioning each reasoning step on partial knowledge, Iter-RetGen generates reasoning steps and the final answer based on all knowledge retrieved in an iteration. Notably, knowledge retrieved in an iteration is not complementary to that in the previous iteration, but tends to be more comprehensive and relevant. In our preliminary experiments, we have tried conditioning reasoning steps and the final answer on a merged version of knowledge from different iterations, but found no noticeable improvements.\n\n* Limitation 2: `they invoke retrieval multiple times, and may frequently change the prompts by updating newly retrieved knowledge, thus increasing the overheads of both retrieval and generation`\n\n  * To finalize a single answer, methods like ReAct and Self-Ask have to invoke multiple rounds of retrieval and generation to gather a comprehensive set of knowledge, with different rounds focusing on complementary aspects and updating the prompt accordingly.\n  * **How does Iter-RetGen alleviate this limitation?** Unlike ReAct and Self-Ask invoking multiple rounds of retrieval and generation to finalize a single answer, Iter-RetGen produces an answer in each iteration, with each iteration consisting of a single round of retrieval and generation. We exploit the synergy between retrieval and generation with multiple iterations. The number of iterations is a hyper parameter, and usually, 2 or 3 iterations suffice to give the most performance boost. As can be seen in Table 3, Iter-RetGen using 2 iterations already achieves significantly higher or competitive accuracy while causing fewer overheads of retrieval and generation than ReAct and Self-Ask.\n\n> The improvement of retrieval is mostly due to the distillation process, not by the inclusion of y1. However, the method to distill the reranker to retrieval has been proposed in Re2G\n\nWe used the off-the-shelf Contriever-MSMARCO for all experiments except those in Table 4. In Table 2, the improvements of Iter-RetGen over baselines come from synergizing retrieval and generation by iterating retrieval-augmented generation and generation-augmented retrieval.\n\nWe also shown that generation also benefits retrieval adaptation. Apart from distilling knowledge from a re-ranker, we also include model generation $y_1$ to the optimization process, so that we can (1) retrieve more relevant documents for the retriever to train on and (2) help the retriever better address the semantic gaps between a question and its supporting knowledge by using $y_1$ as an additional input to the teacher model (i.e., the re-ranker).\n\nAs shown in Table 4, `distillation with $y_1$` consistently outperforms `distillation without $y_1$` on both HotPotQA and Feverous, with significant improvement (3.4\\% absolute gain) on Feverous.\n\n> Can you provide some insights on how the performance of ITER-GEn compared to Re2G? That is to say, we finetune the generation based on the retrieval outputs instead of using in-context learning?\n\nAs different tasks have different answer spaces (e.g., short-form qa vs. boolean qa), we used in-context learning mainly to ensure the model generates task-specific answers. Finetuning the generation model is definitely an alternative way to shape the output space, and potentially improves utilization of retrieved knowledge.\n\nIter-RetGen is based on the observation that a model-generated answer can bridge the semantic gaps between a question and its supporting knowledge, which we believe is applicable to generation systems including Re2G, i.e., to retrieve more relevant knowledge based on generated answers, and then generate better answers based on updated knowledge.\n\n> Missing Reference to the Re2G paper\n\nThanks for the pointer! We will cite this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is simple yet effective in a number of tasks and datasets.\n2. The experimental design in 4.6.2 is interesting. It helps shed light of how the proposed method leverage parametric and non-parametric knowledge.\n3. The authors propose a new metric based on LLM for evaluation.", "reasons_to_reject": "1. Given that the method is simple and the idea of interleaving between retrieval and generation is not new, the novelty of this paper might be limited.\n2. The work is motivated by the issues presented from line 063-070: (1) previous methods fail to process all the retrieved knowledge; (2) previous studies invoke retrieval multiple times. However, it is not clear to me how such issues affect the method design. For example, how ITER-GEN can \"process all the retrieved knowledge\"? In addition, theoretically, ITER-GEN also needs to invoke retrieval multiple times. \n3. The improvement of retrieval is mostly due to the distillation process, not by the inclusion of y1. However, the method to distill the reranker to retrieval has been proposed in Re2G [1].\n\n[1] Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2G: Retrieve, Rerank, Generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2701–2715, Seattle, United States. Association for Computational Linguistics.", "questions_to_authors": "Q1. Can you elaborate how the previous methods \"fail to process all the retrieved knowledge\"? And how your method can achieve it?\nQ2. Can you provide some insights on how the performance of ITER-GEn compared to Re2G? That is to say, we finetune the generation based on the retrieval outputs instead of using in-context learning?\n\nUpdated: I like to thank the authors for their feedbacks. However, the reply does not clearly address Q1, and since the novelty is not the strongest point of the paper, I like to keep my scores as it is.", "paper_topic_and_main_contributions": "The paper proposes ITER-RETGEN, which synergies retrieval and generation in an iterative manner. The process starts with a generation output, then concatenate the generated output with the retrieval result to generate next generated output. The process repeats until we reach a maximum number of iterations. The authors evaluate the proposed method on different tasks and datasets and show the effectiveness of ITER-RETGEN over Self-Ask and other baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The following shows the similar process of distilling reranker to retrieval (Section 3.4). As such, the authors should cite this reference.\n\n[1] Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2G: Retrieve, Rerank, Generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2701–2715, Seattle, United States. Association for Computational Linguistics."}}, {"review_id": "zuk8jiYJkb", "reviewer": "Reviewer_ij2Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time, insightful comments, and questions. We provided our responses below.\n\n> The concept presented in this paper is rather straightforward and seemingly trivial. The iterative retrieval approach, which uses the output from the previous round as input in the current retrieval process, is not a novel idea, and the paper does not highlight any unique aspects related to LLM. The study primarily applies this concept to retrieval-augmented LLM research with minor modifications, even lacking a heuristic solution to determine the number of iterative rounds.\n\nWe demonstrated in this paper that, Iter-RetGen, despite being simple, obtains strong results on various knowledge-intensive tasks, outperforming state-of-the-art methods including ReAct and Self-Ask while causing fewer overheads of both retrieval and generation. We also explored how we can further adapt the retriever with model generations, and achieved significantly better results.\n\nAn important aspect related to LLMs is whether they can leverage both intrinsic and extrinsic knowledge flexibly and effectively. We showed that Iter-RetGen performs significantly better, regardless of whether in-context extrinsic knowledge mentions the answers or not.\n\nAs for how to determine the number iterations, we showed in Table 2 that the second iteration usually gives the greatest performance boost, and already achieves significantly better or competitive results compared with baselines. Therefore, we recommend using 2 to 3 iterations. With the number of iterations increasing, performance improves more slowly, and overheads increase, too.\n\n> A notable omission in the paper is the absence of a comparison with the DSP framework, which would be the most appropriate baseline for this study.\n\nTo compare with the method in the DSP paper, we used the [code](https://github.com/stanfordnlp/dspy/issues/85) shared by the author. We made the following adjustments for fair comparisons:\n\n* We changed DSP to use the same (1) model (i.e., text-davinci-003), (2) retrieval system (i.e., Contriever-MSMARCO on Wikipedia 2017 for HotPotQA), (3) number of paragraphs retrieved for each query (i.e., 5), (4) in-context QA pairs (for demonstration), and (5) the evaluation set (i.e., the first 500 questions from the dev set of HotPotQA) as in our paper.\n* The original DSP method produces an answer using sampling followed by majority voting. We changed DSP to use greedy decoding to produce the answer.\n\nHere are the experimental results on HotPotQA.\n\n| Method            | EM       | F1       | Acc      |\n| ----------------- | -------- | -------- | -------- |\n| Direct            | 31.6     | 44.7     | 53.3     |\n| ReAct             | 24.9     | 44.7     | 61.1     |\n| Self-Ask          | 36.8     | 55.2     | 64.8     |\n| DSP               | 43.8     | 55.0     | 60.8     |\n| Iter-RetGen (T=2) | **44.1** | **58.6** | **71.2** |\n\nIter-RetGen outperforms DSP in terms of the three metrics.\n\n> The experiments conducted solely utilize the text-davinci-003 model, with no accessible open-sourced models included. Moreover, the paper lacks any discussion concerning potential data leakage in the tested benchmarks when using text-davinci-003.\n\nIt would be interesting but challenging to measure data leakage without access to the training data. However, we believe Llama2 should be much less prone to data leakage than text-davinci-003, if such a concern exists.\n\nWe experimented with the open-sourced Llama2-13B and Llama2-70B, without changing anything else. The accuracies of model predictions are presented as follows:\n\n| Model                  | Llama2-13B |                 |            | Llama2-70B |                 |            |\n| ---------------------- | ---------- | --------------- | ---------- | ---------- | --------------- | ---------- |\n| Dataset                | HotPotQA   | 2WikiMultiHopQA | StrategyQA | HotPotQA   | 2WikiMultiHopQA | StrategyQA |\n| Direct (w/o retrieval) | 36.4       | 31.6            | 60.5       | 47.2       | 39.0            | 72.7       |\n| CoT (w/o retrieval)    | 43.0       | 33.2            | 63.7       | 55.2       | 46.0            | 72.7       |\n|                        |            |                 |            |            |                 |            |\n| Direct                 | 51.8       | 38.6            | 63.3       | 58.6       | 45.1            | 73.3       |\n| ReAct                  | 36.0       | 27.5            | 61.5       | 42.6       | 36.8            | 69.5       |\n| Self-Ask               | 45.8       | 38.5            | 63.3       | 58.4       | 53.2            | 71.7       |\n|                        |            |                 |            |            |                 |            |\n| Iter-RetGen (T=1)      | 53.8       | 44.6            | 62.8       | 64.4       | 55.1            | 74.8       |\n| Iter-RetGen (T=2)      | **57.8**   | **48.0**        | **67.2**   | **67.8**   | **57.9**        | **76.6**   |\n\nAs can be seen in the table, Iter-RetGen consistently outperforms all baselines significantly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The present paper demonstrates strong motivation and brings up a crucial and intriguing question in the realm of LLM.\n- With its well-crafted writing and descriptive figures, this paper effectively conveys its message.\n- The paper offers a comprehensive set of experiments complemented by an extensive analysis that covers various aspects in detail.", "reasons_to_reject": "- The concept presented in this paper is rather straightforward and seemingly trivial. The iterative retrieval approach, which uses the output from the previous round as input in the current retrieval process, is not a novel idea, and the paper does not highlight any unique aspects related to LLM. The study primarily applies this concept to retrieval-augmented LLM research with minor modifications, even lacking a heuristic solution to determine the number of iterative rounds.\n- A notable omission in the paper is the absence of a comparison with the DSP framework, which would be the most appropriate baseline for this study.\n- The experiments conducted solely utilize the text-davinci-003 model, with no accessible open-sourced models included. Moreover, the paper lacks any discussion concerning potential data leakage in the tested benchmarks when using text-davinci-003.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an iterative retrieval-generation synergy framework for large language model and shows improved performance in question answering, fact verification and commonsense reasoning tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method to synergize retrieval and generation iteratively by using a response from a previous iteration at a retrieval step to refine the output. Evaluation on 4 multi-hope QA datasets, 1 fact verification and 1 commonsense task show consistent improvements.

Reviewers agreed that the method is simple yet effective (vbix, yjLF, ij2Q), and acknowledged that experiments are comprehensive, along with a new, more reliable metric proposed (vbix, yjLF, ij2Q) and insightful analysis (yjLF).

However, two reviewers pointed out limited novelty (yjLF, ij2Q). Also, reviewers mentioned how the method solves the issues with previous method mentioned in the paper—failing to process all retrieved knowledge and needing to use retrieval multiple times—is unclear (vbix, yjLF). Authors are encouraged to revise the paper to address this concern in the next version of the paper.
Decision: Accept-Findings