Query Rewriting in Retrieval-Augmented Large Language Models
Authors: Xinbei Ma
Yeyun Gong
Pengcheng He
hai zhao
Nan Duan
Venue: EMNLP 2023 Main (2023)
Keywords: large language model
retrieval augmentation
query rewriting.

Abstract: Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline, making remarkable progress in knowledge-intensive tasks. This work introduces a new framework, Rewrite-Retrieve-Read instead of the previous retrieve-then-read for the retrieval-augmented LLMs from the perspective of the query rewriting. Unlike prior studies focusing on adapting either the retriever or the reader, our approach pays attention to the adaptation of the search query itself, for there is inevitably a gap between the input text and the needed knowledge in retrieval. We first prompt an LLM to generate the query, then use a web search engine to retrieve contexts. Furthermore, to better align the query to the frozen modules, we propose a trainable scheme for our pipeline. A small language model is adopted as a trainable rewriter to cater to the black-box LLM reader. The rewriter is trained using the feedback of the LLM reader by reinforcement learning. Evaluation is conducted on downstream tasks, open-domain QA and multiple-choice QA.  Experiments results show consistent performance improvement, indicating that our framework is proven effective and scalable, and brings a new framework for retrieval-augmented LLM.

Query Rewriting for Retrieval-Augmente Large Language Models
Xinbei Ma Yeyun Gong H. Pengcheng He* ", Hai Zhao 1,2, Nan Duan
'Department of Computer Science and Engineering, Shanghai Jiao Tong University
KKey Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University MMicrosoft Research Asia "Microsoft Azure Al sj tjtmabbesjjuu edu. cn, zhaohai @cs. sjtu, edu. cn, (yegong, nanduanJeme jemiaannnmmmoooooo crosoft. com, Herbert. www.heeeeeee.cm.. com
Abstract
of the real world. Thus, LLMs still have to face the issue of hallucination (Yao et al., 2023; Bang et al., 2023) and temporal misalignment (Rötteer and Pierrehumbert, 2021; Luu et al., 2022; Jang et al., 2022). This affects the reliability of LLMs and hinders wider practical application, because the consistency between the LLM responses with the real world needs further validation. Exist- ing work has proved that incorporating external knowledge (i.e., non-parametric knowledge) with internal knowledge (i.e., parametric knowledge) can effectively alleviate hallucination, especially for knowledge-intensive tasks. In fact, retrieval- augmented LLMs have been shown so effective that they have been regarded as standard solu- tion to alleviate the factuality drawbacks in naive LLM generations. Retrieval augmentation is ap- plied to select relative passages as external contexts for the language model, which retrieve-then-rent framework (Lewis et al., 2020b; Karpukhin et al., 2020; Izacard et al., 2022). Take the open-domain task (open-domain QA) as an example, a retriever first searches for related documents for question. Then the LLM receives the question and the documents, then predicts answer.
Large Language Models (LLMs) play pow- erful, black-box readers in the retrieve-then- read pipeline, making remarkable progress knowledge-intensive tasks. This work in- troduces new framework, newrite-Retrieve Read instead of the previous rerrieve-then-rede for the retrieval-augmented LLMs from the per- spective of the query rewriting. Unlike prior studies focusing on adapting either the retriever or the reader, our approach pays attention to the adaptation of the search query itself, for there is inevitably a gap between the input text and the needed knowledge in retrieval. We first prompt an LLM to generate the query, then use a web search engine to retrieve con- texts. Furthermore, to better align the query to the frozen modules, we propose a trainable scheme for our pipeline. A small language trainable rewriter to cater
model is adopted to the black-box LLM reader. The rewriter is trained using the feedback of the LLM reader by reinforcement learning. Evaluation is con- ducted on downstream tasks, open-domait QA and multiple-choice QA. Experiments results show consistent performance improvement, in- dicating that our framework is proven effective and scalable, and brings new framework for retrieval-augmented LLM
As most LLMs are only accessible through infer- ence APIs, they play the part of black-box frozen readers in the pipeline. This makes previous re- trieval augmentation methods that require complete access (Lewis et al., 2020b; Guu et al., 2020; Izac- ard et al., 2022) no longer feasible. Recent studies on retrieval-augmented language models lean more on the LLM-oriented adaptation. An idea is to train dense retrieval model to cater to the frozen lan- guage model (Shi et al., 2023). By using feedback from the LLM as a training objective, the retrieval model is tuned for better LLM input contexts. An- other research line focuses on the design of inter- actions between the retriever and the reader (Yao et al., 2023; Khattab et al., 2022), where both the
1 Introduction
Large Language Models (LLMs) have shown re- markable abilities for human language processing and extraordinary scalability and adaptability in few- or zero-shot settings. et al., 2022; Brown et al., 2020; Chowdhery et al., 2022), How- ever, the training process depends on large-scale high-quality corpora but without the perception
Work done during an intersship Microsoft Research Asia. Equal contribution. &Corresponding author.
This paper was partially supported by Joint Research Project Yangize River Delta Science and Technology Inno- vation Community (No. 2022CSJGG1400)

---

retriever and the reader are usually frozen. The idea is trigger the emergent ability through carefully crafted prompts or a sophisticated prompt pipeline. Multiple interactions with external knowledge al- low the L.M to approach the correct answer step by step.
Vicuna-13B (Chiang et al., 2023) as the LLM reader. The results show that query rewriting con- sistently improves the retrieve-augmented LLM performance. The results also indicate that the smaller language model can be competent for query rewriting.
However, there are still problems remaining to be solved. Existing approaches overlook the adap- tation of the query, i.c., the input of the retrieve- then-read pipeline. The retrieval query is either original from datasets directly determined by the black-box generation, thus is always fixed. How- ever, there is inevitably a gap between the input text and the knowledge that is really needed to query. This limits performance and places a burden on retrieval capability enhancement and prompt engineering.
To sum up, proposed novel retrieval- augmentation method, rewrite- is the first framework where the imput text is adapted for the frozen retriever and LLM reader. We introduce a tuneable scheme with a small, trainable model, achieving performance gains with less resource consumption.
Related Work
2.1 Retrieval Augmentation
Language models require external knowledge to al- leviate the factuality drawbacks. Retrieval augmen- tation has been regarded as the standard effective solution. With a retrieval module, related passages are provided to the language model as the context of the original input. Thus factual information like common sense or real-time news helps with output prediction through contextualized reading compre- hension.
In consideration of this issue, this paper pro- poses
new framework for retrieval augmentation, which can be further tuned for adapting to LLMs. In front of the retriever, a step of rewriting the input added, filling the gap between the given input and retrieval need, as is shown in Figure 1. We adopt the off-the-shelf tool, an internet search engine, as the retriever, which avoids the maintenance of the search index and access up-to-date knowledge (Lazaridou et al., 2022). Different from previous studies (Khattab et al., 2022; Yao al., 2023) that require the mem- ory of multiple interaction rounds between the re- triever and the LLM for each sample, the motiva- tion of our rewriting step is to clarify the retrieval need from the input text.
Earlier studies use sparse retriever (Chen et al., 2017) or dense retriever (Karpukhin et al., 2020) in front of pre-trained language model (PrLM).
The neural retriever and reader are both PrLMs of trainable size like BERT (Devlin et al., 2019) or BART (Lewis et al., 2020a). Hence, the whole serriee--tee--eadeer framework is a tuneable end- to-end system, where the retrieved contexts can be regarded as the intermediate results (Karpukhin et al., 2020; Lewis et al., 2020b). Approaches to smooth the two-step framework are proposed to op- timize the retrieval and the reading comprehension (Sachan et al., 2021; Lee et al., 2022: Jiang et al., 2022). More recently, retrieval remains powerful enhancement as the size of models and data scales rapidly (Mallen et al., 2022; Shi al., 2023; Brown et al., 2020). On the other hand, retrieval enhance- ment can compensate for the shortfall in parameter size, compared to large-scale language models. For example, by jointly training the retriever and the reader, Atlas (Izacard et al., 2022) shows few-shot performance on par with 540B PalM (Chowdhery et al., 2022) but be of 50x smaller size.
We also propose a trainable scheme for our rewrite-retrieve-read framework (Figure 1 (c)). The black-box retriever and the reader form frozen system. To further smooth the steps of our pipeline, we apply small, trainable language model to perform the rewriting step, denoted as the rewriter. The rewriter trained by reinforcement learning using the LLM performance as a reward, learning to adapt the retrieval query to improve the reader on downstream tasks.
Our proposed methods are evaluated on Enowledge-intensive downstream tasks including open-domain QA (HotpoQA (Yang et al., 2018), AmbigNQ (Min et al., 2020), PopQA (Mallen al., 2022)) and multiple choice QA (MMLU (Hendryeks et al., 2021)). The experiments are implemented on la-ge (Raffel et al., 2020) as the rewriter, ChatGPT (Ouyang et al., 2022) and
The Internet as a knowledge base More related to our work, the search engine can assume the role of the retriever and use the Internet as the source of

---

Input
Input
Input
Example
Input:
Black-box LLM
Small PrLM
What profession Nicholas Ray
Retriever
Rewriter
Rewriter
Elia Kazan have in commun?
Query
Query
Query: Nicholas Ray profession
Query:
profession
Documents
Web Search
Web Search
Retriever
Retriever
Elii Karan was an American film and Ubeatree director, producer.
seroenwriter and actor, deseribed
Black-box LLM
Documents
Documents
Nicholas Ray American auther and director, original name Raymond Nicholas Kienzle, born August 1911. Calevville, Wisconsin, U.S....
Reader
Black-box L.LM
Black-box LLM
Oulput
Reader
Reader
(render HGi (reitiever
dircctor
Oulput
Reward
Output
(a) Retrieve-then-read (bJRewvrte-rrttiiov-rraddd
(c) Trainable rewrite-retrieve-read
Figure 1: Overview of proposed pipeline. From left to right, we show (a) standard retrieve-then-read method, (b) LLM as query rewriter for our rewrite-retrieve-red pipeline, and (c) our pipeline with trainable rewriter.
external knowledge. Komeili al. (2022) use an internet search for relevant information based on the dialogue history to perform dialogue response generation. SceKeR (Shuster et al., 2022) use a single Transformer to iteratively perform search query generation, then knowledge extraction for dialogue generation and sentence completion. For large-scale models, web search still shows effec- tive for knowledge augmentation (Lazaridou et al., 2022), fact-checking (Menick et al., 2022), and LLM agent enhancement (Yao al., 2023).
(Yao et al., 2023) and Self-Ask (Press al., 2022) combines the Chain-of-Thought (CoT) (Wei al., 2022; Wang et al., 2022) and inter-actions with web APIs. Only relying on prompt construction, Re- Act provides novel baselines for interactive tasks. Demoostrrace-Search-Poooo (DSP) (Khattab et al.. 2022) defines sophisticated pipeline between an LLM and retriever. Unlike ReAct, DSP integrates prompts for demonstration bootstrap besides multi- hop breakdown and retrieval.
Despite the promising performance in the zero or few-shot setting, the behavior of LLMSs sometimes needs adjustments. A feasible approach is to ap- pend trainable small models in front of or after the LLM. The small models, of the system, can be fine-tuned for optimization. RePlug (Shi et al., 2023) is proposed to fine-tune a dense retriever for the frozen LLM in the retrieve- then-read pipeline. The retriever is trained under the LLM's s supervision to retrieve documents that are suitable for the LLM. With the same purpose, Directional Stimulus Prompting (Li et al., 2023) deploys small model to provide the LLM with stimulus (e.g., keywords for summarization, di- alogue actions for response generation), which is updated according to the LLM reward.
2.2 Cooperation with Black-box LLMs
Large Language Models, such as ChatGPT (Ouyang et al., 2022), Codex (Chen et al., 2021), PaLM (Chowdhery et al., 2022), emerge impres- sive natural language processing ability as well as remarkable scalability. This leads to tendency to embrace LLMss on wide range of NLP tasks. However, LLMs only accessible in most cases, which is because Some like Chat- GPT are not open-source and kept private: (ii) The large parameter scale requires computational re- sources that are not always affordable to users. This constraint means nothing is available except input and output texts.
part of the parameters
black box
Existing studies have proved that LLMs' abili- ties can be better leveraged by carefully designed interaction methods. GenRead (Yu et al., 2023) prompts an L.LM generate context instead of deploying a retriever, showing that LLMs can re- trieve internal knowledge by prompting. ReAct
Different from the inspiring work mentioned above, our proposed pipeline contains a query rewriting step in front of the retrieve-then-read module. We further propose a trainable scheme with small rewriting model, which is a novel enhancement for sarrivaal-augment LLM by re-

---

constructing the search query.
pseudo dataset for the query rewriting task. In- spired by recent distillation methods (Hsich et al., 2023; Ho et al., 2022), we prompt the LLM to rewrite the original questions x in the training set and collect the generated queries as pseudo la- bels. The collected samples are then filtered: Those that get correct predictions from the LLM reader are selected into the warm-up dataset, denoted Drrain {(x, k)lğ y). The rewriter Gg is fine- tuned on Drrain with the standard log- likeliioooodd as the training objective, denoted
3 Methodology
We present Rewrite-Retriev--Reie a pipeline that improves the retrieval-augne augmented LLM from the perspective of query rewriting. Figure shows an overview. This section first introduces the pipeline framework in section 3.1, then the trainable scheme in section 3.2.
3.1 Rewrite-Retrieve-Read
A task with retrieval augmentation can be de- noted as follows. Given a dataset of a knowledge- intensive task (e.g.. open-domain QA), D {z, i 0, 2........ is the imput to the pipeline, is the expected output (e.g., the correct answer). Our pipeline consists of three steps. (i) Query rewrite: generate query it for required knowledge based the original input x. (ii) Retrieve: search for related context, doc. (iii) Read: comprehend the input along with contexts [doc, x:] and predict the output ý.
Learm
Σ logpot it x),,)). (1)
The rewriter model after warm-up shows mod- est performance, which depends on the pseudo data quality and rewriter capability, Highly relying on the human- prompt line, # can be sub- optimal. The relatively small scale of the rewriter size is also a limitation of the performance after the warm-up. Then we turn to reinforcement learning to align the rewriter to the following retriever and LLM reader.
x (e.g., a question)
A straightforward but effective method is to ask an LLM to rewrite queries to search for informa- tion that is potentially needed. We use a few-shot prompt to encourage the LLM to think, and the output can be none, one
3.2.2 Reinforcement Learning
To further fine-tune the rewriter to cater to the LLM reader, we adopt a policy gradient reinforcement learning framework.
queries to search.
3.2 Trainable Scheme
Task Formulation In the context of reinforce- ment learning, the rewriter optimization is for- mulated as a Markov Decision Process 5-tuple (S, A, P. R, 7) (i) The state space S is a finite set limited by the vocabulary and the sequence length. (ii) The action space A is equals the vocabulary. (iii The transition probability P is determined by the policy network, which is the rewriter model Gg- (iv) The reward function gives a reward value that depends on the current state. The pol- icy gradient is derived from rewards, used as the training objective. (v) 7 denotes the discount fac- tor. More specifically, the rewriter Go after the warm-up is the initial policy model mo- At each step I, the action at is to generate the next token åg based on the observation of the present state, 8t = [x, (r]. When the generation is stopped by the End-Of-Semtence token, one episode is ended. After finishing the retrieval and reading, a reward is computed by evaluating the final output, i.e., a score for the LLM reader prediction.
Besides, total reliance on frozen LLM has shown some drawbacks. Reasoning errors or invalid search hinders the performance (Yao et al., 2023; BehnamGhader et al., 2022). On the other hand, retrieved knowledge may sometimes mislead and compromise the language model (Mallen et al., 2022). To better align to the frozen modules, it is feasible to add a trainable model and adapt it by taking the LLM reader feedback reward.
Based on our framework, we further propose to utilize trainable small language model to take over the rewriting step, as is shown in the right part of Figure 1. The trainable model is initial- ized with the pre-trained 15-large (770M) (Raffel et al., 2020), denoted as trainable rewriter, Go- The rewriter is first trained on pseudo data to warm up ($3.2.1), then continually trained by reinforcement learning ($3.2.2).
3.2.1 Rewriter Warm-up
The task, query rewriting, quite different from the pre-training objective of woweenccett-seuennee generative models like T5. First, we construct a
Policy Optimization We adopt Proximal Policy Optimization (PPO) (Schulman et al., 2017), fol- lowing (Ramamurthy et al., 2022). Maximization

---

of the expectation of the reward is formulated as
in-context learning (Brown al., 2020; Min et al., 2022). Our prompt follows the formulation of fin- struction, demonstrations, inputJ, where the input is x. The instruction is straightforward and demon- (2) strations are 1-3 random examples from training sets and are kept constant across all runs, mainly for the task-specific output format illustration, i.e., short phrase an answer for HotpoiQA, and an option answer for MMLU. For the training scheme in $3.2, we fine-tuning T5 the rewriter. Retriever We use the Bing search engine as the retriever. It requires no candidate index construc- tion like a dense retriever, nor candidates like a textbook. But it allows for a wide knowledge scope and up-to-time factuality. With Bing API, the re- trieval is performed in two approaches. (i) For all retrieved web pages, concatenate the snippets that are related sentences selected by Bing. This method is similar to using search engine in a browser, input a query and press Enter, then col- lect the texts shown on the search result page. (ii) For retrieved web pages, we request the URLs and parser to get all the texts. This is similar to clicking on items on the search result page. Then we use BM25 to keep those with higher relevance scores with the query, reducing the document length.
max E p [R(x,))
max E (e.oiroo min K (st, az); clip (,,, 0, - E, +)) A (s),
(ct St)
k1,0 po (at st)
where 0' is the temporarily fixed policy for sam- pling and 8 is updated. A denotes the advantage function, which is formulated based on the estima- tion of value network Vg. The value network V is initialized from the policy network To. The formu- lation follows Generalized Advantage Estimation (GAE) (Schulman et al., 2015).
6.- = R (st: at) + Vo (Se Vo (8),
20
19 (s,, a1) = Σ
X 61es,
r'=0
where A is the bias-variance trade-off parameter. The reward function R reflects the quality of the generated queries, which needs to be consistent with the final evaluation of the task. å is fed to the retriever and the reader for final prediction J. A part of the reward function is the measures of compared to the golden label y (e.g., exact match and F1 of the predicted answers), denoted as Rim- Besides, a KL-divergence regularization is added to prevent the model from deviating too far from the initialization (Ramamurthy et al., 2022; Ziegler et al., 2019).
Reader The reader a frozen LLM, where we adopt ChatGPT (gpt-3.5-turbl and Vicuna-13B. It performs reading comprehension and prediction with few-shot in-context learning. In our prompt, following the brief instruction and the demonstra- tions, the input is z or [doc, { with retrieval aug- mentation.
R (st, az) = R/m v) BKL (No (o))
(4)
It has been proved that both the phrasing of prompt lines (Zhang et al., 2023a) and the selection of demonstrations show effects on the in-context learning performance (Su et al., 2022; Zhang et al., 2023b). As it is not the focus of this work, we pay no more attention to prompt editing.
The final loss function is composed of policy loss and value loss.
T min mainkk,, A clip A²'),
Lo = TST
7ES 0=
5 Experiments
Ls | ΣΣ TES t= Cyp La AwLo.
(Vo (8:) -Ra)3
5.1 Task Settings
5.1.1
Open-domain QA
(5)
Three open-domain QA datasets are used for evalu- ation. (i) HotPotQA (Yang et al., 2018 8) consists of complex questions that require multi-hop reason- ing. We evaluate the full test set. (ii) AmbigNQ (Min et al., 2020) provides disambiguated version of Natural Questions (NQ) (K wwiatkowski et al., 2019). For ambiguous questions in NQ, minimal constraints are added to break it into several similar
Here, S denotes the sampled set, and T is for step numbers.
4 Implementation
Rewriter For the frozen pipeline in $3.1, we prompt an LLM to rewrite the query with few-shot

---

Direct prompt
Answer the question in the following format, end the answer with [demonstration) Question: (x} Answer:
Reader prompt in retrieval-augment pipelines
Answer the question in the following format, end the answer with *ss* [demonstration] Question: [doc] (x)
Answer:
Prompts for LLM as a frozen rewriter
Open-domaie QA: Think step by step to answer this question, and provide search engine queries for knowledge that you need. Split the queries with and end the queries with ***", (demonstration) Question: {z) Answer: Multiple choice QA: Provide a better search query for web search engine to answer the given question, end the queries with [demonstration] Question: {x] Answer:
Table Prompt lines used for the L.MSs.
but specific questions. The first 1000 samples are evaluated the test set. (iii) PopQA (Mallen et al., 2022) includes long-tail distributions as it contains more low-popularity knowledge than other popular QA tasks. We split the dataset into 13k for training and 714 for testing.
We also use Vicuna- 13B as the reader for evalua- tion due to the rate limit issue of ChatGPT. More information on datasets and training setup are pre- sented in the appendix.
5.2 Baselines
Open-domain QA benchmarks are sets of guestion-answer pairs denoted {(g, a);). We use ChatGPT for both the reader and the frozen rewriter. The evaluation metrics are Exact Match (EM and Fi scores. For the reward function in RL, we use an indicator to reward if the retrieved content hits the answer and penalize if misses the answer, de- noted as Hit. The total reward is a weighted sum of EM, F1, and Hit.
The following settings are implemented to eval- uate and support our methods. (i) Direct: The standard in-context learning without any augmen- tations. (ii) Retrieve-then-read: The standard method. Retrieved documents are concatenated with the question. as frozen rewriter: As is introduced in $3.1, we prompt a frozen LLM reason and generate queries by few-shot in-context learning. (iv) Train- able rewriter: Applying the fine-tuned rewriter, (6) the output queries are used by the retriever and the reader. Table presents prompt line forms. Please note that the prompts for prediction are kept the same for each task.
(iii) LLM
Hit [ in doc, -1 else
Rim FM + X + An Hit.
5.1.2 Multiple-choice QA
For multiple-choice QA, our evaluation is con- ducted on Massive Multi-task Language Under- standing (MMLU) (Hendrycks et al.. 2021), exam question dataset including 4 categories: Hu- manities, STEM, Social Sciences, and Other. Each category split into 80% for the training set and 20% for the test set.
5.3 Results
Experimental results on open-domain QA are re- ported in Table 2. For the three datasets, query rewriting consistently brings performance gain with both frozen rewriter and trainable rewriter. On AmbigNQ and PopQA, the standard retrieval augments the reader, indicating useful external knowledge retrieved. On HotpotQA, the stan- dard retrieval hurts the reader. This shows that using complex questions as queries cannot com- pensate for the parametric knowledge, but bring noises instead (Mallen al., 2022). This suggests that multi-hop questions are suitable queries for the web search engine. The scores increase by adding the rewriting step. On PopQA, our trainable rewriter surpasses standard retrieval while being inferior to the LLM rewriter. This indicates that the
Multiple-choice QA can be formulated as {lg' ) where o' - -q. Co. C1, C22 ca). denotes the options, generally there are four for each ques- tion. The retrieved documents that are included in the officially provided contaminated lists ignored. The questions with options rewritten into search queries. The answer is one option. EM is reported as metrics and used for the reward.
Rim E.M.
(7)
We use ChatGPT as a frozen rewriter and the reader.

---

(a)HotpotQA
distillation of query rewriting is sub-optimal. The scores on multiple-choice QA are presented Table With ChatGPT as reader, can be ob- served that query rewriting improves the scores in most of the settings, except for the social sciences category. With Vicuna as a reader, our method achieves more gains the four categories com- pared to ChatGPT. This agrees with the intuition that more powerful reader has parametric memories, thus more difficult compensate with external knowledge.
45
33
44
32
31
retrieve-then-res
rewriter
30
41
5
15
25
Interation
(b)AmbigNO
48
47-
60
59
45-
Model
EM
F
Retereaet
55
44
LLM rewriter
HotpotQA
57
Direct
32,36
43.05
2
4
10
Retrieve-then-raad
30.47
41,34
Interation
LLM rewriter
32.80
43.85
(c/PoDQA
Trainable rewriter
34.38
45.97
46
49
NQ
48
Direct
42. 10
53.05
44
47
Netrieve-then-reae
45,80
58,50
43
46
LLM rewriter
46.40
58.74 74
45
Trainable rewriter
47.80
60.71
41
44
LLM rewriter
PopQA
40
43
Direct Retrieve-thent then-read
41.94
44.61
10
43.20
47.53
Interation
LLM rewriter
46.00
49.74
Trainable rewriter
45.72
49.51
Figure 2: Reinforcement learning validation scores of (a)HotpotQA, (b)AmbigNO, and (c)PopQA. The solid lines show EM (red) and (blue) numbers through training iterations. The dashed lines EM scores of the standard retrieve-then- then-read method (orange) and retrieval with L.M the rewriter (green).
Table 2: Metries of open-domain QA.
MMLU
EM
Human.
STEM Other
Social
ChaiGPT
Direct
75.6
58.8
69.0
71.6
It can be observed that the curves show upward trends with some fluctuations on all the datasets. For multi-hop questions in HotpotQA, the standard retrieval is relatively weaker. Complex questions can be not specific search queries and show a larger gap from rewritten queries, i.e., the green and red lines. (ii) On AmbigNQ and PopQA. our method surpasses the baselines after several iterations (3 or 4). This indicates that the RL training stage can compensate for the insufficiency of the distillation on the pseudo data during warm-up training. (iii) In particular, on PopQA, the trainable rewriter re- mains inferior to the LLM rewriter. This can be explained as the dataset is constructed for adaptive retrieval (Mallen et al., 2022), which only uses re- trieval where it helps to avoid harmful redundant retrieval. Thus, "None" is a possible query that means no retrieval. This causes more complex- ity and uncertainty. L.M rewriter knows better when the retrieval is needed for itself as reader, although the rewriting step is not concatenated as
Retrieve-then read
76.7
70.0
78.2
LLM rewriter
77.0
72.6
76.4
Vicuna-13 13B
Direct
39.8
34.9
50.2
46.6
teerieve-then-read
40.2
39.8
55.2
50.6
LLM rewriter
42.0
41.5
57.1
52.2
Trainable rewriter
43.2
40.9
59.3
51.2
Table Metries of multiple choice QA.
6
Analysis
6.1
Training Process
The training process includes two stages, warm-up and reinforcement learning. This section shows the validation scores of the three open-domain QA datasets for further analysis. Figure 2 presents the metric scores through training iterations in the process of reinforcement learning. As the rewriting models have been warmed up on the pseudo data before RL, scores at "O iteration" denote the ability acquired from the warm-up training.

---

the input context of the reader.
Example I: multi-hop question
Hit Correct
We calculate the performance of query "None". The questions that can be correctly answered with- out retrieval (i.e., the "Direct" method) are those samples that need no more context. Comparing this retrieval-free set with those that rewritten to be "None" query, the F score of the LLM rewriter is 71.9% and the T5 rewriter score 67.1%. If we consider the questions that can be correctly an- swered without retrieval but go wrong with retrieval as the retrieval-free feeeval-feeeeeee set, the F1 scores are 78.7% for LLM rewriter and 77.4% for T5.
Q0: The youngest daughter of Lady Mary-Gaye Curzon stars with Douglas Smith and Lucien Laviscount what 2017 film?
x
Q1: the youngest daughter of Lady Mary-Gaye Curzon; 2017 film stars Douglas Smith and Lucien Laviscount
Q2: Lady Mary-Gaye Curzon youngest daughter
2017 film with Douglas Smith and Lucien Laviscount
Example 2:
Q0: What 2000 movic does the song "All Star" appear in?
Q1: movie "All Star" 2000
Q2: movie Star" song
Model
EM
Hit ratio
Example 3: multiple choice
No retrieval
42.10
53.05
Q0: A car-manufacturing factory considering site for its plant. Which of the following would community planners be most concerned with before allowing the plant to he built? Options: A. The amount of materials stored in the plant B. The hours of operations the new plant The effect plant will have on the environment D. The work environment for the employees al the plant
Upper bound
58.40
69.45
100
Retrieve-then-reud
w/ snippet
38,70
50.50
61.1
w/ BM25
45.80
58,50
76.4
LLM rewriter
X
x
w/ snippet
39.80
52.64
63.5
w/ BM25
46.40
58.74
77.5
Trainable rewriter
BM25
47,80
60.71
82.2
Table 4: Retrieval analysis on AmbigNQ.
Q1: What would community planners most concerned with before allowing a car- manufacturing factory to be built?
6.2
Retrieval Result
Our proposed method is a pipeline framework, in- stead of an end-to-end system. The query rewrit- ing first affects the retrieved context, then the con- text makes a difference to the output of the reader. Hence, QA metrics are indirect measurements. We take closer look at the retrieved context and the reader capability through the retrieval metric, hit ratio. After text normalization, the hit rate is com- puted measure whether the retrieved context con- tains the correct answers.
Figure 3: Examples for intuitive illustration. QO denotes original input, QI is from the LLM rewriter, and Q2 is from the trained T5 rewriter. Hit means retriever recall the answer, while Correct for the reader output.
while query rewriting makes progress both set- tings. We also observed that the improvement in the hit rate of the retriever is more significant than the improvement in the reader. This consistent with the findings in related search (Mallen et al., 2022; Liu et al., 2023).
Table 4 shows the scores on AmbigNQ. The scores in the second line are computed on selec- tion of the samples whose retrieved contexts hit correct answers (under the standard retrieve-then- read setting). The scores show the approximate upper bound ability of the reader with retrieval aug- mentation, abbreviated as the "upper bound" score. The effectiveness of retrieval is proved compared to the no retrieval setting (the first line). For each retrieval method, two settings are presented: (i) collecting Bing snippets, (ii) selecting from URLs by BM25. The metrics show that content selection with BM25 recalls better documents than snippets,
6.3 Case Study
To intuitively show how the query rewriting makes difference in the retrieved contexts and prediction performance, we present examples in Figure to compare the original questions and the queries. example 1, the original question asks for film that the youngest daughter of Lady Mary- Curzon co-stars with two certain actors. Both query and query 2 put the keyword film forward, closely fol- lowing the youngest daughter of Lady Mary-Gaye Curzon. With both, the actress Charlotte Calthorpe and her movie information can be retrieved and the answer is included. The second an example where the query from the LLM rewriter failed but
"Our trainable rewriter is adapted to the retriever using BM25 during RI. training. Using the output queries of the test set after training. the snippet hit rate is 73.4%.

---

the query from T3 gets the correct answer. The number 2000 is misunderstood in query 1, while query 2 keeps 200 movie together, avoiding mean- ingless retrieval. Example 3 is for multiple choice. The query simplifies the background and enhances the keyword community planner. The retrieve con- texts are mainly about Introduction to Community Planning where the answer environment appears several times.
References
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do. Yan Xu, and Pascale Fung. 2023, A multitask, multilin- gual, multimodal evaluation of chatgpt on reason- ing, hallucination, and interactivity. arXiv preprint Siii:2302.04023..
Parishad BehnamGhader, Santiago Miret, and Siva Reddy, 2022. Can retriever-augmented language models reason? the blame game between the triever and the language model. arXiv preprint arXiv:2212.09146.
7 Conclusion
This paper introduces the Rewrite-Retriev--R pipeline, where query rewriting step is added for the retrieval-augmented LLM. This approach is applicable for adopting a frozen large language model as the reader and real-time web search engine as the retriever. Further, we propose to ap- ply a tuneable small language model the rewriter, which can be trained to cater the frozen retriever and reader. The training implementation consists of two stages, warm-up and reinforcement learn- ing. Evaluation and analyses on open-domain QA and multiple-choice QA show the effectiveness of query rewriting. Our work proposes a novel retrieval-augmented black-box LLM framework, proves that the retrieval augmentation can be en- hanced from the aspect of query rewriting, and provides new method for integrating trainable modules into black-box LLMs.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-191..
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Association for Computational Linguistics (ACL).
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert. Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing large language models trained code. CoRR, abs/2107 fass/207..0374 03374..
Limitations
We acknowledge the limitations of this work. (i) There is still a trade-off between generalization and specialization among downstream tasks. Adding a training process, the scalability to direct transfer is compromised, compared to few-shot in-context learning. (ii) The research line of LLM agent has shown impressive performance but relies on mul- tiple calls to the LLM for each sample (Khattab al., 2022; Yao et al., 2023), where the LLM plays as an agent to flexibly call the retriever multi- ple times, reads the context in earlier hops, and generates follow-up questions. Different from these studies, our motivation is to enhance the one- turn retriever-then-readd framework with trainable query rewriter. (iii) Using web search engine as the retriever also leads to some limitations. Neu- ral dense retrievers that are based on professional, filtered knowledge bases may potentially achieve better and controllable retrieval. More discussion is included in the appendix.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vieuna: An open- source chatbot impressing gpt-4 with 90%* chattptt quality.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint orXiv:2204.02311
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of

---

deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. NAACL-HLT 2019. Minneupolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmentt augmented dialogue generation. In Proceed- ings of the 60th Anmual Meeting of the Association for Computational Linguistics Volume I: Long Pa- pers), pages 8460-8478, Dublin, Ireland. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938. PMLR.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115.
Dan Hendrycks. Collin Burns Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021, Measuring massive multitask language understanding, Proceedings of the International Con- ference on Learning Representations (ICLR).
Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paran- jape. Christopher Manning, and Kyoung-Ge Woo. 2022. You only need one model for open-domain question answering. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 3047-3060, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Namgyu Ho, Laura Schmid, and Se- Young Yun. 2022. Large language models reasoning teachers. arXiv preprint (rXXi:2212.10077
Cheng- Yu Hsich, Chun-Liang Li. Chib-Kuan Ych, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. ArXiv, abs/2305.02301.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: denoising sequence-to- no-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties, ACI. 2020, Online, July 5-10, 2020, pages 7871 7871-7880. Association for Computational Linguisties.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot Learning with Retrieval Aug- mented Language Models.
Joel Jang. Seonghyeon Ye, Changho Lee, Sohee Yang. Joongbo Shin, Janghoon Han, Gyconghun Kim, and Minjoon Seo. 2022. Temporalwiki: A lifelong bench- mark for training and evaluating ever-evolving lan- guage models.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020b. Retrieval- for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-994 99744
generation
Zhengbao Jiang, Luyu Gao, Jun Araki, Haibo Ding, Zhiruo Wang. Jamie Callan, and Graham Neubig. 2022. Retrieval as attention: End-to-end learning of retrieval and reading within single transformer. In Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), Abu Dhabi, UAE.
7ekun Li, Baolin Peng. Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. 2023. Guiding large language models via directional stimulus prompting. arXiv preprint arXiv:2302.11 2520.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape. Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How lan- guage models use long contexts. arXiv preprint 0rXiv:2307.03172
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan. Kar- ishma Mandyam, and Noah A. Smith. 2022. Time waits for no one! analysis and challenges of tem- poral misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944-5958, Seattle, United States. Association for Computational Lin- guistics.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search- predict: Composing retrieval and language mod- els for knowledge-intensiee NLP. arXiv preprint arXiv:2212. 14024.

---

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Investigating effectiveness and limitations of parametric and non- parametric memories. arXiv preprint.
Devendra Singh Sachan, Siva Reddy, William L. Hamil- ton, Chris Dyer, and Dani Yogatama. 2021. End-to- end training of multi-document reader and retriever for open-domain question answering. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 25968-25981.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young. Lucy Campbell- Gillingham, Geoffrey Irving, et al. 2022. Teaching language models to support answers with verified quotes. Xrxiv preprint arXiv:2208.11147.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-114 11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage esti- mation. arXiv preprint wrXiv:5566002388
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv: 17Xi::1077.0347.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open-domain questions. In EMNLP.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends huggingface. arXiv preprint arXiv:2303.1 175.80800.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems. 35:27730-27744.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James, Mike Lewis, Luke 7ettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXtv:2301.12652
Ofir Press. Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03330.
Kurt Shuster, Mojtaha Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston, 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arub Emirates, December 7-11, 2022, pages 373-393, Association for Computational Lin- guistics.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toollim: Facilitating large language models master 16000+ real-world apis. ArXiv preprint, abs/2307.16788.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits transfer learning with a unified text- to-text transformer. Journal of Machine Learning Research, 21(14)):1-67.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf. Luke Zettlemoyer, Noah A Smith, al. 2022. Selec- tive annotation makes language models better few- shot learners. arXiv preprint arXiv: 2tt.::22200119.7..
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, and Denny Zhou 2022. Self- consistency improves chain of thought reasoning in language models. CORR, abs/2203.11171
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NewrIPS.
Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400-2412, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for

---

diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Task
Training Set
Warm-up
Set
HorpotQA
90.4k
37.5k
7.4k
AmbigNQ
19.4k
8.6k
lk
PopQA
13.0k
6.0k
0.7k
Humanities
3.8k
1.5k
0.9k
STEM
2.4k
0.9k
0.6k
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023, ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
Other
2.6k
1.3k
0.6k
Social Science
2.4k
1.3k
0.6k
Table 5: Metries multiple choice QA.
steps to 5120, 10 threads, 512 steps for each. After sampling, the policy network is trained for (2.3.4) epochs, with learning rate as 2c-6 and batch size as (8.16). ' and X are 1.0. 3 in Eq. 4 is dy- namically adapted according to Ramamurthy et al. (2022); Ziegler et al. (2019),
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In International Confer- ence for Learning Representation (ICLR).
Tianjun Zhang, Xuezhi Wang. Denny Zhou, Dale Schu-
urmans, and Joseph E Gonzalez. 2023a. Tempera:
KL. (I (T)) KL 0.22, 0.2
= clip
Test-time prompt editing via reinforcement learning. The Eleventh International Conference on Learn- ing Representations.
KLtarget
B2+1 Br KBer),
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompt- ing large language models. In The Eleventh In- ternational Conference on Learning Representations (ICLR 2023).
where KLiarget is set to 0.2, K3 is set to 0.1. Po is initialized to be 0.001. The generation strat- egy follows the 4-beam search and returns the sequence. In the implementation of the BM25- based retriever, the textboxes from searched URLs are parsed from HTML code. We compute BM25 scores between the paragraph from each textbox and the query following the scikit-learn package, then keep those with higher scores until the re- served context reaches max length. In reinforce- ment learning, the results of AmbigNQ are with the BM25 method, while others snippets context.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. arXiv preprint arXhv:199.08893.
A Warm-up Dataset
For the warm-up training of the tuneable rewriter, we construct pseudo dataset for the query rewrit- ing task. For benchmarks that provide official train- ing and test splits (HotpotQA and AmbigNQ), we use the whole training set. For those that have no official splits (PopQA and MMLU), we randomly split the full dataset. In detail, PopQA contains 16 types of questions, thus split into 13k for training and 714 for testing following stratified sampling. For MMLU, each of the 4 categories is randomly split into 80% for the training set and 20% for the test set. Then the training sets cach bench- mark are used to derive the pseudo dataset for the query rewriting, i.e., Drrain = {(x, ž)l = y). We present the statistics of the splits and warm-up dataset in Table 5.
C
Web Search: Tool Use
Our proposed pipeline integrates an externally built web search engine as the retriever module. We present more discussion the advantages and dis- advantages here.
The usage of external tools expands the abil- ity boundary of language models, compensating for the parametric knowledge, and grounding the capabilities of language models to interact with en- vironments (Qin et al., 2023; Schick et al., 2023). Recent studies show trend to leverage plug-and- and- play tools like search engines to enhance language agents (Lazaridou et al., 2022; Menick et al., 2022; Shuster al., 2022: Shen et al., 2023). Search engine APIs are well-developed retrievers, saving efforts to build and maintain another retriever, like a Contriever. Accessible to the whole Internet, the web search retrieves from a wide-range, up-to-date
B Setup Details
For warm-up, we train the T5-large with 3e-5 learn- ing rate, ( (16, 20) batch size, for 16,8,12) epuchs. For reinforcement learning, we set the sampling

---

knowledge base. The temporal misalignment prob- lem on fixed candidate database can be alleviated. On the other hand, web search APIs are commer- cial products requiring subseriptions. Also, the vast amount of knowledge on the web can be difficult to control. The retrieved context from the Internet can be occasionally inconsistent, redundant, and toxic, which hinders the LLM reader.
Beyond retrieval augmentation, in general scope, other tools called by LLMs, like code in- terpreters, online models, and expert applications, are all similar to search engines, without trainable parameters to optimize. There could be a gap be- tween the LM and these tools. This paper proposes an idea to align them through a trainable small model.


Reviews:
[{"review_id": "OhAKWwclam", "reviewer": "Reviewer_ksRg", "summary": "", "questions": "Question A: What is the retriever HIT rate when using the Trainable T5 query rewriter? \nQuestion B: What are the specifics of the warmup dataset, number of samples, etc? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable feedback.\n(Notations: R: Reason to Reject; Q: Question for the authors)\n\n**R1&QB: Dataset for the trainable T5 warmup**\n\nThe dataset is introduced in sections 3.2.1 and 5.1.\n\nFor benchmarks that provide official training and test splits (HotpotQA and AmbigQA), we use the whole training set; For others (PopQA and MMLU), we randomly split them (#Line 422, 438). \n\nFor warmup, the training set is used to derive pseudo dataset as described in #Line 288-295.\n\nWe present the statistics of the training splits and warmup data here. We will add this to the paper and release the datasets.\n|\t   |HotpotQA\t|AmbigQA\t|PopQA\t|Human\t|STEM\t|Other\t|Social|\n|-------|---------------|----------------|-----------|-----------|-----------|-----------|-------|\n|Training\t|90.4k\t|19.4k\t        |13k \t|3.8k\t|2.4k\t|2.6k\t|2.4k|\n|Warmup\t|37.5k\t|8.6k\t        |6k\t        |1.5k\t|0.9k\t|1.3k\t|1.3k|\n\n**R2: Random demonstrations**\n\nThe demonstrations are from training sets and are kept constant across all runs. The demonstrations are mainly for output format illustration (#Line 371-372), i.e., a short phrase as an answer for HotpotQA, and an option as an answer for MMLU.\n\n**R3: Details for reproducibility**\n\nThanks for your advice. Some experiment setups are presented in Appendix A. We surely will add more details to the paper and release our code, results, and data. \n\n**R4&QA: HIT rate with trainable T5 query rewriter**\n\nThe Hit score is originally computed for preliminary experiments to prove the effectiveness of query rewriting and to provide an upper bound. To reply to this question, we conduct a quick experiment on AmbigQA with the T5 rewriter, and the best hit scores are 82.2 with BM25 and 70 with snippet.\n\nWe thank you for this question and will add more Hit scores.\n\n**Typos Grammar Style and Presentation Improvements**\n\nThank you for your careful reading. We will improve the writing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a new cascaded system that rewrites queries to retrieve better documents for improved performance on QA tasks using an LLM reader. \n\nThe smaller T5 rewriter model warmed up on a specially curated pseudo data and fine-tuned using LLM feedback as the RL reward outperformed the more compute intensive LLM query rewriter in all QA tasks except PopQA. \n\n", "reasons_to_reject": "The paper's results is currently not reproducible. No information has been provided on the curated dataset for the trainable T5 warmup. \n\nNo mention of the in context examples used to prompt the model to rewrite the query. Only mentioned that random samples are used, but no information on whether the demonstrations are samples randomly for each run or kept constant across all runs. \n\nNeed more details for reproducibility of the results. \n\nResults for retriever HIT rate when using the Trainable T5 query rewriter not provided. ", "questions_to_authors": "Question A: What is the retriever HIT rate when using the Trainable T5 query rewriter? \nQuestion B: What are the specifics of the warmup dataset, number of samples, etc? ", "paper_topic_and_main_contributions": "The paper focuses on how to best optimize the retrieved passages to augment the LLM and obtain best QA performance. \n\nThe main contribution of this paper is an NLP engineering experiment. \n\nThe paper shows improvements in QA performance with the proposed rewrite-retrieve-read QA approach. The approach entails rewriting the query to retrieve documents that better fit the question. Two rewriters are used - a frozen LLM rewriter and a smaller T5 model trained using RL to perform query rewriting based on rewards computed using LLM reader feedback. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall peer easy to understand. Few grammatical mistakes, would suggest another review. \n\nHowever, Analysis section was a bit difficult to understand and required multiple reading passes to get clarity on the point trying to be conveyed, especially section 6.1"}}, {"review_id": "2nfu3PHxmq", "reviewer": "Reviewer_GP32", "summary": "", "questions": "* Question A: Can you describe how you obtain the demonstrations for your prompts that you mention in Table 1 and various parts of the paper? \n* Question B: Can you provide a description of BM25? It is currently missing from the paper based on what I can tell and would be good to clarify.\n* Question C: Does your RL training procedure apply a dense or sparse reward function?\n* Question D: Can you explain the fluctuations in performance in Figure 2? It appears that performance varies a lot and it's a bit hard to tell why that might be happening.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable feedback.\n(Notations: R: Reason to Reject; Q: Question for the authors)\n\n**R1: Search results**\n\nWe really appreciate this advice. Web search retrieves flexible, various external knowledge (#Line 600). In our preliminary experiments, the Hit scores are relatively consistent.\n\nAs a convenient tool, web search API has been widely adopted as a plug-and-play, auxiliary component for LM-based systems (#Line 174-187). Following these methods, we apply web search as a light sparse retriever, while our proposed method is applicable to all sparse retrievers. We will surely expand experiments and present more detailed analyses.\n\n**R2: Details about choosing different architectures & PPO-based training procedure**\n\n**Architectures:**\nRelevant content is scattered in the paper.\n\nReader: Our main idea lies in tunable augmentation for LLMs. ChatGPT is an appropriate foundation model. Vicuna-13B is an acceptable alternative that our computing resources can afford (#Line 450).\n\nRetriever: Web search is applied as it is a flexible and light retriever, and the Internet is a wide, open knowledge source avoiding search index reconstruction (#Line 376, 174-187). \n\nRewriter: (1) Using ChatGPT as a rewriter provides an unsupervised upper bound, and is a preliminary verification for the proposed pipeline. (2) For the trainable rewriter, T5 is a seq2seq LM pre-trained with a task-specific prefix. We define the prefix “rewrite a better search query” to tune T5 to our rewriting task.\n\n**PPO:** \nThe concerned training settings are addressed as follows and will be strengthened in the later version of this paper.\n\nHotpotQA and AmbigQA provide both training and test splits, we use the whole training set. For PopQA and MMLU, we randomly split them for training and evaluation (#Line 422, 438). We present the statistics of the training splits and warmup data (#Line 293-295) here.  The convergence is observed by comparing the changes in training loss and validation score.\n\nWe will add details to the paper and release the datasets.\n\n|\t   |HotpotQA\t|AmbigQA\t|PopQA\t|Human\t|STEM\t|Other\t|Social|\n|-------|---------------|----------------|-----------|-----------|-----------|-----------|-------|\n|Training\t|90.4k\t|19.4k\t        |13k \t|3.8k\t|2.4k\t|2.6k\t|2.4k|\n|Warmup\t|37.5k\t|8.6k\t        |6k\t        |1.5k\t|0.9k\t|1.3k\t|1.3k|\n\n**R3: Query rewriting examples**\n\nIn Figure 3, the rewritten queries show differences compared to the original inputs (Q0), The examples mainly show the changes of redundancy removing, swapping, and keyword reserving.\nWe thank you for this advice and will improve the case study. \n\n**QA: Demonstrations**\n\nThe demonstrations are random samples from training set and are kept constant across all runs. The demonstrations are mainly for output format illustration (#Line 371-372), i.e., a short phrase as an answer for HotpotQA, and an option as an answer for MMLU.\n\n**QB: BM25**\n\nThank you. We will add more detailed descriptions. \n\nFor each searched URL, the textboxes are parsed from HTML code. We compute BM25 scores between the paragraph from each textbox and the query (following scikit-learn package), then keep those with higher scores until the reserved context reaches a max length (#Line 386-390).\n\n**QC: Dense or sparse reward function**\n\nThis is a good question as the sparse reward problem indeed exists, however, it does not matter too much for our proposed solution due to the following detailed settings. 1) The output of query rewriting is not long. There is relatively small room in need for exploration. \n2) The warmed-up rewriter has acquired some domain knowledge.\n\n**QD: Fluctuations in Figure 2**\n\nThe fluctuations mostly occur in HotpotQA, which can be attributed to the uncertain hop step numbers that bring about extra reasoning difficulty.\n\n**Missing References & Typos Grammar Style And Presentation Improvements**\n\nThanks very much for your advice. We will add this reference and correct the typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposes a new method for retriever-based LLMs changing the pipeline from retrieve-then-read to rewrite-retrieve-read. The method, and relevant details, are generally well described and clearly laid out.\n* The results presented in the experiments generally support the notion that query rewriting improves performance of LLMs on a diverse set of QA benchmarks testing.\n* The authors provide an analysis of that provide further insight into the inner working of the proposed method, including specific query rewriting examples and a discussion when training a rewriter may or may not be beneficial.\n* The paper is generally well written with relevant descriptions of methods, experiments, results and analysis.", "reasons_to_reject": "* It is unclear if retrieval through a web API can provide consistent results given that the results of API queries could change. It would be good for the authors to include a discussion on this.\n* The motivation for choosing different architectures as different parts of the pipeline could be more detailed, as could the details of the PPO-based training procedure (e.g. how many samples were used for training and refinement and how was convergence determined).\n* The query rewriting examples in Figure 3 show only minimal changes to the text of the query. This makes the effect of the rewriter a bit nebulous. A way of showing the impact of the rewriter on changing the queries (other than final results) could be useful.", "questions_to_authors": "* Question A: Can you describe how you obtain the demonstrations for your prompts that you mention in Table 1 and various parts of the paper? \n* Question B: Can you provide a description of BM25? It is currently missing from the paper based on what I can tell and would be good to clarify.\n* Question C: Does your RL training procedure apply a dense or sparse reward function?\n* Question D: Can you explain the fluctuations in performance in Figure 2? It appears that performance varies a lot and it's a bit hard to tell why that might be happening.", "paper_topic_and_main_contributions": "The papers proposes a new method for retriever augmented LLMs that includes query rewriting, which changes the pipeline of retriever-based LLMs from retrieve-the-read to rewrite-retrieve-read. The authors first introduce some of the common problems with LLMs that can be addressed by retriever-based models, such as hallucinations and factuality errors. Next, the authors describe some shortcomings with the currently applied retrieve-then-read pipeline and propose their rewrite-retrieve-read pipeline that involves query rewriting by an additional LLM in the loop of retriever-based models. Next, the authors describe related work on retriever LLMs, using internet search queries as a knowledge base and interactions with black-box commercial LLMs with APIs.\n\nIn the methods section, the paper describes the important components of retriever based language models, as well as their method. This includes the training scheme for query rewriting with a warm-up objective and a reinforcement learning (PPO) based training. The authors also describe implementation details of their method, including model types used for each part of the pipeline, as well as different training settings (e.g. trainable and non-trainable query rewriters).\n\nIn their experiments, the authors test their method on various QA datasets, including HotPotQA, AmbigQA and PopQA for open-domain QA and MMLU for multiple-choice QA, all of which test different capabilities of the pipeline. In their results, the authors show that rewriter queries for retriever-based models generally improves performance for open-domain QA and for multiple choice QA (with the exception of social science). The authors of their results, including training details, retrieval abilities and a case study. The analysis is followed by a conclusions and a brief discussion of limitations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 099: We adopt *an*\n* Line 188: Cooperate with Black-box LLMs -> Cooperation with Black-box LLMs\n* Line 398: proved -> proven"}, "extra_text": {"Missing_References": "The authors mention reasoning errors of retriever-based models in Section 3.2, which can benefit from including insights from:\n\nBehnamGhader, Parishad, Santiago Miret, and Siva Reddy. \"Can retriever-augmented language models reason? the blame game between the retriever and the language model.\" arXiv preprint arXiv:2212.09146 (2022)."}}, {"review_id": "GybR51dJUk", "reviewer": "Reviewer_rmM6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable feedback.\n(Notations: R: Reason to Reject; Q: Question for the authors)\n\n**R1: Novelty & compute/performance tradeoffs**\n\nWe clarify the novelty and compute/performance tradeoffs compared with REPLUG, the first work to adapt a dense retrieval model to the LM.\n\n**Novelty:**\n\nOur work enhances the retrieval-augmented LLM from totally another perspective. REPLUG finetunes a dense retriever, while our method finetunes a rewriter to align the query to the retriever and reader. \n\nCompared to REPLUG, our novel motivation is in two folds: (1) Input texts themselves need to be optimized (#Line 86-92). (2) Query rewriting can align queries to sparse, frozen retrievers, avoiding the maintenance of the search index over time (#Line 99-102).\n\n**Compute/performance tradeoffs:**\n\nIn the training process, REPLUG finetunes a Contriever (BERT-base) with a 36M-candidate set and performs an ensemble between top-20 docs. Our method finetunes a rewriter (T5-large), taking about 0.5-1.5 hours for each epoch (including many retries due to the rate limit issue of ChatGPT, which causes extra delay).\n\nIn terms of performance, compared to the approach with frozen pipeline, the trainable Contriever improves open-domain QA by 1.22% on average and MMLU by 0.56%. Our trainable T5 rewriter improves open-domain QA by 2.41% on average and MMLU by 0.84%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem to tackle is well-motivated. \n2. The proposed approach is solid and reasonable. \n3. the experiments are quite convincing with the case study + analysis are quite convincing. \n", "reasons_to_reject": "1. I'm concerned about the novelty of the method, and how this way of learning to rewrite the query compares to other approaches that finetunes the reader or the retriever, in compute/performance tradeoffs. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper adapts the search query to improve the retrieval augmented system by adding a query rewriting step and freeze the LM and the retriever. The methods trains a smaller model as the rewriter. The rewriter is first warmed up by training on pseudo rewrites that leads to  correct y, and then finetuned via reinforcement learning, with the reward function be a task specific metric (like Exact Match). The methods is evaluated on open-domain QA, and multiple choice QA, and show empirical gains. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LpFUVdQv1F", "reviewer": "Reviewer_JBWk", "summary": "", "questions": "What is the relative contribution of the warm-up and RL phases for the trained rewriter approach?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and careful review. \n(Notations: R: Reason to Reject; Q: Question for the authors)\n\n**R1: DSP & Self-ask**\n\nDSP and Self-ask are excellent works on retrieval-augmented LLM. But we clarify that our method differs from these works.\n\n1) Our main idea is the trainable pipeline, adding a trainable small model rather than relying on the expensive, frozen LLM for everything. \n2) The rewriting in their methods relies on multiple interactions with the LLM (#Line 76-82), e.g., to read the context retrieved in earlier hops and generate follow-up questions, which requires multiple calls to the LLM for each sample. Their approaches are closer to the *LLM agent* research line (like ReAct, etc.), from which our motivation differs. Our motivation is to enhance the retriever-then-read framework with a trainable query rewriter.\n\nOur statement (#Line 133-135) does not intentionally ignore contributions of previous work. What we tried to emphasize is that our proposed framework is the first to adapt the input text by a trainable rewriter to the frozen retriever and LLM reader (Similar expressions are in #Line 271-273, 114-117). \n\nWe will improve the expression to avoid misunderstanding.\n\n\n**R2&R3: DSP scores**\n\nDSP achieves impressive scores on HotpotQA, which is in no contradiction to that our experiments can demonstrate the effectiveness of our proposed method.\n\n1) As mentioned in our reply to R1, DSP emphasizes sophisticated multiple interactions with the LLM agent. For each question, the LLM is called iteratively for each action step and takes an action based on previous retrieval results. In contrast, only focusing on query rewriting, our method only calls LLM once as the reader (and extra once if the LLM is used as the rewriter).\n\n2) DSP also includes a demonstration enhancement step with a 16-shot training set, while we use unchanged random 1-3 demos (#Line 371-372).\n\n3) ChatGPT API being a limited resource, related works often use different evaluation settings for different approaches. DSP is tested on 1000 samples while we use the total 7405 test samples.\n\nWe hope our above responses could clarify our novel motivation and the differences from related works.\n\n**R4: Web search**\n\nIn #Line 477, we want to express that using original questions in HotpotQA as the retrieval query is not suitable, and a rewriting step is required, as mentioned in motivation (#Line 86-92).\n\nWeb search does have pros and cons (#Line 600) but has been a widely used retrieval tool for LLMs (#Line 174-187). The proposed method is applicable to all sparse retrievers, while web search has advantages as a light sparse retriever with a wide knowledge source. \nWe thank you for this question and will add more discussion to our paper.\n\n**Q1: Warm-up and RL phases**\n\nThis is discussed in Section 6.1. Figure 2 shows the scores during RL, where the start points (0 iteration) show the scores after warm-up. The warm up step is for a warm start, where the rewriter is inferior to the LLM rewriter. Although the distillation on the pseudo data is insufficient, the RL phase compensates for the gap.\n\n**Missing References**\n\nThank you. We will add the paper of Self-ask as a reference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a creative paper with informative evaluations. I really like the idea of having small, custom built models for in-context learning systems, as an alternative to relying on LLMs for everything.\n", "reasons_to_reject": "1. The paper says, \"rewrite-retrieve-read is the first framework where input text is adapted for the frozen retriever and LLM reader.\" This is not correct. The DSP paper, which the authors cite, uses exactly this sort of approach, and for very similar tasks. DSP also does query rewriting for conversational QA. Self-Ask is another approach where the query is iteratively rewritten to improve retrieval and make the job easier for the LLM.\n\n2. The DSP paper also evaluates on HotPotQA. The DSP program does iterative query rewriting to break the question down. The EM reported is 51.4% EM, which is substantially higher than the best result reported in this paper: 34.38% EM. The DSP baseline retrieve-then-read system is at 36.9. I am fine with the idea that the goal of the current paper is to test query rewriting, and so maybe we don't expect SOTA overall, but the DSP approach is so similar that I can't see a justification for excuding it, and the fact that the retrieve-then-read baseline is stronger is certainly informative.\n\n3. It is not surprising that a trainable rewriter does better than a system that has to reply on frozen components. That said, the DSP program that got 51.4% EM on HotPotQA uses only frozen components.\n\n4. The retrieval engine for the paper is a Web search engine. I suppose this has independent interest, but it is a challenge for reproducibility, and the authors themselves seem to indicate that it was a suboptimal choice (line 477). It might have been more productive to use a retriever and retrieval model for the relevant domains.", "questions_to_authors": "What is the relative contribution of the warm-up and RL phases for the trained rewriter approach?\n", "paper_topic_and_main_contributions": "This paper proposes a Rewrite-Retrieve-Read pipeline for in-context learning. The essence of the idea is to allow an LM to rewrite incoming queries to make them more amenable to effective retrieval-augmented generation. The paper explores two rewriting approaches: one using a frozen LLM and one using a custom T5-large module trained with reinforcement learning. Overall, the approach yields solid results, with the best results coming from the approach that trains a custom query rewriter.\n", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please see above."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a rewrite-retrieve-read approach as opposed to a retrieve-read approach in order to refine the retrieval query to be tailored toward the needed knowledge that should be retrieved from the knowledge source. The rewriter model is learned using reinforcement learning where the reward is based on the end prediction from the language model. Evaluation on three open-domain QA datasets and 1 multi-choice dataset reveals the effectiveness of the method.

The reviewers acknowledged that the contribution is in proposal of a new method that is shown to be effective (ksRg, GP32, rmM6), experiments and analysis are thorough (GP32, rmM6), and the paper is well-written (GP32).

Reviewers raised concerns on heavy reliance on close-sourced models such as web API (GP32, JBWk) and qualitative analysis (rewriting seems minor based on examples) (GP32). Also, reviewers pointed out lack of details reported in the paper (ksRg) and lack of comparison to prior work (rmM6, JBWk). These concerns are sufficiently addressed by authors during the discussion period. Authors are encouraged to add details and discussion in the final version of the paper if accepted.
Decision: Accept-Main