Expand, Highlight, Generate: RL-driven Document Generation for Passage Reranking
Authors: Arian Askari
Mohammad Aliannejadi
Chuan Meng
Evangelos Kanoulas
Suzan Verberne
Venue: EMNLP 2023 Main (2023)
Keywords: Synthetic document generation
Data augmentation
Information retrieval
TL;DR: This paper presents novel synthetic document generators for data augmentation in information retrieval, leveraging generative large language models to produce synthetic documents and enhance query quality, leading to effective retrieval models.

Abstract: Generating synthetic training data based on large language models (LLMs) for ranking models has gained attention recently. Prior studies use LLMs to build pseudo query-document pairs by generating synthetic queries from documents in a corpus. In this paper, we propose a new perspective of data augmentation: generating synthetic documents from queries. To achieve this, we propose DocGen, that consists of a three-step pipeline that utilizes the few-shot capabilities of LLMs. DocGen pipeline performs synthetic document generation by (i) expanding, (ii) highlighting the original query, and then (iii) generating a synthetic document that is likely to be relevant to the query. To further improve the relevance between generated synthetic documents and their corresponding queries, we propose DocGen-RL, which regards the estimated relevance of the document as a reward and leverages reinforcement learning (RL) to optimize DocGen pipeline. Extensive experiments demonstrate that DocGen pipeline and DocGen-RL significantly outperform existing state-of-theart data augmentation methods, such as InPars, indicating that our new perspective of generating documents leverages the capacity of LLMs in generating synthetic data more effectively. We release the code, generated data, and model checkpoints to foster research in this area.

Expand, Highlight, Generate:
RL-driven Document Generation for Passage Reranking
Arian Askari', Mohammad Aliannejadi², Chuan Meng²
Evangelos Kanoulas², Suzan Verberne*
LLiddn University baaskari,ssverbeeee @ liacs.leidenuniv.ll
2 University of Amsterdam imm.lianmeeadi.ccmmrg. www.me @uva.nl
Abstract
erating synthetic queries given documents in an existing corpus.
Generating synthetic training data based on large language models (LLMs) for ranking models has gained attention recently. Prior studies use L.LMs to build pseudo query- document pairs by generating synthetic queries from documents in corpus. In this paper, we propose new perspective data augmen- tation: generating synthetic documents from queries. To achieve this, we propose DocGen, that consists of three-step pipeline that uti- lizes the few-shot capabilities of LLMss. Doc- Gen pipeline performs synthetic document gen- eration by (i) expanding. (ii) highlighting the original query, and then (iii) generating a syn- thetic document that is likely to be relevant to the query. To further improve the relevance be- tween generated synthetic documents and their corresponding queries, we propose DoeGen- RI., which regards the estimated relevance the document as a reward and leverages reinforcement learning (RL) to optimize Doc- Gen pipeline. Extensive experiments demon- strate that DocGen and DocGen-RL signifi- cantly outperform existing ofate-of-the-art data augmentation methods, such as InPars, indi- cating that our new perspective of generating documents leverages the capacity of LLMs in generating synthetic data more effectively. We release the code, generated data, and model checkpoints to foster research in this area'.
However, previous work suffers from the fol- lowing limitations: (i) generated synthetic queries tend to have larger lexical overlap with relevant documents than human queries, leading to mis- match between synthetic and real data; (ii) generat- ing queries from a document in an existing corpus cannot cover queries for which the relevant infor- mation is spread over multiple documents; more- over, in reality, there could be difficult and complex queries for which there are very few to zero rele- vant documents in existing corpora; thus retriever trained on such data is less likely to learn to retrieve effectively for complex queries.
To overcome these limitations, we propose to generate synthetic documents from given queries, ? This paradigm has three benefits: (i) taking user queries as input makes the resulting training data more representative of real information needs; (ii) generating synthetic documents enables create relevant documents for complex queries that have few or no relevant documents in an existing corpus; LLMs are pre-trained on a vast number of corpora, and have the potential in generating relevant documents for complex queries by aggre- gating relevant information they have seen during pre-training; (iii) Yu et al. (2023) show that generat- ing document is closer to the language modeling objective of the LLM pre-training than generat- ing queries, thus generating documents can better utilize the knowledge stored LLMs parameters, producing hugher-quality text.
1 Introduction
Data augmentation for information retrieval (IR) promising research
has gained attention area. Previous studies use large language mod- cls (LLMs) (Zhao et al., 2023) to generate synthetic training data for retrievers (Jeronymo et al., 2023; Dai et al., 2023; Boytsov et al., 2023; Bonifacio et al., 2022), significantly improving effectiveness of unsupervised retrievers. Specifically, these stud- ies all build pseudo query-document pairs by gen-
We use the few-shot capabilities of LLMs (Dong et al., 2022) to generate synthetic documents given queries. Our preliminary experiments identified two challenges: (i) feeding raw queries to LLM could result low-quality synthetic documents?;
'In this paper. we interchangeably use the words 'docu- ment' and "passage.' "This observation is bused on open-source L.Mss, which
'nttps://g thub. com/a aan-askar1/ docgen

---

Generating synthetic document per query by DocGen
Filtering the pairs of query and synthelic document that are unlikaly be relevant
Training a ManaT5 re-ranker on the filtered data
set of queries
Figure I: The illustration of training passage reranking using the DoeGen.
we think is because raw queries contain limited helpful information to trigger the knowledge stored in an LLM; and (ii) it is important to ensure the close relevance between a query and its correspond- ing synthetic document. To address the first chal- lenge, we propose three-stage pipeline for doc- ument generation, named DocGen pipeline, that effectively uses the few-shot capabilities of LLMs. The idea of the DocGen pipeline to enrich raw queries before generating synthetic documents to effectively trigger the knowledge stored in an LLM. Specifically, the DocGen pipeline consists of three steps: (i) query expansion, which expands a raw query to clarify the information need, (ii) query highlighting, which highlights the important terms of the expanded query, and (iii) synthetic document generation, which generates the final synthetic doc- ument given the query after expansion and high- lighting. To solve the second challenge, we propose RocGen-RL, which regards the relevance between a generated document and its corresponding query reward function, and propose to leverage rein- forcement learning (RL) to optimize the DocGen pipeline. Our preliminary experiments show that challenging step: ineffec- tive highlighting, i.e., highlighting words that are not semantically the most important words in the query, has a negative impact on the final relevance. Therefore, DocGen-RL optimizes query highlight- ing stage through RL.
Walker, 1994) is widely leveraged (Anand et al., 2021; Nogueira et al., 2020) for finding the top-k documents to be reranked. In this work, we focus on data augmentation for the reranking stage where BM25 is used for first-stage passage retrieval and MonoT5 is used for reranking passages.
Our main contributions are as follows:
We propose a new direction on data augmenta- tion for IR: generating synthetic documents from queries. We devise the DocGen pipeline, a three- step few-shot learning- baaninggbbssed pipeline, which im- proves the quality of synthetic document genera- tion by expanding and highlighting queries. To enhance the relevance between generated syn- thetic documents and their corresponding queries, we propose DocGen-Ree RL,, which regards the rele- vance as a reward and leverages RL to optimize the DocGen pipeline.
We conduct a comprehensive analysis of the pro- posed methodology. Through rigorous evaluation and experimentation, we thoroughly examine the impact of each step of the DocGen pipeline and RL training, shedding light on their strengths, limitations, and potential areas of improvement.
query highlighting
2 Related Work
Data augmentation for IR using LLMs. In- Pars (Bonifacio et al., 2022), Promptagator (Dai et al., 2023), and InPars-v2 (Jeronymo et al., 2023) utilize LLMS to generate synthetic queries for given documents. Particularly, InPars-v2 achieves state-of-the-art results on the BEIR bench- mark (Thakur et al., 2021) by using open-source language model, GPT-J-6B (EleutherAI, 2023), for query generation and a powerful external reranker, MonoT5-MS MARCO (Nogueira et al., 2020), filter the top- 10k high-quality pairs of synthetic query-document pairs. Pseudo queries have also been used beyond data augmentation. For instance, Abolghasemi et al. (2023) leverages the pseudo queries for the estimation of retrievability bias in document retrieval.
The commonly used ranking pipeline consists of first-stage retriever, e.g., BM25 (Robertson and Walker, 1994), that efficiently retrieves set of doc- uments from the full document collection, followed by one or more rerankers (Nogueira et al., 2020) that improve the initial ranking. Currently, the most effective rerankers are T5-based rankers with a sequence-to-sequeee
architecture. In this pa- per, we refer to these rerankers as MonoT5. In the common reranking set-up, BM25 (Robertson and
contrasts with closed-source LLMs used for commercial pur- poses, such as CharGPT and GPT 3,5/4.0. These Mosed-source LLMs seem capable of generating higher-quality text directly from raw queries. However, we do not consider them to ensure our research remains independent from commercial organi- zations. Also, for closed models, it is unclear if there is data leakage between their pre-train pre-training data and the datasets we used for evaluation in this paper.
Synthetic document generation given queries for data augmentation has not been explored in prior work. The recent work by Askari et al. (2023a,b) compares the effectiveness of models trained by

---

Expanded and Highlighted query
Original query "converling stereo signal to mono signal called'
Expanded query "What the conversion of stereo signal to mono signal?'
Expanding query ICR
Highlighting query w/ ICR
'What the [conversion] of [stereo signal] [mono signal]?" www.............................
Document generation
Decument generation
Document generation
"Converting stereo signal to mono signal is callec
"The stereo signel the same es the mono signal. stareo signal is the same as the mono signal. The"
conversion stereo signal mono signal process that converts a signal from digital to analog signal. The signal then converted back to digital"
Figure 2: The DoeGen's 's pipeline: query expansion, query highlighting, and synthetic document generation. Few- shot examples (three) are provided at each step.
ChatGPT-ggenratter and human-written passages. That study has two limitations: ChatGPT is a closed-source LLM, and they only investigate the zero-shot setting of the LLM. In contrast, we fully investigate the effectiveness of open-source LLMs with using few-shot (three) examples in unified framework for modeling the task of data augmenta- tion for IR models. Gao et al. (2022) generates a hypothetical document and encodes the document into an embedding vector by an unsupervised con- trastively learned encoder to retrieve similar real documents based on vector similarity.
limited training data. Ramamurthy et al. (2023) present RL4LMs library for faciliating training gen- crative language models with RL and show how controlling the stability of RL could achieve high effectiveness for training LLMs. They implement proximal policy optimization (PPO)* the policy gradient method as recent works have shown PPO to be strictly superior to reinforce multiple do- mains. To the best of our knowledge, there no work on using RL on LLMs for IR data generation.
3 Synthetic Document Generation
Generating documents from queries is also ex- plored by Yu et al. (2023). They propose Gen- Read for the open-domain question answering task, achieving offtte----hh-aatt art performance. GenRead first prompts an LLM to generate contextual docu- ments based on given question, and then reads the generated documents to produce the final answer. Our work differs with GenRead as we focus on data augmentation for passage reranking, however, we can use GenRead as strong bascline for generat- ing documents for given queries to build synthetic query- document pairs. Recently, (Gabburo et al., 2023) propose data augmentation techniques im- prove generative QA model using supervision from automatic question answering evaluators.
In this section, we present the details of our pro- posed method, DocGen, and its reinforcement learning extension, DocGen-RL.
Task definition. Given a query E Q, where Q is set of queries, the goal of synthetic document generation is to generate document such that is relevant q, and use C {d,9) collection train retriever or reranking model.
3.1 DoeGen
We tackle the document generation task for IR, specifically the passage reranking task. DocGen consists of three main components illustrated in Figure 1: (i) document generation: DocGen pipeline that generates an initial set of synthetic documents for given queries; (ii) consistency filter- ing: it filters out the low-quality query-document pairs, using a consistency filtering approach (Dai et al., 2023); and (iii) reranking: fine-tunes MonoT5 reranker on the filtered synthetic data. Next, we detail different components of the Doc- Gen pipeline.
Reinforcement learning on LLMs. RL has been applied enhance models in various nat- ural language processing tasks. These tasks in- clude machine translation (Wu et al., 2016; Kiege- land and Kreutzer, 2021), summarization (Stien- non et al., 2020; Paulus et al., 2017), dialogue systems (Jaques et al., 2020), question genera- tion (Pang and He, 2021), and more. Ramamurthy al. (2023) gives extensive overview of re- search on RL for LLMs up to 2022. Li et al. (2023) train a policy language model with RI. leading to significant performance improvements across sum- marization and dialogue response generation with
3.1.1 Few-shot Guided Document Generation As illustrated in Figure 2, the document generation pipeline of DocGen itself consists of three steps,
"We refer readers to (Schulman et al., 2017) for a detailed explanation.

---

namely, (i) query expansion; (ii) query highlight- ing: and (iii) document generation. We implement each of the three steps using a prompt composed of three examples for few-shot learning.
optimal character for highlighting can be seen in Figure 4 and Section A.3 in the appendix.
Document generation. For few-shot document generation, we use the same examples with mod- ified queries that are expanded and highlighted - from the prompt template provided by Bonifacio et al. (2022). The InPars prompts consist of threc examples, each containing a query and relevant document. The prompt template can be seen in Figure 5 in the appendix.
Prompt design. Each prompt is the concatena- tion of prefix P; and query q where the prefix P. is the prompt template for step and consists of N pairs of examples, e.g., for query expansion, the Pi equals ((), ..., (qN; [ where qN refers to the original query and ON refers to the expanded query. For each step, we manually design a prompt, which is always the same regardless of the input query 9, i.c., we can potentially generate millions of synthetic training examples using only N manu- ally annotated examples. Given a query q in step i, we feed p:llg q the LLM that generates the desired output.
3.1.2 Consistency Filtering
To ensure the quality of the generated documents we apply the common consistency filtering ap- proach, originally proposed for synthetic query gen- eration (Dai et al., 2023). Consistency filtering has been proved crucial for synthetic data generation on QA tasks (Lewis et al., 2021). Dai et al. (2023) shows that consistency filtering based on the gen- erated data alone can work well. Inspired by these works, first use the expanded and highlighted query and synthetically generated document pairs to train an initial retriever which we call MonoTS CF. Given a docummet pair (q, d), we use the MonoT5 CF to predict the most relevant passages for q. We keep the query-document pair in the final dataset, only when d is the top-1 document returned by the retriever. One can argue that this technique is flawed because the filtering model (MonoT5 is trained on the same noisy synthetic data that it in- tends filter. However, Dai et al. (2023) show this filtering technique substantially reduces the num- ber of synthetic data and significantly improves retrieval performance.
Query expansion. Our preliminary experi- ments reveal that query expansion and highlighting improve the quality of the generated documents. We observe that expanding short sewword-bssed queries to longer queries in the form of natural language while highlighting the important words enhances the ability of LLMs to generate docu- ments with higher quality. using state-of-the-art IR methods on query expan- sion is out of the scope of our study since we do not apply query expansion to improve document retrieval. Instead, our goal is to re-construct the query in natural language for which the LLM can generate documents more effectively. The prompt template can be seen in Figure 3 in the appendix.
noteworthy that
Query highlighting. Weller et al. (2023) shows that using specific terms or phrases can trigger the LLM's memory to manipulate its output. For exam- ple, in their work they show that adding the phrase "As Wikipedia indicates" in the prompt manipulates the LLM to recall the text from Wikipedia more often. Similarly, we find that there are terms in BLOOM's training data (Scao al., 2022) that are highlighted with square brackets. Therefore, inspired by Weller et al. (2023) finding and our pre- liminary analysis, we highlight important words of the query using square brackets to manipulate the LLM to pay more attention to those words while it considers the expanded query in the document gen- eration process. As demonstrated in Figure 2, highlight the important words of a query by adding square brackets around them, e.g., "What is the [conversion] of [stereo signal] to [mono signal]?" Examples of this prompt template and analysis on
3.1.3 Reranking
We fine-tune a MonoT5 on the filtered data from the previous step and call it DocGen reranker.
3.2 RL-guided Document Generation
DocGen-RL is motivated by the challenges we encounter during query highlighting in few-shot guided document generation. We find out that when using few-shot examples, the model occasionally unreasonably highlights words that are not seman- tically the most important words of the query. For example, in some cases stop words, question marks, or other punctuation marks are highlighted. It is important emphasize that highlighting the query words is not a straightforward task with a single correct solution. The ultimate goal is to highlight query words that lead to generating higher-qualiyy

---

documents. Given the nature of the task and its dependence on the document generation module, one naive solution would be to try all possible high- lighting combinations and find the optimum, which is an NP-hard problem.
the vocabulary of the LLM based on the current weight values of the LLM. The vocabulary of the LLM represents the set of possible actions for the policy LM (agent). The policy LM takes into ac- count the input query and previous tokens to gener- ate the next token. The process continues until an end end-of-sequence token is generated, indicating the completion of the query.
To address this challenge, we employ RL to en- sure and improve the robustness of query highlight- ing and ultimately enhance the quality of document generation. During RL training, we optimize the process of query highlighting with the goal of gen- erating documents of higher quality that are more relevant to the highlighted query. To achieve this, we pass the highlighted query to the LLM, gener- ate document with few-shot learning, and use the predicted relevance by DocGen as the reward for the LLM that highlights the query. We optimize the LLM for highlighting while retaining the few-shot examples of the prompt during RL training. By using the DocGen reranker as our reward function, we make our reward function completely based on few-shot learning with three examples, dependent only few shots (three examples), as we train Doc- Gen reranker without any supervision of the actual dataset, i.e., MS MARCO.
Experimental Setup
Retrieval methods. We use a two-stage retrieval architecture (Nogueira and Cho, 2019) consisting of initial retrieval with BM25, followed by a neu- ral reranker. The collection is indexed using py- serini. 5 We retrieve 1000 candidate documents for each query using BM25. Subsequently, we rerank the candidate documents using MonoT5, which is an adaptation of the T5 model (Raffel et al., 2020) for text ranking proposed by Nogueira et (2020). We fine-tune MonoT5-base (220M parameters) with a constant learning rate of 10-3 and an equal number of positive and negative exam- ples in each batch of size 64. We did not conduct experiments with the 3B and I IB versions due to their computational cost.
3.2.1 Optimization Objective
Our goal is to optimize the query highlighting tasks in an RL setup by a policy language model (PLM) (Schulman et al., 2017) in order improve the quality of LLM's synthetic document genera- tion. To achieve this, we maximize the relevance measure, R. between the query after expansion and highlighting, and the generated document, y. The relevance measure R is the predicted relevance score by the DocGen reranker. Formally, we aim to maximize the following:
Baselines for data augmentation. We select the state-of-the-ar data augmentation method us- ing LLMs for the passage reranking task, InPars. as our main baseline. In addition to the InPars, we replicate GenRead (Yu et al., 2023) that recently has set a new state-of-the-ard for the open- open-domain question answering task, in which the focus is not on data augmentation for training rerankers, but the goal is generate relevant document for the given query and then generate the final answer from the generated document. We consider the few-shot variation of GenRead as another baseline for gener- ating documents given the query. We use the same prompts released by the author. Moreover, we replicate Query2Doc (Q2D) (Wang et al., 2023) that generate documents given a query in order to expand the query and improve retrieval effective- ness using the expanded query. We adapt Q2D for data augmentation by using their prompt and gener- ating documents. We use the generated documents for building the training data.
Eg-DD Pgr-D,x>-ppa pe-ppe m(
(oooo-DD--ppaallaa----pp--- (/) x) R[xx, y)].
(I)
In the three-step pipeline of DocGen, freeze the parameters of the LLMs for the first (cxpanding query) and the third (generating documents) steps, our aim is to optimize the LLM for the second step (highlighting query) as a policy LM and maximize the above objective 1 by using proximal policy optimization (PPO). We use DocGen reranker as our reward function.
The process of generating a sequence of high- lighted query tokens can be seen Markov de- cision process (MDP). In this process, there are different states, actions, rewards, and probabilities of transitioning between states. During each step, the agent chooses an action, which is a token, from
To ensure a fair comparison, we do not compare our results with InPars-v2 (Jeronymo et al., 2023) and Promptagator (Dai et al., 2023) results. InPars-
htttts/s//gttth...////sssssssss thub. http:hh..c.oy////eeeeeiieeeo

---

Table 1: Main results. On each dataset, we generate synthetic data given the queries from the training set for document generation methods and given the randomly sampled documents from the corpus for query generation methods. Note that on DL'20, we still utilize the training queries of MS MARCO since TREC-DL is solely an evaluation query set based on the corpus of MS MARCO. Significance shown with for DocGen compared to the best baseline (GenRead) and with for DocGen-RI. compared to DoeGen. We use MonoT5 with 220M parameters for all the rerankers. The cut-off for nDCG and MRR is 10 and for MAP is 1000, respectively. Statistical significance was measured with a paired t-test (p < 0.05) with Bonferroni correction for multiple testing.
NQ
MS MARCO
DL'20 MAP
HotpotQA nDCG
Fever nDCG
Retriever
Data Augmentor
nDCG
MRR
nDCG
First-stage retriever
BM25 (replicated)
.329
.187
.286
.480
.633
.651
Rerankers
MonoTS
InPars (Bonifacio et al., 2022)
.335
.259
.360
.576
MonoT5
InPars (replicated)
.337
.223
.357
.569
.627
.653
Mo
GenRead (replicated)
.368
.230
.354
.570
.629
.668
MonoT5
Q2D (replicated)
.309
.158
.252
.437
.610
.634
Rerankers w/ DocGen
MonoTS
DocGen (Ours)
.467
.2751
.3981
.580
.671
.693t
MonoT5
DocGen-RL (Ours)
.517"
.332*
.421"
.618
.663"
.720'
Human Annotations
M5
.567
.381
.491
.714
.695
.802
v2 uses the reranker that trained on MS MARCO to filter the generated synthetic data; after the syn- thetic data filtering and generation, InPars-v2 first warms up passage reranker using the whole train- ing set of MS MARCO and then continue training the reranker the generated synthetic data. How- ever, DocGen has not used the supervision from MS MARCO for either synthetic data filtering or passage reranker training. Promptagator performs first-stage retrieval (full ranking) with customized prompts per dataset, which is different from our ap- proach where we focus on reranking with a single prompt for all the datasets.
et al., 2018) datasets. We report the official metrics for each dataset, which are NDCG@ for BEIR's s datasets (NQ, HotPotQA, and Fever), MRR @ 10 for MS MARCO, and NDCG @ 10 and MAP@ 1000 for TREC-DL'20. The MS MARCO- passage dataset contains about million natural language queries (average length: 7.5 words) and has been extensively used train neural retrievers for ranking because of the large number of queries. Following prior work on MS MARCO (Khattab and Zaharia, 2020; Lin et al., 2021; MacAvancy et al., 2020; Zhuang and Zuccon, 2021; Zhuang et al., 2021), we use the dev set (~ 7k queries) for our empirical evaluation. TREC-DL'20 con- sists of 54 queries, and the retrieval for these queries is based the passage corpus of MS MARCO. We follow the common practice on TREC-DL (Craswell et al., 2021, 2020) and use DDGG@1 and M MAP@ 1000 for evaluation.
To have a fair comparison between data augmen- tation approaches in replication, use the same LLM (BLO0M-560M) and same filtering approach for all of the data augmentation method- ologies. We then generate an equal amount of syn- thetic data which 100, 000 synthetically gener- ated queries or documents followed by (Bonifacio et al., 2022).
LLM selection. In our preliminary experiments, we examine four LLMs, namely, LLaMA-7B (Tou- vron et al., 2023), Alpaca-Loora (Wang, 2023), Flan- T5- xxl (Chung et al., 2022), BLOOM- (Scao et al., 2022). We observe that the smallest and most efficient model, BLO0M-560M, consistently generates higher-quality documents compared the other models for our document generation task. Therefore, we select BLOOM-560M as the main
Dataset and metrics. We conduct our ex- periments the MS MARCO- pacco-p collec- tion (Nguyen et al., 2016) and the TREC Deep Learning track (TREC-DL'20) (Craswell et al.. 2021), well as the BEIR versions of Natural Questions (NQ) (Kwiatkowski et al., 2019), Hot- PotQA (Yang et al., 2018), and Fever (Thorne

---

What is the effectiveness of DocGen compared to the existing sate-of-theeann base- lines for data augmentation in IR models? LLM for our experiments. We found that Alpaca- Lora and Flan-T5 tend generate less informative and shorter text. LLaMA tends to produce noisy text, e.g., e. LaTeX Code.
Table Ablation study on DocGen and RL-training analysis on DocGen-RL. using nDCG@ 10 for evalua- tion.
Dataset
NQ-test
Training policy LLM with RL. We train the policy network for 30k episodes, 5 epochs per batch with batch size of 2 and learning rate of 10-6, We set the other hyperparameters to the optimal parameters that are suggested in (Li et al., 2023). If the agent (i.e., LLM) chooses an action (i.e., token), which is not a token the given query that is going to be highlighted, we assign a penalty reward of 0.25, drawing inspiration from (Li et al., 2023) that gives the same penalty for find- ing hint keywords from text to do summarization.
Ablation study on DocGen
DocGen w/o expanding
.370
DocGen w/o highlighting
.363
DocGen wo expanding & highlighting DocGen
.351
.467
RL-training analysis on DocGen-RL DocGen only RL on highlighting (DocGen-RL.)
.517
DocGen only RL on expanding DocGen + only RL on doe generation
.473
.448
Table 3: Impact of scaling on DocGen. Evaluation on NQ-test in terms of nDCG@11
5 Results
In this section, we address the following research questions, assessing the effectiveness of our pro- posed methods, DocGen and DocGen-RL, from different perspectives:
BLOOM-560M and ba-e (220M)
.467
BLOOM-3B TS-large (770M)
.482
.495
RQ1:
tion of the table focuses on the ablation study of DocGen's 's few-shot learning-based pipeline. ex- plores the impact of removing specific components on the model's performance. The results for these settings indicate that highlighting is more important than expanding while both steps do contribute to the effectiveness of the model. The second section of the table presents the analysis study conducted on DocGen-RL. The results in second section show that RL training on highlighting can achieve the highest improvement for DocGen. Similarly, like RL training for highlighting, we keep other steps frozen when we perform RL training for expand- ing or document generation, and we use the same reward function. RL training for expanding queries can slightly improve effectiveness. while RL train- ing for document generation decreases effective- ness. This could be because generating document more challenging task, and training the L.L on this task using RL could be more challenging. Scaling impact (RQ4). We investigate the im- pact of scale on DocGen from two perspectives: (i) the number of parameters of BLOOM for gen- cration, and (ii) the number of parameters of the trained MonoT5 the augmented data by Doc- Gen's pipeline. To address (i), analyze what extent DocGen improves by increasing the number of LLM's parameters and evaluate the effective- ness of DocGen with a version of BLOOM that
RQ2: To what extent does employing RL im-
prove our pipeline?
RQ3: What the impact of each step of few-
shot learning and reinforcement learning on DocGen and DocGen-RI?
RQ4: To what extent does scaling LLM or
MonoT5 improve the proposed method?
Main results (RQ1 and RQ2). Table presents the results of training MonoT5 on the augmented data using DocGen and DocGen-RL, comparing it to the previous statc-of-the-ar few-shot learn- ing data augmentation method, InPars, and other competitive baselines including GenRead (Yu et al., 2023). BM25 achieves competitive scores to InPars and GenRead HotpotQA and Fever. Overall, both variations of our proposed method, DoeGen and DoLGen-RL, outperform all the baselines and demonstrate significant improvements in retrieval performance across multiple datasets.
Ablation and RL analysis (RQ3). To inves- tigate the impact of each step of few-shot learn- ing and reinforcement learning on DocGen and DocGen-RI, conduct ablation study, of which the results are shown Table to evaluate the im- pact of different components and variations on the performance of DocGen and DocGen-Re models in terms of n @ 10 on NQ-test. The first sec-

---

Table 4: Analysis of the overlap between synthetic and realistic data on the NQ dataset. Overlap refers to the average number of words overlapping between query and document. Q, D, and W refer the queries and documents and words. The term "Expanded" refers to the fact that we expand the human queries in the DocGen pipeline. As we remove highlighting marks from the final training data constructed by DocGen, expanding the query the only step in the DocGen pipeline that has an effect on the length of the query.
Augmentor
Query
Documents
Overlap
Avg # W in Q
Avg # W in D
InPars (Bonifacio et al., 2022)
Synthetic Human
Human
6.54
6.78
GenRead (Yu et al., 2023)
Synthetie
4.39
57.02
DocGen
Expanded
Synthetie
2.22
7.72
69.89
DocGen-RL
Expanded
Synthetie
2.58
7.72
71.24
Human
Human
3.20
9.15
79.02
Table 5: DocGen and DocGen-RL VS. human super- vised using nDCG@ evaluation.
query and generated document, as the LLMs tend to select the important words from the document as the query words during query generation, which is dissimilar from human queries in that there can be cases that the relevant document has less mutual words with the query. That is why semantical search is important. We see that InPars, which generates queries, has the highest overlap between queries and documents, and GenRead, which also generates documents similarly to DocGen, has less lexical overlap compared to InPars.
Dataset
TREC DL'20
100k generated doc before CF
DocGen
.580
DocgGen-RL
.618
1M generated doc before CF
DocGen
.652
DoegGen-RL.
.688
Supervised (Human)
Can generated documents replace human- annotated documents? We investigate this ques- tion by employing DocGen and DocGen-RL to generate synthetic documents for the one mil- lion queries of the training set queries of MS MARCO. We then compare these results with training MonoT5 on the MS MARCO training set vs. synthetic data generated by DocGen and DocGen-RL, Table 5 demonstrates the promising potential of utilizing DocGen and DocGen-RL as alternatives to human-annotated datasets as we achieve higher effectiveness that is slightly less than human-annotated data by increasing the size of our synthetic dataset.
MonoT5-MS MARCO .714
is about 5.5 times bigger, with three billion pa- rameters (BLOOM-3B) instead of the version with BLOOM 560 million parameters. For (ii), we fine- tune MonoT5 with 770 million parameters, which is about 3.5 times larger, instead of the version with 200 million parameters. Table 3 shows that we achieve significant improvement over the re- sults by increasing the scale of either BLOOM or MonoTS parameters.
6 Discussion
Gap between synthetic and realistic data. We in- vestigate to what extent different data augmentation methods are close to the human data in terms of the average length of queries and documents, as well the average number of word overlaps within the query and document. Table 4 shows that the gener- ated documents by DocGen and DocGen-RL RLL, and the modified expanded queries, closer to the hu- man data compared to the other data augmentation methods. Moreover, we observe that generating queries from documents by InPars (Bonifacio et al., 2022) leads to a high lexical overlap between the
Computational cost analysis. Table repre- sents an analysis of the inference time between DocGen and InPars. We report two variants of In- Pars: (i) the original InPars (Bonifacio et al., 2022) implementation which adopts GPT-J-6B, and (ii) more efficient variant which adopts BLOOM- 560M. We observe that both DoeGen and DocGen- RL are over twice efficient InPars-GPT in inference (107 seconds 237 seconds). How- ever, they are less efficient than InPars-BLOOO (72 seconds vs. 107 seconds). We believe that the additional computational costs can be justified

---

Table 6: Efficiency analysis of different data augmentation methods for information retrieval.
Data Augmentor
LLM
Latency
nDCG@ on NQ
InPars (Bonifacio al., 2022)
GPT-J-6B
237 seconds
0.335
DocGen
BL00MM-56M
107 seconds
0.467
DocGen-RL
BL00M-560M
107 seconds
0.517
InPars (replicated)
BL00M-560M
72 seconds
0.337
Table 7: Impact of DocGen and DocGen-RL on building training dataset for further fine-tuning MonoT3- MSMARCO vs. fine-tuning Mono'5 from scratch.
Data Augmentor
Retriever
nDCG@ 10
DocGen
MonoT5
0.467
DocGen
MonoT5-MSMARCO
0.492
DocGen-RI.
MonoT5
0.517
DocGen-RL.
0.539
by the large performance gain that both models achieve (nDCG@ 10 on NQ: 0.517 (DocGen-RL) vs. 0.337 DmPars-BLOOME
fore generating documents. Extensive experiments demonstrate that both DocGen and DocGen-RL significantly outperform existing approaches, in- cluding InPars. This work showcases the potential of leveraging LLMs for document generation and highlights the significance of document augmen- tation in information retrieval. We emphasize the importance of improving the quality of training datasets. The findings of this study contribute advancing the state-of-the-art in IR and provide valuable insights for future research in this domain.
Impact of DocGen on trained retriever. To analyze the impact of DocGen the effective- ness of the already trained retriever model, fine-tuned starting from the checkpoint of MonoT5- MSMARCO with 220M parameters pre-trained on MS MARCO, and analyze to what extent fur- ther fine-tuning it on the augmented data will have an impact on its effectiveness. Table shows that DocGen can make improvements in this scenario well while this impact is not quite large. This observation is in line with InPars V2 (Jeronymo al., 2023)) which shows that further fine-tuning from MoonoTS-MSMANE improvement. This is because further fine-tuning a well-trained MonoTS could diverge the retriever from an optimal weight.
Limitations
We showed this paper that generative LLMs can create data to train effective retrieval models. We should stress that other aspects of evaluation have not been investigated in this paper, specifically the effect of biased information in the generated doc- uments biases in the document ranking. It is well-known that LLMs have biases reflected in their output. this work we do study the effect of such biases on the performance of the reranker, po- tentially biasing the retrieved documents. Another problem of generative LLMs is that the factuality of the output cannot be guaranteed. Even though fac- tually incorrect information in the generated data (as result of LLM's hallucination) is not likely to be harmful in the IR context, because only in- formation that is truly contained in the document collection can be retrieved by a retrieval model. We do not systematically study and quantify the effect hallucinated data on the performance
does lead to large
7 Conclusions
This paper presents new perspective for construct- ing training datascts in information retrieval with- out human labels. It proposes two novel query-to- document generators, called DocGen and DocGen- RL. The motivation behind this approach is to overcome the limitations of existing methods that generate queries from documents. Additionally, we automatically enhance low-quality queries be-
auttp://hhgiiggiccc..cssmrimmmma msmarco

---

the ranker.
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling eetrucctionfincture language models. arXiv preprint arXiv:2210.1 11416.
Ethics Statement
Reiterating the importance of understanding and quantifying LLMs bias in generating data, we be- lieve that this can lead to unforeseen ethical con- sequences. Therefore, we need to carefully study potential biases that exist in the data and formalize their impact on the generated data and the trained model. While in this work we demonstrate the po- tential of LLMs in generating synthetic data to im- prove IR systems, we believe that such approaches should be applied with great care in real-world search systems and the system designers should take into account the existence of such biases
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. trec 2020 deep learning track. arXiv preprint arXiv:210..07662.
Overview of the
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos. and Ellen M Voorhees. 2020. Overview of the tree 2019 deep learning track. arXiv preprint 0rXiv:2003.078/0
Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2023. Promptagator: Few- shot dense retrieval from examples. ICLR.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00 000.442.........
References
Amin Abolghasemi. Suzan Verberne, Arian Askari, and Leif Azzopardi. 2023. Retrievability bias estimation using synthetically generated queries. In Proceed- ings the 32nd ACM International Conference on In- formation and Knowledge Management, CIKM '23, page 37 5712-3710 16, New York, NY, USA. Association for Computing Machinery.
EleutherAT. 2023.
Gpt-j-6b: a transformer model trained using ben wang's 's mesh transformer jax.
Matteo Gabburo, Siddhant Garg, Rik Koncel- Kedziorski, and Alessandro Moschitti. 2023. Learn- ing answer generation using supervision from auto- matic question answering evaluators. arXiv preprint arXiv:2305. 15344.
Mayank Anand, Jiarui Zhang, Shane Ding, Ji Xin, and Jimmy Lin. 2021. Serverless bm25 search and bert reranking. In DESIRES, pages 3-9.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without rele- vance labels. arXiv preprint arXiv:2212. 2.2.09222. /0496.
Arian Askari, Mohammad Aliannejadi, Evangelos Kanoulas, and Suzan Verberne. 2023a. Generating synthetic documents for cross-encoder re-rankers: A comparative study of chatgpt and human experts. arXiv preprint arxiv:2305.02320.
Natasha Jaques, Judy Hanwen Shen, Asma Ghande- harioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. 2020. Human-centric dialog training via offline reinforce- ment learning. arXiv preprini arXiv:2010.0588
Arian Askari, Mohammad Aliannejadi, Evangelos Kanoulas, and Suzan Verberne. 2023b. A test col- lection of synthetic documents for training rankers: Chatgpt vs. human experts. In Proceedings of the 32nd ACM International Conference on Informa- tion and Knowledge Management, CIKM '23, page 5311-5315, New York, NY, USA. Association for Computing Machinery.
Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio. Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprini arXiv:2301.08833
Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Unsupervised dataset generation for information retrieval. In Pro- ceedings of the 45th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, page 2387-2392, New York, NY, USA. Association for Computing Machinery.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39- 48.
Samuel Kiegeland and Julia Kreutzer. 2021. Revisiting the weaknesses of reinforcement learning for neu- ral machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1673-1681.
Leonid Boytsov, Preksha Patel, Vivek Sourabh, Riddhi Nisar, Sayani Kundu, Ramya Ramanathan, and Fric Nyberg. 2023. Inpars-light: Cost-effective unsuper- vised training of efficient rankers. arXiv preprint arXiv:2301.0299

---

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Epstein, Ilaa Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: benchmark for question answering research. Trunsactions of the Association for Computational Linguistics, 7:452- 466.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with umified text-to-text trans- former. The Journal of Machine Learning Research, ((1):5445-555)
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy opti- mization. In Proceedings of the Eleventh Inter- national Conference on Learning Representations (ICLR 2023).
Hugo Laurençon, Lucile Saulnier, Thomas Wang. Christopher Akiki, Albert Villamova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Ed- uardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1. composite mul- tilingual dataset. Advances in Neural Information Processing Systems,
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Min- ervini, Heinrich Küttler, Alcksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. Paq: 65 mil- lion probably-asked questions and what you do with them. Transactions of the Association for Com- putational Linguistics, 9: 9:109811155.
Stephen E Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SI- GIR '94, pages 232-241. Springer.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.051000
Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. 2023. Guiding large language models via directional stimulus prompting. arXiv preprint arXiv:2302.11 11520.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021. Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures on Human Language Technologies, 14(55
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint 1xXiv:177.06344
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Expansion via prediction of impor- tance with contextualization. In Proceedings of the 43rd International ACM SIGIR conference re- search and development Information Retrieval, pages 1573-1576.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. In CoCo@ NIPs.
James Thorne, Andreas Vlachos, Christos Christodoullpooulos, and Arpit Mittal. 2018. Fever: large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techmologies, Volume 1 (Long Papers), pages 809-819.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Pas- sage re-ranking with bert. arXiv preprint arXiv:9901.04085
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre- model. In Findings
trained sequence-to-sequeen of the Asxociation for Computational Linguistics: EMNLP 2020, pages 708-718.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2302v 13971.
Richard Yuanzhe Pang and He. 2021. Text gener- ation by learning from demonstrations. In Interna- tional Conference Learning Representations.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. deep reinforced model for abstractive sum- marization. arXiv preprint arXiv: 1rXiv:1705.04300 705.04304.
Eric Wang. 2023. Alpaca-lora: Instruction-follooiing Ilama model.

---

Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doe: Query expansion with large language models. arXiv preprint ar xxiv:2303.076611
Example 1:
Query: Is little caffeine ok during pregnancy? Query Expanded: What the recommended amount of caffeine intake during pregnancy. there any potential risks associated with consuming small amounts of caffeine while pregnant?
Orion Weller, Mare Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. according to..." prompting language mod- els improves quoting from pre-training data. arXiv preprint arXiv:2305. 73252.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's s neural machine translation system: Bridging the gap between human and machine trans- lation, arXiv preprint (XXi::609.08/444
Example 2:
Query: What fruit is native to Australia? Query Expanded: Which fruit exclusive to Australia and provide some additional details about it?
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380.
Example 3:
Query: How large the canadian military? Query Expanded: What the size of the canadian military ahd what is the number of active personnel and reserve members?
Example 4:
Query: (query_text Query Expanded:
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh Inter- national Conference on Learning Representations.
Figure 3: Query expansion with in-contextrreasoning prompt.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 18223.
Example
Query: What is the recommended amount of caffeine intake during pregnancy. and are there any potential risks associated with consuming small amounts of caffeine while pregnant? Query Highlighted: What recommended amount [caffeine] intake during [pregnancy]. and are there any potential risks associated with consuming small amounts of [caffeine] while [pregnant]?
Shengyao Zhuang, Hang Li, and Guido Zuccon. 2021. Deep query likelihood model for information re- trieval. In European Conference on Information Re- trieval, pages 463-470, Springer.
Shengyao Zhuang and Guido Zuccon. 2021. Tilde: Term independent likelihood model for passage re- ranking. In Proceedings of the 44th International ACM SIGIR Conference Research and Develop- ment Information Retrieval, pages 1483-1492.
Example 2:
Query: Which fruit is exclusive to Australia and provide some additional details about it?
Query Highlighted: Which [fruit] exclusive to [Australia]] and provide some additional details about it?
A Appendix
Example 3:
Query: What is the size canadian military ahd what is the number of active personnel and reserve members? Query Highlighted: What the size of the (canadian militayy] ahd what is the number of active personnel and reserve members?
A.1 Prompts
Figure 3, 4, and show the prompts for expanding the orignal query, highlighint the expanded query, and document generation given expanded and high- lighted query.
Example
Query: [query_text)
Query Highlighted:
A.2 Safeguard Data Leakage
In order to being sure that there is data leakage between the evaluation datasets in our experiments and the training set the BLOOM, we assess the BLOOM training data documentation (Laurençon
Figure 4: Query highlighting with in-context-reasoning prompt.

---

Table 8: Impact of highlighting character on DocGen.
Example1:
Query: What is the recommended amount of [caffaine] intake during [pregnancy], are there any patential risks associated with consuming small amounts of [ceffeine] while [oregnant]? Relevant Document: We don't know a lot about the effects of caffeine during pregnancy you and your baby. So limit the amount you get each day. you are pregnant, limit caffeine to 200 milligrama each day. This about the amount in 1% 8-ounce cups of coffee or one 12-aunoe cup of coffee.
Evaluation on NQ-test in terms of nDCG@ 10.
Highlighting character
(e.g,, what are <calling cards>?)
.301
'(" (e.g., what are (calling cards ??)
.319
ees (e.g., what are *calling cards*?)
.360
Query: Which [fruit] is exclusive to (Australia) and provide some additional details about it?
'0' (e.g., what are (calling cards)?)
.391
Dacument: Passiflora herbertiana,
passion fruit
native to Australia. Fruits are green-skinned, white fleshed, with unknown edible rating. Some
'LJ' (i.e., what are [calling cards]?)
.467
list the sweet and lasty, while others list the fruits as being bilter and inedible assiflora herbertiene, rare passion fruit native to Fruits are green-skinned, white fleshed, with unknown edible reling. Some sources list the fruit as edible, sweet fasty, while others the fruits being bitter and inedible.
edible,
DocGen w/o highlighting
.363
Example
Query: what goes bottom of shephard's
Relevant document: (begin(itemize)
Viem 50g butter
Query; What is the size of the [canedian miltary] ahd what is the number active personnel and reserve members?
Vaegin (temize]iitem melted
Relevant Document: The Canadian Amedd Forces. The first large Scale Canadian peacekaeping mission started in Egypt on November 1956.
lend(uiisjt
'niiter flour
epprodmately Force and 25,000 reservist members in the Canadian miliary. In Canadia, August 9 designated as National Peacekeepers' Day
Regular
Yullne((15))
Winebreak
Vipsum(11
Vifill
Waspce((2o)
Quary: (quary_text Relevant Document:
Mextbft
Mextit }
btatbbllltt
Ipar
Figure 5: Document generation with in-context- reasoning prompt.
Incindent
liggkiip
Inewline
et al., 2022) if any of the BEIR collection datasets or MSMARCO dataset are involved in the training dataset of the BLOOM. We made sure that there is no mutual dataset within BLOOM training data, BigScience Corpus (Laurençon et al., 2022), and our evaluation datasets.
*Mend (enumerate)""n "lend" WIN VI
&8.8mml 1. W.l. & & & i &in Wi 8888 18 188818 &ilin! 888.811
8&& 118 8l1 W8&&&
Figure 6: Example of a noisy generated document by LLaMA.
A.3 Optimal character for highlighting.
A.4 Example of noisy generated document by LLaMA
To determine the most effective highlighting char- acter, we compare various characters, including less than and greater than symbols ('<>),, aster- isks ('***), square brackets("U'), parentheses ('O'), and curly braces ('()).. We evaluate these high- lighting characters based on their impact on the quality of training data by assessing the perfor- mance of MonoT5 fine-tuned on augmented data for NQ queries using each highlighting character. Table 8 demonstrates that using parentheses and square brackets as highlighting characters leads to improvement while using other highlighting char- acters can even lead drop in effectiveness. We think this is because probably those symbols do not trigger any part of the memory of the LLM and act as noise. The square bracket that is used by DoeGen is shown to be an optimal highlighting character.
Figure 6 provides an example of generated docu- ment by LLaMA that has a large amount of noisy text. This could show an example why LLaMA is not suitable for our task. We observed that for more than half of the queries of Natural Question (NQ) dataset, LLaMA produces noisy text.


Reviews:
[{"review_id": "fsOpgKVr3u", "reviewer": "Reviewer_LWa7", "summary": "", "questions": "A. In section 4 (Experimental setup) ln 431-432 it is mentioned “DocGen has not used the supervision from MS MARCO for either synthetic data passage reranker training”. I wonder if there’s anything preventing utilizing the available supervision for filtering or warming up?\nB. RL training can be unstable. I wonder if the authors investigated the stability of  the trained policy network and its robustness, as well as the variance of results from DocGen-RL?\nC. How does the computational cost of DocGen and DocGen-RL compare to the one of the previous methodology? Specifically I wonder how much additional computational cost are brought by adopting RL policy.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Question A*\n> In section 4 (Experimental setup) ln 431-432 it is mentioned “DocGen has not used the supervision from MS MARCO for either synthetic data passage reranker training”. I wonder if there’s anything preventing utilizing the available supervision for filtering or warming up?\n\nWe believe effective external supervision is not consistently available across different domains or languages. This is why we focus on proposing an approach that is not dependent on external supervision and only relies on a large language model to generate synthetic documents. For example, it is more practical to apply our approach to languages other than English where large language models for those languages can be trained, but other supervision signals such as document ranking can be scarce.\n\n---\n*Question B*\n> RL training can be unstable. I wonder if the authors investigated the stability of the trained policy network and its robustness, as well as the variance of results from DocGen-RL?\n\nThank you for your comment. We agree that testing the robustness of the model is important. We implemented the RL part of DocGen-RL based on RL4LMs, an RL library to fine-tune language models based on human preferences. RL4LMs uses an adaptive KL controller to reduce the instability of RL training, hence we expect the model to be stable. To verify this, we tested the stability and variance of the DocGen-RL's performance by varying the random seed on the Natural Question (NQ) dataset. As shown in the following table, DocGen-RL's effectiveness has low sensitivity to the randomness brought by random seeds. We plan to extend this experiment to all the datasets and add the results to the camera-ready version of the paper.\n\n| Data Augmentor | Retriever    | nDCG@10 | nDCG@10 | nDCG@10 | nDCG@10 | nDCG@10 | nDCG@10| nDCG@10  | \n|-----------|-------------------|--------|--------|--------|--------|--------|--------|--------|\n|  |  | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Seed 5 | Avg. | Std. | \n| DocGen-RL  (Ours) |  MonoT5 | 0.5175 | 0.5171 | 0.5189 | 0.5250 | 0.5177 | 0.5192 | 0.0032| \n\n---\n\n*Question C*\n> How does the computational cost of DocGen and DocGen-RL compare to the one of the previous methodology? Specifically, I wonder how much additional computational cost is brought by adopting RL policy.\n\nTo address this question, we compare the latency of InPars (Bonifacio et al., 2022), DocGen, and DocGen-RL in the following table. Please note that we report two variants of InPars: (i) the original InPars (Bonifacio et al., 2022) implementation which adopts GPT-J-6B, and (ii) a more efficient variant which adopts BLOOM-560M.\n\n| Data Augmentor                  | LLM | Latency     |  nDCG@10 on NQ\n|---------------------------------|----------------------|-------------|-------------|\n| InPars (Bonifacio et al., 2022) | GPT-J-6B             | 237 seconds | 0.335 |\n| DocGen                          | BLOOM-560M           | 107 seconds |  0.467 |\n| DocGen-RL                       | BLOOM-560M           | 107 seconds | 0.517 |\n| InPars (replicated by us)       | BLOOM-560M           | 72 seconds  | 0.337 |\n\n    \nFrom the results, we see that both DocGen and DocGen-RL are over twice as efficient as InPars-GPT in inference (107 seconds vs. 237 seconds). However, they are less efficient than InPars-BLOOM (72 seconds vs. 107 seconds). We believe that the additional computational costs can be justified by the large performance gain that both models achieve (nDCG@10 on NQ: 0.517 (DocGen-RL) vs. 0.337 (InPars-BLOOM)).\n\nRegarding *\"I wonder how much additional computational cost are brought by adopting RL policy.\"*: Compared to the DocGen, the additional RL training for DocGen-RL can be done offline and requires less than 18 hours of training with a single Nvidia P6000 GPU. This can be considered relatively low, especially when compared with the training cost of large language models. Moreover, it is noteworthy to mention that the inference times of DocGen and DocGen-RL are equal, since they both have the same architecture, while DocGen-RL is significantly more effective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of combining query highlighting and RL is novel. The effectiveness of the RL-guided document generation is showcased in the ablation section, and in the main results section the DocGen-RL also shows stat-sig advantage DocGen standalone. The results in the paper indicate employing RL on LLMs in IR data generation can be a promising direction.\n- The paper is clear-written with comprehensive analysis and experimental results provided for the readers to better understand the contributions of each component in the devised methodology.", "reasons_to_reject": "There is no obvious concern from the reviewer.", "questions_to_authors": "A. In section 4 (Experimental setup) ln 431-432 it is mentioned “DocGen has not used the supervision from MS MARCO for either synthetic data passage reranker training”. I wonder if there’s anything preventing utilizing the available supervision for filtering or warming up?\nB. RL training can be unstable. I wonder if the authors investigated the stability of  the trained policy network and its robustness, as well as the variance of results from DocGen-RL?\nC. How does the computational cost of DocGen and DocGen-RL compare to the one of the previous methodology? Specifically I wonder how much additional computational cost are brought by adopting RL policy.", "paper_topic_and_main_contributions": "The paper presents two query-to-document generators, DocGen and DocGen-RL, with the motivation to improve existing methods generating queries from documents. Contributions are in both (1) raw queries enrichment including query expansion and query highlighting, and (2) leveraging few shots Reinforcement Learning to enhance query and generated document relevance (which is claimed to be the first work employing RL on LLMs in the field of IR data generation). To present the effectiveness of the devised pipelines and showcase the contribution of each improvement added, the authors provide analysis and experiments results with extensive benchmarking data sets, the analysis includes performance lift of the proposed methods over SOTA baselines, ablation studies on DocGen and RL-training analysis on DocGen-RL and scaling impact. The authors also provide quantitative comparison between the synthetic and realistic data.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Swn87vDoZJ", "reviewer": "Reviewer_yxMC", "summary": "", "questions": "A. Could the authors elaborate on whether they use expended queries during the inference process for BM25 and MonoT5 models?  \n \nB. What is the motivation to apply LLMs, instead of existing query expansion methods, for query expansion?  \n\nC. It is overclaimed to say that the proposed method leverages the maximum capacity of LLMs in generating synthetic data. (Line 27)\n\nD. It may be better to have the ablation study about the number of examples used in the prompt.  \n \n \n6. Claveau, V. Neural text generation for query expansion in information retrieval. IEEEWICACM Int. Conf. Web Intell. 202–209 (2021) doi:10.1145/3486622.3493957.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Question A*\n> Could the authors elaborate on whether they use expended queries during the inference process for BM25 and MonoT5 models?  \n\nNo, we did not use expanded queries as the input for any of the retrievers during inference. In our paper, query expansion is only used for data augmentation, in which we expand raw queries so as to better trigger the knowledge stored in an LLM and, as a result, generate higher-quality documents.\n\n---\n\n*Question B*\n\n> What is the motivation to apply LLMs, instead of existing query expansion methods, for query expansion?\n\nThere are two reasons why we consider LLM-based query expansion: (i) we consider LLM-based query expansion since LLMs are able to conduct query expansion without the need for large amount of training data, pre-defined rules, or fine-tuning; DocGen only uses three demonstration examples to bootstrap the LLM; and (ii) there is a fundamental difference in the query expansion performed by DocGen compared to existing methods. Our objective is to rewrite the query and make it closer to natural language (rather than keyword-based), so the LLM can respond to it effectively. This objective is different than the traditional query expansion where the goal is expanding the query with additional words to enhance the likelihood of retrieving relevant documents given the expanded query. Our preliminary experiments revealed that LLMs respond more effectively to natural language queries.\n\n---\n\n*Question C*\n\n> It is overclaimed to say that the proposed method leverages the maximum capacity of LLMs in generating synthetic data. (Line 27)\n\nThank you for pointing this out. We will change this claim to \"... our new perspective aims to utilize the capacity of LLMs in generating synthetic data **more effectively**\" in the camera-ready version.\n\n---\n\n*Question D*\n\n> It may be better to have the ablation study about the number of examples used in the prompt. Claveau, V. Neural text generation for query expansion in information retrieval. IEEEWICACM Int. Conf. Web Intell. 202–209 (2021) doi:10.1145/3486622.3493957.\n\nThanks for your suggestion. Our choice of 3 examples was motivated by the related work (Bonifacio et al., 2022) and our preliminary experiments using different numbers of examples. We will add an ablation study to the camera-ready version of the paper on the impact of different numbers of examples on the DocGen and DocGen-RL compared to the InPars.\n\n--- \n\n*Reason to reject A*\n\n> This work assumes they have abundant user queries, which can be used to generate query-document pairs for data augmentation to handle the data sparsity issue. However, in a data sparsity situation, it may be more natural to have a certain number of documents without the corresponding user queries instead of the opposite. It is unclear in which situation we will have abundant, like 10k, user queries before we have abundant query-document pairs. \n\nAn example of a situation in which our approach would be useful is in commercial search engines where there is no sparsity in terms of the number of submitted queries. We summarize the advantages of our approach in this scenario in the following:\n\n1. **For complex queries.** For certain queries, the relevant information may be spread across multiple documents, and by using LLMs for synthetic document generation, the reranker could be optimized to provide the most optimal response that consists of information from several documents.\n\n2. **For new or trending queries.** The search engine service provider can optimize its retriever based on the recently received and repeated (trending) queries by using DocGen to generate synthetic documents as training data given those queries, for which there are no available relevance judgments.\n\n3. **Complementary perspective.** Our proposed methods can also be viewed as a complementary approach to query generation methodologies. In a real-world pipeline for data augmentation, both approaches may be used.\n\n\n--- \n\n*Reason to reject B*\n\n>  It is not solid enough to skip the baseline model Promptagator, even if it performs first-stage retrieval with customized prompts per dataset. For example, the authors can compare the end-to-end performance of Promptagator and their method, both with customized prompts per dataset.\n\nWe replicate Promptagator using customized prompts for training rerankers on the Natural Question dataset and report its effectiveness compared to InPars, and our proposed DocGen and DocGen-RL in the following table.\n\n| Data augmentor | nDCG@10  | Retriever | LLM  |\n|----------------|-------|-----------|------------|\n| Promptagator   | 0.341 | MonoT5    | Bloom-560M |\n| InPars         | 0.335 | MonoT5    | Bloom-560M |\n| DocGen         | 0.467 | MonoT5    | Bloom-560M |\n| DocGen-RL      | 0.517 | MonoT5    | Bloom-560M |\n\n\nThe results show that DocGen and DocGen-RL (both w/o customized prompt) still lead to more effective rerankers than Promptagator (w/ customized prompt). In this work, we focus on reranking and leave the analysis of the impact of synthetic document generation on training first-stage retrievers for future work.\n\n\n--- \n\n*Reason to reject C*\n\n> (Minor weakness): Compared with previous LLMs-based studies, the proposed DocGen-RL requires training based on reinforcement learning, costing lots of computational \n\nCompared to DocGen, the additional RL training for DocGen-RL can be done offline and does not need a large amount of computational resources. It only takes less than 18 hours with a single Nvidia P6000 GPU to do the RL training. This can be considered relatively low, especially when compared with the training cost of large language models. Moreover, it is noteworthy that the inference times of DocGen and DocGen-RL are equal, since they both have the same architecture, while DocGen-RL is significantly more effective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is a reasonable paradigm to incorporate LLMs to generate contextual documents given a query, which aligns with the research on autoregressive information retrieval like [1,2,3,4] and corresponding applications like knowledge-intensive tasks[5], although this work applies this idea to data augmentation phase.  \n\nPrevious studies focus on feeding a document to LLMs to generate a query. However, it is not trivial to generate a query relevant to multiple documents due to the token limitation of LLMs. This proposed paradigm can solve this issue.  \n\nThe proposed method outperforms other unsupervised baseline methods like InPars, GenRead, and Q2D on the MS MARCO, NQ, HotpotQA, Fever, and TREC-DL'20 datasets with the help of query expansion, keyword highlighting, and reinforcement learning. \n\nExtensive analysis has been conducted about the main experiments, ablation studies, and the gap between synthetic and realistic data.  \n\n1. Cao, N. D., Izacard, G., Riedel, S. & Petroni, F. Autoregressive Entity Retrieval. Arxiv (2020) doi:10.48550/arxiv.2010.00904. \n\n2. Tay, Y. et al. Transformer Memory as a Differentiable Search Index. Arxiv (2022) doi:10.48550/arxiv.2202.06991.  \n\n3. Lai, T. M., Ji, H. & Zhai, C. Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linking. Arxiv (2022) doi:10.48550/arxiv.2202.13404. \n\n4. Bevilacqua, M., Ottaviano, G., Lewis, P., Yih, S., Riedel, S., & Petroni, F. (2022). Autoregressive search engines: Generating substrings as document identifiers. Advances in Neural Information Processing Systems, 35, 31668-31683. \n\n5. Yu, W., Iter, D., Wang, S., Xu, Y., Ju, M., Sanyal, S., ... & Jiang, M. (2022). Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063.", "reasons_to_reject": "A. This work assumes they have abundant user queries, which can be used to generate query-document pairs for data augmentation to handle the data sparsity issue. However, in a data sparsity situation, it may be more natural to have a certain number of documents without the corresponding user queries instead of the opposite. It is unclear in which situation we will have abundant, like 10k, user queries before we have abundant query-document pairs.  \n\nB. It is not solid enough to skip the baseline model Promptagator, even if it performs first-stage retrieval with customized prompts per dataset. For example, the authors can compare the end-to-end performance of Promptagator and their method, both with customized prompts per dataset.  \n\nC. (Minor weakness): Compared with previous LLMs-based studies, the proposed DocGen-RL requires training based on reinforcement learning, costing lots of computational resources.", "questions_to_authors": "A. Could the authors elaborate on whether they use expended queries during the inference process for BM25 and MonoT5 models?  \n \nB. What is the motivation to apply LLMs, instead of existing query expansion methods, for query expansion?  \n\nC. It is overclaimed to say that the proposed method leverages the maximum capacity of LLMs in generating synthetic data. (Line 27)\n\nD. It may be better to have the ablation study about the number of examples used in the prompt.  \n \n \n6. Claveau, V. Neural text generation for query expansion in information retrieval. IEEEWICACM Int. Conf. Web Intell. 202–209 (2021) doi:10.1145/3486622.3493957.", "paper_topic_and_main_contributions": "This work explores a new paradigm to incorporate large language models (LLMs) for data augmentation to improve the passage reranking in information retrieval (IR) tasks by generating synthetic relevant passages given a query, while previous studies focus on generating synthetic queries given a document. Along with their exploration, they propose the approach DocGen, which applies query expansion and keyword highlighting to trigger the LLMs to generate better relevant documents, and DocGen-RL, a reinforcement learning (RL) based method to improve further the keyword highlighting process by updating LLM’s parameters based on a relevance reward from a trained T5-based retriever. Extensive experiments and analyses are conducted to address the following questions: (1) whether their method outperforms unsupervised baselines. (2) which components contribute to the improved performance? (3) whether scaling LLMs can improve the performance. (4) What is the gap between synthetic and realistic data?  \n\nThe contribution lies in the proposed new data augmentation paradigm, a.k.a. generating document given a user query, the proposed three-step pipeline, named DocGen, and RL-based method, named DocGen-RL, which both outperform existing unsupervised baselines and achieves a promising performance based on this data augmentation paradigm. Extensive experiments and analysis shed light on future research.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typo in Line 246: the quality of the generated documents 2. The number 2 here is meaningless.  \n\nIt may be better to mention that CF refers to Consistency Filtering in Table 5. \n\nIt may not be necessary to discuss the 3.1.2 Consistency Filtering section in detail since this method is almost the same as in [7]. \n\nIt may be better to remove Figure 1 or add detailed components to it. \n \n \n7. PROMPTAGATOR : FEW-SHOT DENSE RETRIEVAL FROM 8 EXAMPLES."}}, {"review_id": "vCRHIxkq7T", "reviewer": "Reviewer_y3cg", "summary": "", "questions": "* how to determine important words in square brackets in the prompt examples? Did you find any patterns of LLM output what do they prefer to highlight, before and after RL training? \n\n* The authors follow Promptagator to conduct consistency filtering. Any reason why only the top-1 is selected? Is there any results on Top-K analysis? The w/ and w/o CF parts are missing in the paper.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Question A*\n\n> how to determine important words in square brackets in the prompt examples? Did you find any patterns of LLM output what do they prefer to highlight, before and after RL training?\n\nRegarding determining important words in the examples, we empirically determined that highlighting words around the main subject of the query is beneficial and also feels natural. Therefore, we chose to highlight tokens with significant roles in the prompt examples.\n\nWe analyze the highlighting pattern before and after RL training and report the number of highlighted tokens based on their linguistic roles using the NLTK library.\n\n| Highlighted word POS | Before RL                           | After RL      |\n|-----------------------|-------------------------------------|---------------|\n| Pronouns              | 44,993 words                        | 30,040 words  |\n| Nouns                 | 120,665 words                       | 151,236 words |\n| Entities              | 87,942 words                        | 120,160 words |\n| Verbs                 | 38,566 words                        | 39,851 words  |\n| Non-highlighted words | 981,811 words                       | 930,336 words |\n| Total number of words:           1,281,795 words\n\n\nAs shown in the above table, we find that the number of highlighted pronouns is reduced after RL training, while the number of highlighted nouns and entities increases. However, a considerable number of highlighted pronouns still remains highlighted after RL training. Furthermore, we observe that the number of highlighted verbs remains nearly the same.\n\n---\n\n*Question B*\n\n> The authors follow Promptagator to conduct consistency filtering. Any reason why only the top-1 is selected? Is there any results on Top-K analysis? The w/ and w/o CF parts are missing in the paper.\n\nThank you for the constructive suggestion. In the following table, we report analysis on considering top-k for consistency filtering, ranging from 1 to 5, where we report the performance of DocGen-RL w/ and w/o Consistency Filtering (CF) on the Natural Question dataset.\nThe following table demonstrates that including synthetic documents ranked at lower positions within the training data leads to reduced effectiveness. This observation aligns with prior research, which advocated for selecting the top-ranked document (k=1) to ensure training data quality.\n\n| Top-k | nDCG@10   |\n|-------|--------|\n| 1     | 0.5175 |\n| 2     | 0.5009 |\n| 3     | 0.4887 |\n| 4     | 0.4401 |\n| 5     | 0.4026 |\n\n\nRegarding **\"The w/ and w/o CF parts are missing in the paper.\"**, we will add this experiment to our result table for the camera-ready version of the paper. As an example, we report the below table that showcases the effectiveness of DocGen, both with and without CF on the Natural Question dataset in terms of nDCG@10. As expected, both models experience a drop in effectiveness without CF, underscoring the role of consistency filtering in maintaining the quality of synthetically generated data.\n\n| Data Augmentor | Retriever | nDCG@10 |\n|----------------|-----------|---------|\n| DocGen w/o CF  | MonoT5    | 0.375   |\n| DocGen w/ CF   | MonoT5    | 0.467   |\n\n---\n\n*Reason to reject A*\n\n> Is it possible that the performance gap in Table 1 is coming from the instruction following ability of different LLMs? For example, BLOOM-560M has better ability to generate documents as the authors indicated in line 476, however, it is not good and generate queries so the query-generator baselines have more disadvantages.\n\nFirst, let us clarify that we do not claim that BLOOM-560M is better at generating documents than generating queries; in line 476 we say that BLOOM-560M has better document-generating capabilities among the four LLMs that we investigated (LLaMA-7B, Alpaca-Lora, Flan-T5-xxl, BLOOM-560M). To prevent unfair advantages over query generation approaches, we compared our method to the original implementation of InPars, which uses an effective model for query generation (Table 1 of the paper, row 2, InPars (Bonifacio et al., 2022)). Moreover, our methods achieve better effectiveness with using BLOOM-560M which is 12 times smaller than GPT-J-6B used by InPars in its original results.\n\n> Why ChatGPT is not considered to be used as the base generator?\n\nThere are two reasons:\n\n- **Reliability.** We do not consider the closed-source models such as ChatGPT and GPT 3.5/4.0 to ensure the reliability of our evaluation because it is unclear if there is a data leakage between the pre-training data of these closed-source models and the datasets that we need to use for evaluation.\n\n- **Reproducibility.** We would like to ensure the reproducibility of our paper. ChatGPT and GPT 3.5/4.0 keep updating, and it is hard for researchers to fully replicate their results.\n\n\n---\n\n*Reason to reject B*\n\n> The results are compared fully without using the supervision data (and that’s why InPars-V2 and Promptagator are not in the comparison as the authors justified). However, it will be great to see how does this document augmentation can help on top of the existing supervised retriever, since we already have those training signals and the trained retrievers can be used directly OOTB for other domains. So it makes more sense to show that no matter with augmented queries or augmented documents (or both together), how can it add additional performance boost to the final retriever model.\n\nThank you for your advice. We investigated this based on your feedback.\n\nTo analyze the impact of DocGen on the effectiveness of the final retriever model, we fine-tuned starting from the checkpoint of MonoT5-MSMARCO with 220M parameters, pre-trained on MS MARCO, and analyze to what extent further fine-tuning it on the augmented data will have an impact on its effectiveness. \n\n| Data Augmentor | Retriever      | nDCG@10 |\n|----------------|----------------|---------|\n| DocGen         | MonoT5         | 0.467   |\n| DocGen         | MonoT5-MSMARCO | 0.492   |\n| DocGen-RL      | MonoT5         | 0.517   |\n| DocGen-RL      | MonoT5-MSMARCO | 0.539   |\n\n\nThe above table shows that DocGen can make improvements in this scenario as well while this impact is not quite large. This observation is in line with InPars V2 (Jeronymo et al.,2023) which shows that further fine-tuning from MonoT5-MSMARCO does not lead to a large improvement. This is because further fine-tuning a well-trained MonoT5 could diverge the retriever from an optimal weight.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper shows good results comparing to the common (document —> query) path, which showed a potential data augmentation direction for document retrieval. \n* The paper shows easy-to-follow ablation study to understand most parts of its design such as w/o query expansion,  w/o highlighting, w/o RL.", "reasons_to_reject": "* Is it possible that the performance gap in Table 1 is coming from the instruction following ability of different LLMs? For example, BLOOM-560M has better ability to generate documents as the authors indicated in line 476, however, it is not good and generate queries so the query-generator baselines have more disadvantages. Why ChatGPT is not considered to be used as the base generator? \n\n* The results are compared fully without using the supervision data (and that’s why InPars-V2 and Promptagator are not in the comparison as the authors justified). However, it will be great to see how does this document augmentation can help on top of the existing supervised retriever, since we already have those training signals and the trained retrievers can be used directly OOTB for other domains. So it makes more sense to show that no matter with augmented queries or augmented documents (or both together), how can it add additional performance boost to the final retriever model.", "questions_to_authors": "* how to determine important words in square brackets in the prompt examples? Did you find any patterns of LLM output what do they prefer to highlight, before and after RL training? \n\n* The authors follow Promptagator to conduct consistency filtering. Any reason why only the top-1 is selected? Is there any results on Top-K analysis? The w/ and w/o CF parts are missing in the paper.", "paper_topic_and_main_contributions": "* The paper is about improving neural based document retrieval, focusing on generated synthetic documents.\n* The main contribution of this work is 1) its investigation of how to generate good synthetic documents given queries, and 2) its RL training to determine query highlights that can help LLMs generate better documents.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kH2KiC8gZn", "reviewer": "Reviewer_qmXW", "summary": "", "questions": "- The paper mentions that BLOOM-550M was the best model among the considered ones. Could the other models have lower performance than BLOOM, thanks to the implicit highlighting method learned from it during its training (lines 265-267)?\n- How many times the top-1 retrieved document (mentioned in lines 302-304) is the generated one?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Question A*\n\n> The paper mentions that BLOOM-550M was the best model among the considered ones. Could the other models have lower performance than BLOOM, thanks to the implicit highlighting method learned from it during its training (lines 265-267)?\n\nBased on our preliminary experiment on four LLMs, namely, LLaMA-7B (Tou-vron et al., 2023), Alpaca-Lora (Wang, 2023), Flan-T5-xxl (Chung et al., 2022), and BLOOM-560M (Scao et al., 2022), all of the four LLMs could indeed take advantage of highlighting and the impact of highlighting is not only limited to BLOOM-560M.\n\nIt is noteworthy that in our preliminary experiments, we observed that LLaMA tends to produce noisy text (e.g., LaTeX code) for the queries that we used, and Alpaca-Lora tends to produce very short documents. We will add examples of such cases in the appendix of the paper's camera-ready version. You can also view an example of this case for LLaMA at the end of our rebuttal response.\n\n---\n\n*Question B*\n\n> How many times the top-1 retrieved document (mentioned in lines 302-304) is the generated one?\n\nWe analyzed this for all of the used datasets in this work. Our analysis reveals that, on average across the datasets, the top-1 retrieved document corresponds to the synthetically generated document in 52\\% of instances for DocGen-RL and 44\\% of instances for DocGen.\n\n\n---\n\n*Reason to reject A*\n\n> Although the experimental setting is well defined and sufficient, the paper could benefit from (i) results considering different models and approaches (e.g., LLama for the generation, DeBERTa for the ranker, and DPR for the retrieval), (ii) a qualitative analysis of the generated documents o support Table 4 and the Discussion Section (e.g., by adding some example of good and bad generations).\n\nRegarding **(i)**, we have tried LLaMA for the generation but it tended to produce noisy text (with LaTeX code) for more than half of the questions. We also tried Alpaca-Lora but it tended to produce very short documents. We will add examples of such cases in the appendix of our paper's camera-ready version. We also added an example of noisy text produced by LLaMA at the end of this rebuttal.\n\nFor the sake of limiting the scope of our work, we selected reranking with MonoT5 following InPars as a state-of-the-art ranker. We believe experimenting with DeBERTa and DPR would be possible as our method is a data augmentation methodology that can potentially be beneficial to any retrievers.  We will do this experiment in future work.\n\nRegarding **(ii)**, thanks for suggesting this insightful analysis. We will add this in the camera-ready version as a qualitative analysis in order to support Table 4. We will show how generating queries from documents will lead to more mutual words than generating documents from queries. \n\n---\n\n*Reason to reject B*\n\n> The paper presentation could be improved (with a minor review) since it is not always easy to follow, and sometimes it is verbose (see the suggestions).\n\nThank you for the suggestion. We have addressed all the \"Grammar Style And Presentation Improvements\" suggested by you and other reviewers in the camera-ready version.\n\n\n--- \n\n## Example of a noisy generated document by LLaMA \n\nIn the following, we provide an example of a generated document by LLaMA that has a large amount of noisy text. This could show an example of why LLaMA is not suitable for our task. We observed that for more than half of the queries of NQ, LLaMA produces noisy text. \n\n\n```Example 4:\nQuery: what goes on the bottom of shepherd's pie\nRelevant document: \\begin{itemize}\n  \\item 50g butter\n\t\\begin {itemize}\\item melted\n  \t\\end{ulis}t\n \t\\n\\item flour\n   \t\\uline{15g} \\\n    \t\\linebreak\n     \t\\lipsum[1]\n      \t\\hfill\n       \t\\vspace{2pt}\n        \t\\textbf{\n          \t\\textit{  }\n           \t\\textbullet\n            \t\\par\n             \t\\noindent\n         \t\\bigskip\n \\newline\n   \\hline\n*\n**\\end {enumerate}*\\n  *\\end* \\\\\\\\\\\\ \\\\\n &&&\\n\\\\\\\\ \\ \\\\ \\ \\  \\\\ \\\\  & \\ & & \\\\ &  &\\n \\\\\\\\  &&&& \\& \\&\\&\\\\& &\\\\\\n \\ &&&&\\&\\\n&&& \\\\& &\\\\ \\\\&&&&```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The presented results show a significant performance improvement.\n- The methodology and the experiments are easy to reproduce.", "reasons_to_reject": "- Although the experimental setting is well defined and sufficient, the paper could benefit from (i) results considering different models and approaches (e.g., LLama for the generation, DeBERTa for the ranker, and DPR for the retrieval), (ii) a qualitative analysis of the generated documents o support Table 4 and the Discussion Section (e.g., by adding some example of good and bad generations).\n- The paper presentation could be improved (with a minor review) since it is not always easy to follow, and sometimes it is verbose (see the suggestions).", "questions_to_authors": "- The paper mentions that BLOOM-550M was the best model among the considered ones. Could the other models have lower performance than BLOOM, thanks to the implicit highlighting method learned from it during its training (lines 265-267)?\n- How many times the top-1 retrieved document (mentioned in lines 302-304) is the generated one?", "paper_topic_and_main_contributions": "This paper presents a novel technique to perform data augmentation named DocGen. This technique consists of a pipeline organized in three sequential steps. In the first step, named \"query expansion,\" short keyword-based queries are expanded to longer queries in natural language to maximize the effectiveness of LLM generation. In the second step, named \"query highlighting\", they use a LLM to highlight relevant words in the expanded query (e.g., \"Is a duck bigger than an elephant?\" -> \"Is a [duck] [bigger than] an [elephant]?\") to explicitly manipulate the document generator to pay more attention to those terms. In the third and last step, the expanded and highlighted query is used as a prompt to generate a relevant document. To further refine the quality of the generated documents, they apply a consistency filter based on the output of a ranker (keeping the best document only).\n\nThey measured the performance of their data augmentation approach on several benchmarks showing that DocGen and DocGen-RL (a finetuned version of DocGen) allow training stronger rankers to outperform existing data augmentation approaches.\n\nTo conclude, the main novelty presented in this paper regards the \"query highlighting\" step, which shows that the generation can be easily improved, leading to a better generation pipeline and, consequently, better results on the downstream tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "A very recent paper (ACL23) that could be related to this. They designed data augmentation techniques to improve a generative QA model:\n- Learning Answer Generation using Supervision from Automatic Question Answering Evaluators", "Typos_Grammar_Style_and_Presentation_Improvements": "- Check the typos:\nline 219 -> an a consistency\nline 246 -> documents 2 (I think it should be \"Fig. 2”)\nline 340 -> few-show\nline 358 -> and the\n\n- Highlight the best results in the tables to improve the readability.\n\n- Repeated concepts could lead the paper to be verbose (e.g., GenRead has been described in both the experimental and related work sections).\n"}}]

Author rebuttal:

Meta-review: This paper introduces DocGen, a new paradigm of generating synthetic documents from queries to augment the data for ranking models. An extended version DocGen-RL is further proposed to enhance the relevance between generated synthetic documents and queries using reinforcement learning. All reviewers find this paper strong and exciting with minor concerns on including more experimental analysis.
Decision: Accept-Main