DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation
Authors: Jiashuo Sun
Xianrui Zhong
Sizhe Zhou
Jiawei Han
Venue: NeurIPS 2025 poster ()
Keywords: Retrieval-Augmented Generation
Large Language Models
Dynamic Reranking
Reinforcement Learning
Primary area: applications
TL;DR: We propose DynamicRAG, which optimizes retrieval-augmented generation by using reinforcement learning to adaptively rerank documents based on LLM's responses as feedback, achieving state-of-the-art results across multiple datasets.

Abstract: Retrieval-augmented generation (RAG) systems combine large language models (LLMs) with external knowledge retrieval, making them highly effective for knowledge-intensive tasks. A crucial but often under-explored component of these systems is the reranker, which refines retrieved documents to enhance generation quality and explainability. The challenge of selecting the optimal number of documents (k) remains unsolved: too few may omit critical information, while too many introduce noise and inefficiencies. Although recent studies have explored LLM-based rerankers, they primarily leverage internal model knowledge and overlook the rich supervisory signals that LLMs can provide, such as using response quality as feedback for optimizing reranking decisions. In this paper, we propose DynamicRAG, a novel RAG framework where the reranker dynamically adjusts both the order and number of retrieved documents based on the query. We model the reranker as an agent optimized through reinforcement learning (RL), using rewards derived from LLM output quality. Across seven knowledge-intensive datasets, DynamicRAG demonstrates superior performance, achieving state-of-the-art results.

DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation
Jiashuo Sun, Xianrui Zhong, Sizhe Zhou, Jiawei Han"
University of Illinois Urbana- Chammaagn (jiashuo5, hanj] @illinois.edu
Abstract
-etgeeell-uggmented generation (RAG) systems combine large language mod- els (LLMs) with external knowledge retrieval, making them highly effective for knowledge-intensive tasks. A crucial but often under-explored component of these systems is the reranker. Since irreeevat documents in RAG systems mislead the generator, the reranker plays a vital role in refining retrieved documents to enhance generation quality and explainability. However, it is challenging to deter- mine the appropriate number of documents (k) that the reranker should select: too few may result missing critical information, while too many introduce noise and inefficiencies. Although recent studies have explored LLM-based rerankers, they primarily leverage internal model knowledge and overlook the rich supervisory signals that LLMs can provide, such as using response quality as feedback for optimizing reranking decisions. In this paper, we propose DynamicR a novel RAG framework where the reranker dynamically adjusts both the order and number of retrieved documents based on the query. We model the reranker as an agent optimized through reinforcement learning (RL), using rewards derived from LLM output quality. Across seven knowledge-intensiin datasets, DynamicRAG demon- strates superior performance, achieving state-of-the-ard results among models of same parameter sizes.
Introduction
eetrieval-augmeent generation (RAG) systems have emerged as powerful approach for combining the strengths of large language models (LLMs) with external knowledge retrieval. This integration has proven highly effective for addressing knowledge-intenste tasks and incorporating up-to-date information into LLMs, leading to notable performance improvements [7] 24 [4] RAG systems often suffer from two critical challenges: misleading irrelevant retrieved documents that can distort the generation process, and the 'lost-n-tthemmidlle phenomenon where important information gets buried within long lists of retrieved candidates. crucial, yet often underappreciated, component that addresses these issues is the reranker, which assesses the relevance of retrieved documents. The reranker is critical for improving the quality of generated text and enhancing explainability, thereby serving as indispensable part of the RAG framework 33
In RAG systems, the reranker's 's primary role to refine the Top-N documents retrieved by the retriever, selecting the l most relevant ones enhance the answer quality. However, determining the optimal k remains challenging problem, as highlighted in previous studies [36] 62]. A that is too small risks omitting critical information, leading to degraded generation quality, while larger may introduce irrelevant content, increasing noise and potentially misleading the generator.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Furthermore, incorporating excessively long contexts can reduce both efficiency and effectiveness, further complicating the balance between relevance and performance RAG systems. Striking the right balance requires adaptive strategies that can dynamically adjust & based on query complexity and document diversity, as shown in Figure
Recent work has demonstrated the effective- ness of LLM-based rerankers p 53 26 23 30
Query
TopN
which leverage large language models' capabil- ities to assess document relevance and improve ranking quality. Other works have also used the understanding capabilities
Retriever
LLMs
(a) RAG Without Reranker.
Query
LLMs through sliding window mechanisms to achieve optimal While these stud-
TopV
Topk
Retriever
Reranker
LLMs
re-ranking results (53 5
ies demonstrate the effectiveness of LLM-based rerankers, they typically rely on fixed document selection thresholds (k) and fail to dynamically adapt to varying query complexity and retrieval quality. Existing approaches primarily exploit LLMs' internal knowledge to score documents independently, overlooking key insight: the actual generation quality when using different numbers of documents provides direct feedback about the optimal k. This natural reward sig- nal enables to apply reinforcement learning, where the LLM's generation quality serves as the reward for selecting the right number of doc- uments. This supervisory signal-derived from the downstream generation task itself-offee a more principled approach to document selec- tion than static thresholds or isolated relevance scoring. For instance, the quality of an LLM's response-gives query and ranked of Setuments--ervee as direct indicator of doc- ument relevance.
(b) Reranker fix N and K for all the questions.
Query
Top
Topk
Retriever
Dynamic Reranker
LLMs
(e) K is dynamic generated for all the question.
Figure 1: Illustration of our dynamic reranker framework. (a) It represents a RAG system with- out a reranker, where the system primarily focuses on training LLMs. (b) It represents a RAG sys- tem with reranker, where the reranker trained to filter the Top-N documents to fixed Top-K, which remains constant for all queries. (C) In con- trast, it represents our dynamic reranker, where both the reranker and the generator are trained si- multaneously. The dynamic reranker adapts to the difficulty of each query by dynamically determin- ing the value of k.
Based on these insights, we propose DynamicRAG, a novel RAG framework where the reranker dynamically adjusts both the order and number of retrieved documents based the query. In DynamicRAG, the reranker is modeled as an agent optimized through reinforcement learning (RL), with rewards derived from the evaluated quality of LLM outputs. The entire training process consists of two stages. First, we adopt behavior cloning by collecting expert trajectories and training the reranker via supervised fine-tuning (SFT). This provides the reranker with basic understanding of the dynamic reranking task while reducing the complexity of the action space. Second, we treat the generator as an interactive environment that provides feedhack, enabling the reranker to explore, collect trajectories, and update itself through reinforcement learning.
We comprehensively evaluate DynamicR on knowledge-intensive tasks across seven datasets, including general QA (NQ (25). TriviaQA 119,, multi-hop reasoning (HotpotQA [59]. 2WikimQA (15), long-form generation (ASQA (11, ELI5 (0), and fact verification (FEVER (55)). Addition- ally, we assess the recall results of DynamicRACo AG's reranker on NQ and HotpotQA. Experimental results show that DynamicRAG significantly outperforms existing fine-tuned and prompting proaches, achieving state-of sate-of-tthearr (SOTA) performance among models of same size while requiring substantially less training data.
ap-
2 DynamicRAG
In this section, we propose DynamicRAG. Unlike traditional RAG systems that rely on static ranking methods, DynamicRAGi introduces a dynamic reranking mechanism and leverages feedback from LLM output to further refine the reranker, thereby achieving overall optimization of the RAG system.

---

The DynamicRAG framework consists of three key components: (1) frozen retriever, (2) trainable dynamic reranker, and (3) trainable generator that is optimized effectively leverage the reranker's 's dynamically selected l documents. The retriever retrieves relevant documents from large corpus, while the reranker dynamically determines both the order and the number of documents to passed to the generator to produce an answer. The generator then produces the final output based on the reranker's s selected documents. By iteratively training the reranker and generator, DynamicRAG achieves improvements in the overall efficiency and effectiveness of the RAG system.
2.1 Dynamic Reranking
Traditional reranking approaches rely on static ranking models that determine the relevance of retrieved documents independently of the generation task. These models typically operate within fixed-length input framework, where a list comprising n documents serves as the input, and the output is reordered sequence containing the same n documents. This inherent limitation prevents these static models from dynamically adapting to the specific needs the generation process, particularly in terms of the number and arrangement of the selected documents. In contrast, DynamicRAG uses feedback from the generator to guide the reranking process, allowing it to dynamically adjust both the order and the number of documents selected. This enables the reranker optimize the input of the generator, maximizing the likelihood of producing high-quality output (In this paper, we only consider the list-wise ranking).
To formalize this operation, the reranking pro- single function
Dynamic Reranker Agent
Top-K Docs
cess can be expressed that directly outputs a reordered subset of the initially retrieved documents. Given query and set of retrieved documents {Di, D2, ..... D the reranker computes:
Generale
DynamicRAG
Optimize
Training
Interact
D Beranker,, (q. D),
(1)
Paradigm
where D D is the selected and reordered subset of documents, and 0, represents the pa- rameters of the reranker model. This formula- tion encapsulates both the scoring and selection processes, producing a subset D with dynam- ically adjusted order and size. This dynamic adjustment enables the model to exhibit greater flexibility accommodating diverse queries and the specific requirements of the generation task.
Reward
Results
Environment (Generator)
Figure 2: Illustration of the training paradigm for our method. We treat Dynamic Reranker as an Agent, which interacts with the Environment, gen- erating Top-K docs and receiving rewards to opti- mize itself.
2.2 Training with Interacting
In this section, we outline the training process for the Reranker, where the architectural overview of which is schematically depicted in Figure The training begins with behavior cloning, which equips the Reranker with the basic ability to adjust both the order and the number of selected documents. Building upon this foundational model, we further refine the Reranker's performance through interactions with the environment. This interaction allows the Reranker to collect feedback from multiple trajectories and enhance its decision makiing policy. The complete training procedure is illustrated in Figure and is presented in detail in Algorithm
2.2.1 Behavioral Cloning with Expert Trajectories
Behavioral cloning [58 63 is used to supervised fine-tuning the Reranker by mimicking expert trajectories, allowing the model to learn the fundamental actions required for effective ranking. In this stage, the Reranker focuses on learning how predict the correct intermediate actions a, based on the given task and context. To achieve this, the Reranker is trained on dataset of expert demonstrations, denoted D.. This enables the model to acquire basic instruction-followiii prior knowledge. The training objective is to maximize the likelihood of the expert's 's document selection decisions:
capabilities and leverage
JBC(0)= E.g.D,>>>D,, [log Te (klg,, D)).
(2)
3

---

What released the "Amblence" collection 2001?
Step4: Dynamic Reranker Sampling Trajectories
Stepl: Reirieve Top-N
Step2: Expert Reranker Generates Score
New Question
Doc3, Doc5, Doc35
Documents
Documents
...
1. The Gordon brothers released 'Soothing Sonctuary". ombient music
8g
Top-N Decuments
Doc8, Doc1, Doc2
Documents
0.86
Step5: Rewarding and learning from Experience
2. Four Seasons' and "The Classice' built on the success the Classical
0.42
answerl
0.90
Converted Documentts
...
Generator
Top-Ni Mannhelm Steomroller American Neaclassical new-age music group..
ohswer
0.65
9n
Converted Decumeats
Multi- Dimensional Reward
Step6: Training
Step3: Dynamic Reranker mimie the expert trajectory
Documents
New Question
Argmax r
Doc3. Doc5, Doc35
Top-N Documents
Doc1, DocN,
Doc3
Top-N Documents
Argmin Fy
Dec8, Doct, Doc2
x
Question
Doc4, Doc2,
Doc14
Final Answer
Documents
Top-N
Documents
Inference Phrase
Figure 3: Illustration of our training framework. During the training phase, we have total of six steps. First, we retrieve the Top-N documents based on the given question. Then, we use an expert model score each document and filter a subset of data for behavior cloning by the dynamic reranker. Next, we use the trained dynamic reranker to sample multiple different trajectories. The responses generated by the generator serve as rewards to evaluate the quality of the trajectories, and we select the trajectory pairs with the highest and lowest rewards as the training data for DPO. During the inference phase, DynamicRAG only require two L.M inferences.
where g denotes the query, D = {di d2, d2...,, dN} represents the set of retrieved documents, k* is the expert-demonstraated optimal number of documents, and To (klkg D) represents the conditional probability of selecting & documents from the candidate set D given the query q.
2.2.2 Optimizing the Reranker via Exploration and Feedback
After the initial training with behavioral cloning, the Reranker requires further refinement to align its actions with the goal of maximizing response quality. This is achieved through an interactive learning process, which the Reranker continually interacts with the environment to gather feedback, progressively improving its action policy.
The primary objective is to train a policy ma that maximizes the expected reward across all trajectories T for given environment G and user instruction i:
Tesptore (I)
Eg.t Eg (R( i, i,,
where RCG, 7) quantifies the response quality. However, optimizing this objective is challenging due to the complexity of the action space and the inherent inefficiencies in sampling. To address this, we adopt the Direct Preference Optimization (DPO) framework 44] which simplifies the reward optimization with pairwise comparisons of trajectories.
Formally, given set of sampled trajectories (T, N with rewards (r.),,, identify the trajectory pair (T*,T -T such that:
arg max Ts: T arg min T;
The whole training objective is then defined as:
(0) E((,,r-)) log (3(log (T( ---+t-i -log To( )))
(5)
n G,
To(r-)= To (9, i) [T -9,
To(T+)
(6)
( (G,

---

This objective encourages the Reranker to assign higher probabilities to trajectories with greater rewards.
By iteratively combining environment feedback and RI. optimization, the Reranker transitions from a basic policy to robust, task-specific system capable of generating high-quality responses.
2.2.3 Reward Function Design
To evaluate the quality of the generated response ý in relation to the ground-truth answer Ugt and the contribution of reranked documents {D; ) we employ a multi-dimensional reward function. This function integrates five key aspects of quality: Exact Match (EM), Semantic Similarity (SS), Textual Fluency (TF), Length Penalty (LP) and LLM-Based Evaluation (LLM-Eval). These dimensions collectively provide holistic assessment of the generated output.
Formally, the reward function defined as:
r
EM B SS + TF LP LLM-Eval
(7)
where EM, SS, TF, LP, and LLM-Eval are computed as functions of (yge, 9) and a, 8, X, and are weighting coefficients for each quality dimension.
The individual components are defined as follows:
Exact Match (EM): Measures whether á matches Dot exactly:
{
if Ugt: otherwise.
ExactMatch(yge: 9)
(8)
Semantic Similarity (SS): Assesses the alignment between I and you using BERTScore 64
SemanticSimillrityyuur v) BERTSScre 9).
Textual Fluency (TF): Evaluates fluency using ROUGE 28 metrics:
TextualFluency(1ng (ygt 9) = ROUGE 9)
(10)
Length Penalty (LP): Encourages concise answers by penalizing longer responses:
1
LengthPenalty (i) len
(11)
Uses LLM-based scoring to assess alignment with task requirements, where P denotes the scoring prompt, detailed in Appendix D.3
LLM-Based Evaluation (LLM-Eval):
LLM-Eval (ygtt 9) = LLMScore Vgt 0).
(12)
Experiment
3.1 Datasets and Evaluation Metrics
We evaluate DynamicRAG's performance on comprehensive knowledge-intensive question-answering tasks, spanning over seven datasets and covering different types of challenges: Natural Questions
255. TriviaQA [19, HotpotQA [59], 2WikimQA [15]. FEVER (55), ASQA [5T), and ELI5 [TO]. For the first five datasets (NQ, TriviaQA, HotpotQA, 2WikimQA, and ASQA), follow prior studies
[1] and adopt exact match as the evaluation metric. For TriviaQA and FEVER, we used accuracy,
while for ELI5, we employed ROUGE-L scores to assess performance 62 41
3.2 Baselines
Baselines without Retrieval We evaluate publicly available close-sourced LLMs, including GPT- 3.5-turbo, GPT-4, and GPT-4o. These models represent state-of-the-are LLMs that are not augmented with external retrieval information. The system prompts and instruction formats are shown in Appendix D.3
5

---

Table The DynamicRAG results for different datasets among different backbone models. Results are directly from the original paper. Best results are in bold and the second results are underlined. denotes that FLARE is based on the more powerful 175B lext-davinci-003 model, which we currently do not have access to.
Extra Data
NQ
TriviaQA
HotpotQA EM
2WikinQA
ASQA EM
FEVER Acc
ELI5
Metrics
for Training
EM
EM/Acc
Rg
Baseline Without Retrieval
G.T-3.5
29.9
68.3
27.5
GPT- 89
N/A
40.3
73.3/78.4
34.5
29.8
71.9
87.7
30.3
GPT-4o 38
74.0799.2
36.1
74.1
Baseline With Retrieval
17.4
ReAct 60
N/A
35.1
62.0
1,129k
43.5
36.6
FLARE" 18
N/A
51.0
41.3
Reward-RAA
42.2
75.6/80.4
LLaMA2-78 56
N/A
17.9
-/42.5
16.6
17.9
19.0
30.0
15.6
Reranker
20.6
-/49.6
18.9
21.1
35.6
16.7
LLaMA2-7B-SFT
130k
29.1
53.7159.1
27.1
18.9
23.8
40.6
18.6
Self-RAG (LLaMA2-7B)
150k
36.4
-/6.4
30.0
DRAGIN 552
N/A
23.2
22.0
Smart-RAG (LLaMA2-7B) 12
218k
26.6
Ours (L.LaMA2-7B)
150k
38.7
59.6/70.5
29.4
23.1
41.1
51.2
22.6
L1aMM2-13R [5
N/A
23,6
-/47.0
17.7
18.7
20.5
30.2
19.9
Reranker
NIA
26.5
./5.22
20.4
18.8
23.6
37.1
20.3
11MMA2-33--FF
130k
32.5
1/66.2
27.9
19.1
28.4
45.8
20.1
SelE-AAG ALLaMA2-133
150k
.69.3
31.7
Ours (LLLMA2-13B)
150k
39.1
62.3/72.6
30.1
25.0
46.4
77.2
23.3
LLaMA3-8B 13
36.4
-/57.4
26.1
24.9
34.6
24.0
w/ Reranker
N/A
37.5
-664.5
28.7
25.3
29.8
49.7
23.7
LLaMA3-8B-SFT
39.1
67.5/74.2
31.5
82.1
22.9
Auto-RAG LLBMA3-88- Instruct) 61
Unknown
37.9
60.9V-
30.0
ChaiQA-1.5 (LLaMA3-8B:
32
42.4
33.4
RankRAG (LLaMA3-8BB 8B)
62
470%
50.6
82.9/89.5
35.3
31.4
93.8
Ours (LLaMA3-8B)
78.3/87.4
36.7
56.3
24.6
Baselines with Retrieval We compare our method against several retrieval-augmenteed baselines. The baselines are categorized into four groups follows: RAG-based Basclines: This group includes approaches such as IRCoT ReAct (60), FLARE [8] RA-DIT (29) and Reward-RAG
[36), which leverage agent-liks strategies for attieval-augmennt generation. -LaMA2-7B-baaa
This category consists of standard retrieval-augmennee baselines such as Vanilla-RA -AAI as well as its enhanced versions with additional components like reranker, SFT, Self-RAG M. DRAGIN [52] and Smart-RAse RAG [1]. LLaMA2-1E 1----33B-asee Baselines: Similar to the LLaMA2-7B group, this set includes Vanilla-RAG: Reranker, SFT, and Self-RAG (1], providing larger-scale comparison using the LLaMA2-13B model. 3LLaMA3-B--bar
Baselines:
Baselines: In this group, we consider models based on LLaMA33888 including Vanilla-RAG and its variations with Reranker and SFT. Additionally, we compare our models with more advanced retrieval-augmeened methods such as Auto-RAG [61].
ChatQA-1.5 32
and RankRAG 62
3.3 Implementation Details
Training data and settings Our training data comprises I 150k diverse instruction-outpl pairs, drawn from Alpaca [54], KILT [41) ASQA [5T] and OpenBookQA (35). We employ three models as the base LMs for our dynamic reranker and generator: LLaMA2-7B. LLaMA2- 13B, and LLaMA3-888
For the retriever model, we use the off-the off-the-shelf Contriever-MS MARCO (16) as the default retriever. retrieving up to 45 documents for LLaMA3 and 20 documents for LLaMA2 input, tailored to their respective context window sizes. Unless otherwise specified, the retriever and generator share the parameters. Additional training details can be found in the Appendix
Inference settings For the dynamic reranker, we set the temperature to 0.2 to enhance output diversity. For generator, we use temperature 0 to ensure output stability and reproducibility. By default, we use the top 45 documents from Contriever-MS MARCO [66 as input to the reranker. In contrast, all baseline methods use the top 10 documents from Contriever-Me MS MARCO as input to ensure fair comparison.

---

Table The performance of different Reranker models. Results are directly from the original paper. Best results are in bold and the second results are underlined.
NQ
HotpotQA R@ 10
Training Data
Avg.
Metric
R@5
R@ 10
R@20
R@s
R@20
Gose-SSurce Models
GPT-b.5-Turba
Unknown
77.8
82.5
85.7
52.1
56.6
62.4
GPT-4 39
Unknown
79.3
83.2
85.1
53.2
57.0
61.0
69.8
Open-Source Rerank Models
BM25
N/A
38.0
50.7
57.5
63.0
67.5
Contriever 16
Unknown
73.6
80.2
84.8
53.1
58.7
62.4
monoTT 37
~ 503k
75.6
80.9
84.9
54.8
60.2
63.3
70.0
RankLLaMA
77.8
83.1
86.0
57.1
62.1
64.8
ChaiQA-1.5 (LLBMA3-SB)
32
N/A
68.2
75.7
82.0
37.4
45.0
53.6
RankRAG (LLaMM3-88)
62
50k
80.3
84.0
86.3
57.6
61.8
65.2
Open-Source Generative Models
GENRE 4
406k
34.0
Re3eval
240k
65.4
44.2
SEAL
Unknown
68.2
51.0
DynamicRAG (LLaMA3-8B)
79.3
83.7
86.8
59.1
63.7
67.2
73.7
3.4
Main Results
3.4.1
Comparison against baselines with and without retrieval
Table presents comprehensive comparison of our proposed DynamicRAG approach against various baseline models, categorized into those without retrieval and those incorporating retrieval mechanisms. For baseline models that do not utilize the retrieval. such GPT-3.5-Turbo. GPT-4, and GPT-4o, our approach significantly outperforms them across multiple datasets. Notably, in the NQ dataset, method achieves EM score of 48.4 with LLaMA3-8B, surpassing GPT-4o. These results highlight the effectiveness our retrieval-augmented approach compared to models without retrieval capabilities. When compared agent- agent-based based baselines such IRCoT, ReAct, and Reward- RAG, our DynamicRAG framework consistently achieves state-of-the-art performance across various datasets. Moreover, our approach achieves superior performance compared to other retrieval-based models such as RankRAG and ChatQA-1.5, despite using significantly less training data (~ 10kk examples 4470k for RankR AG). Second, the effectiveness of method is consistent across different backbone sizes (7B, 13B, 8B), showing scalability and robustness.
3.4.2 Comparison with baselines for reranking performance
The Table 2 presents comparison of different reranker models categorized into three groups: close-sourced models, open-sourced rerank models, and open-sourced generative models. Our LLaMA3-8B-basee model, DynamicRAG, demonstrates competitive performance while utilizing only 20k training samples, achieving results comparable to RankRAG, which requires 50k training samples. Furthermore, our model significantly surpasses other open-sourced models, such as monoTS, RankLLaMA, and generative models like Re3eval, across key ranking metrics (R@5, 5, R@ 10, and @@20). This highlights the effectiveness of our approach efficiently utilizing limited training data without compromising performance.
Additionally, our approach adopts list-wise reranking strategy, which contributes superior overall ranking efficiency compared to other models that primarily rely on point-wise methods. Notably, we leverage the quality of generated responses signal for reranking, which significantly enhances model performance, particularly when compared to traditional information retrieval-based models.
3.5
Ablation Studies
We perform various ablation studies to understand the importance of different factors in DynamicRAG.
3.5.1
Impact of Reranker and Generator Size
The Table presents results from different Reranker and Generator configurations in the Dy- namicRAG, evaluating model variants with LLaMA2-7B, LLaMA2-13BB 13B, and LLaMA3-SB, where
7

---

(a) Results of different models and sizes for Reranker and Generator. denotes share the parameters of the Reranker and Generator. use Exact Match as the metric.
(b) The impact of different key components in Dynam- ieRAG among different benchmarks. We use Exact Match as the metric.
NQ
HotpotQA EM
ASQA EM
Avg.
Dy DynamicRAG
NQ EM
HotpotQA EM
EM
Reranker
Generator
Avg.
DynamicRAG
48.4
36.7
56.3
47.1
LLaMA2-7B
1LMMA2-138
HLMMA2-13B
LLaMA2-13B
37.6
38.7
29.8
28.6
35.1
34.3
w/o Retrieval
25.0
25.6
15.7
22.1
LLaMA2 -
LLaMA2 13B
39.1
30.1
34.6
w/o Reranking
36.4
27.2
39.8
34.5
11aMA3-8B
11MMA2-13B
30.2
34.8
w/o RL
44.6
29.6
45.5
39,9
performance improves model size increases. The 13B Reranker and 13B Generator configuration outperforms the 7B-13B setup in both NQ and HotpoiQA, with the average EM score rising from 33.1 to 34.3. Switching to 8B Reranker with 13B Generator results in slight further increase in the average EM score to 34.8, suggesting that a larger Reranker can enhance performance, even with fixed Generator size. The model where both the Reranker and Generator share parameters (denoted by *) achieves the EM of 39.1 on NQ and 30.1 on HotpotQA, yielding an average EM score of 34.6. This improvement indicates that sharing parameters allows the Reranker and Generator to better complement each other, as their tasks can mutually enhance performance.
3.5.2 Effect Key Components on DynamicRAG Performance
The Table 3b) shows the performance of DynamicRAG under various ablation conditions, where key components such as retrieval, reranking, reinforcement learning, and iterative training are removed. Evaluation on NQ, HotpotQA, and ASQA reveals that removing retrieval causes significant drop in performance, with EM scores falling to 25.0 on NQ, 25.6 on HotpotQA, and 15.7 ASQA, leading to an average EM of just 22.1. This emphasizes the critical importance of retrieval in supplying relevant information. Excluding reranking results in a degradation in performance, especially on NQ (EM = 36.4), indicating that reranking has a beneficial effect. Removing RL also hinders performance across all datasets, with the average EM decreased to 39.9, particularly on NQ and ASQA (44.6, 45.5), suggesting that RL significantly benefits performance.
3.6 Model Analysis
3.6.1
Efficiency
From LLM-Calling Perspective We evaluate the maximum number LLM calls required by different RAG models to generate answer when the retriever returns 20 documents. It is evident that the closer model is to the top-left corner, the better it performs, as both effective- ness and efficiency positioned in the top-left corner, demonstrating superior performance compared to other models. Specifically, our model can generate answers with only two LLM calls when the retriever re- turns 20 documents.
so
Rank RAG
Ours
45
ChatQA-113
Reward RAG
Auto RAG
RAG
Vanilla RAG w/R
35
optimized. Our model is
Vanilla RAG
ReAct
30
Smart RAG
IRCOT
2.5 5.0
10.0
12.5
17.5 20.0
From Token Perspective We empirically evaluated computational efficiency against ex- isting methodologies, notably RankRAG. Our architecture processes Question + Top-20 docu- ments concurrently, producing Top-k documents through single Reranker pass before generating final answer via Question + Top-k document integration. In contrast, RankRAG's methodol- ogy necessitates separate processing of Question + individual document pairs, requiring 20 distinct forward passes for a corpus of 20 documents. As-
Figure 4: Comparison of different RAG models in terms of efficiency and effectiveness. The x-axis represents the number of LLM calls, while the y- axis denotes the average performance on the NQ benchmark. Models closer to the top-left corner achieve better overall performance.

---

suming & = 10 for both approaches, Table presents mean runtime metrics (averaged across three
experimental iterations):
Table Comparative analysis of processing latency from token-input perspective.
Input
Time (Seconds)
Question + Top-10 Docs (Vanill-RAGG
0.57
Question Top-20 Docs (4k context window) to rank
0.75
(Question + Single Doc) x 20 for reranking
13.00
Question Top-k Docs (avg lc 12)
0.61
Results demonstrate that contextual integration yields substantially higher computational efficiency compared to multiple LLM invocations. (As the RankRAG implementation is not publicly available, we constructed functional equivalent utilizing LLaMA2-78
matching the original architecture's ccale-deployed within VLLM framework on 8 A 100 GPUs.) Our methodology demonstrates approximately 17x superior throughput relative to RankRAG's sequential scoring approach. Fur- thermore, compared to standard RAG pipelines without reranking, our approach introduces minimal computational overhead overhead--ppecfficalll specificallyy 2.3x latency increase while delivering significant perfor- mance gains. For instance, the NQ benchmark, methodology demonstrates a 9.6 percentage point improvement over vanilla RAG implementations using LLaMA2-7B.
3.6.2 Reranked Document Distribution
We analyzed the reranked results of Dynami-
cRAG on NQ and HotpotQA, as shown in Fig- ure Before RL training, the reranked results
NQ After
HotporGAA Atter RL
NG Before RL
HotporQAA Bafors RL
2000
on NQ and HotpotQA predominantly had val-
ues of 14 and 15. This is because the model
1500
trained solely with SFT tends to favor higher
number of reranked documents, achieving bet- ter downstream performance. However, after RL training, especially with the introduction of the length penalty, leftward shift in & values can be observed, with peaks appearing at 12, 13, and 14. This indicates tendency to output fewer reranked documents. This also proves the effectiveness of our RL training.
1000
500
Number of K
Figure 5: Distribution of reranked document num- bers (k) on NQ and HotpotQA before and after RL training. k is truncated at 15 to ensure fair com- parison, as we restrict & S 15 during both training and sampling.
Due to space limitations, we defer several im- portant analyses to the Appendices. Appendix Clincludes ablation studies on different retriev- ers, Top-N document selection, reward function components, and training experiments with closed-source models. popendix(prooiddeee comprehen- sive implementation details, qualitative examples demonstrating our dynamic reranking approach, and complete prompt templates.
4 Conclusion
This work introduces DynamicRAG, a new reinforcement learning framework to optimize reranking in RAG. By modeling the reranker as an RL agent and using LLM response quality as rewards, it dynamically adjusts the order and number of retrieved documents per query. This dynamic reranking mechanism enhances both the relevance of selected documents and the overall system efficiency. Extensive evaluations on seven Knowledge-intensive datasets demonstrate that DynamicRAC consis- tently outperforms existing fine-tuned and prompting- based approaches, achieving oftte-of-thearr performance.
We selected a maximum 15 documents for reranking to maintain fairness comparison with existing studies, which commonly evaluate top-10 10 documents. Given LLaMA-7B- A 4K token context length and an average document length of approximately 200 tokens, accommodating 15 documents and their associated queries comfortably feasible S0GB A100 GPU.

---

5 Acknowledgement
Research was supported in part by National Science Foundation IIS-19-56133 NSF IIS 25-37827, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (1-GUIDE) by NSF under Award No. 21 2118329. The research has used the Delta/DeltaAl advanced computing and data resource, supported in part by the University of Illinois Urbana- Champaign and through allocation #250851 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants OAC 2320345, #21 #2138259, #2138286, #2138307, #2137603, and #2138296. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Government.
References
[1] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi. Self-rag: Learning retrieve, generate, and critique through self-reflection, 2023. URL URL https:///aattt www.a.ig..aaaaa.a..... wwwgga///////oooo/o 2tt//2281..11111111111111111111.
[2] M. Bevilacqua, G. Ottaviano, P. S. H. Lewis, S. Yih, S. Riedel, and F. Petroni. Autoregressive search engines: Generating substrings as document identifiers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Process- ing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA,, USA, November 28- December 9, 2022, 2022.
[3] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driess- che. J.-B. Lespiau, B. Damoe, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In International Conference Machine Learning, pages
2022.
[4] N. D. Cao, G. Izacard, S. Riedel, and F. Petroni. Autoregressive entity retrieval. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http:ttt/p//peeeer//// http:////pppppaooooooooooooo/oo/ mett/Eorumiidd5x366603
[5] W.-S. Chao, Z. Zheng, H. Zhu, and H. Liu. Make large language model a better ranker, 2024.
httpst:::: :ttttps://rrree ht///arxivvv.... /403. 19181
[6] J. Chen, H. Lin, X. Han, and L. Sun. Benchmarking large language models in retrieval- augmented generation, 2023. htt.ttp:://aiii.orrgg//////////p 23090009.01131
[7] Chen, B. Gutiérrez, and Y. Su. Attention large language models yields efficient zero-shot re-rankers, 2024. URL https://arxii: wrg. 0/1.//211....2....
[8] Devlin, M.-W. Chang, Lee, and Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), Minneapolis, Minnesota, June 2019, Association for Computational Linguistiss.
[9] A. Drozdov, H. Zhuang. Z. Dai, Z. Qin, R. Rahimi, X. Wang, D. Alon, M. Iyyer, A. McCallum, D. Metzler, and K. Hui. Parade: Passage ranking using demonstrations with large language models, 2023. URL httph:ttt//// httphhttt///xtitii http:://axxxvvvvo wwtps//.rggabbbb////e//ooesees/s 2310. 11110.110111..11111
[10] A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. ELI5: long form question answering. A. Korhonen, D. R. Traum, and L. Mårquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3558-3567. Association for Computational Linguistics, 2019.
10

---

[11] R. Gangi Reddy, J. Doo, Y. Xu, M. A. Sultan, D. Swain, A. Sil, and H. Ji. FIRST: Faster improved listwise reranking with single token decoding. Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8642-8652 Miami, Florida, USA, Nov. 2024. Association for Computational URL https/ttt:///ccannttt
Linguistics. doi: 10.18653/v (0/665/v1/200/ http:///2222...mmmmammmmm.....a. 2024. emnlp- 4911
org/
[12] Gao, I.. Li, W. Li, Fu, and B. Dai. Smartrag: Jointly learn rag-related tasks from the environment feedback, 2024.
wwwp://wwww//////11/.11..// abs/aaab...... 18141
[13] A. Grattafiori, A. Dubey, A. Jauhri, et al. The llama 3 herd of models, 2024. URL https
7/arxiv. larxiv.orgg//e absa 2407 21783
[14] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. Realm: -aggmeetd language model pre-training. In International Conference on Machine Learning, 2020.
[15] X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In D. Scott, N. Bel, and C. Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6609-6625. International Committee on Computational Linguistics, 2020.
[16] G. Izacard, M. Caron, I.. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave Unsuper- vised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022. URL htttps://operreviieee http:://pppaarrrreeeeeeeeroorere
KN KN1pXi7b0
[17] G. Izacard, P. Lewis, M. Lomeli, L.
Hosseini, Petroni, T. Schick, Dwivedi- Yu, A. Joulin, S. Riedel, and E. Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, pages 1-43, 2023.
[18] Z. Jiang. F. Xu, Gao, Z. Sun, Q. Liu, J. Dwivedi- Yu, Y. Yang. J. Callan, and G. Neubig. Active retrieval augmented generation. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Dec. 2023. Association for Computational Linguistics.
[19] M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver. Canada, July 30 - August 4, Volume 1: Long Papers, pages 1601-1611, Association for Computational Linguistics, 2017.
[20] V. Karpukhin, B. Oguz, S. Min, P. S. H. Lewis, L. Wu, S. Edunov. D. Chen, and W. Yih. Dense passage retrieval for open- ppmninomain question answering. In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781. Association for Computational I. Linguisties, 2020.
[21] Z. Ke, W. Kong, C. Li, M. Zhang, Q. Mei, and M. Bendersky. Bridging the preference gap between retrievers and 1lms. In L. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 10438-10451, Association for Computational Linguistics, 2024. doi: 10.18653/V (018865//1/2222..0 //doi. (28653/v1/20244.00 ac long..lllooo.n..mmmm 562
URL https
[22] U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations (ICI.R), 2020.
[23] E. Khramtsova, S. Zhuang, M. Baktashmotlagh, and G. Zuccon. Leveraging llms for unsuper- vised dense retriever ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, page 1307-1317 ACM, July 2024. doi: 10.1 00.11455/66677723557738 URI, http://dx doi org/10..11/0 1111//111111/3336677
3657798

---

[24] M. Kulkarni, P. Tangarajan, K. Kim, and A. Trivedi. Reinforcement learning for optimizing rag for domain chatbots. arXiv preprint arXiv:2401.0680/ 2024.
[25] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452 466, 2019.
[26] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474, 2020.
[27] P. Liang, R. Bommasani, Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang. D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang. C. Cosgrove, C. D. Manning. C. Ré. D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong. H. Ren, H. Yao, J. Wang, K. Santhanam, L. OIT, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, Ganguli, T. Hashimoto, T. leard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Mai, Y. Zhang, and Y. Koreeda. Holistic evaluation of language models, 2023. URL https::/aaaaa
org/
abs/2211. 09110
[28] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, July 2004. Association for Computational Linguistics.
[29] X. V. Lin, X. Chen, M. Chen, W. Shi, M. Lomeli, R. James, P. Rodriguez, J. Kahn, G. Szilvasy, M. Lewis, L. Zettlemoyer, and W. Yih. RA-DIT: retrieval-augmmentdd dual instruction tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[30] W. Liu, Y. Zhu, and Z. Dou. Demorank: Selecting effective demonstrations for large language models in ranking task, 2024. URL htttps:/aaaxi..orgg/bbbbaa
org hbtp////b///2/...//// 12t./
[31] Y. Liu, M. Oit, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
[32] Z. Liu, W. Ping, R. Roy, P. Xu, C. Lee, M. Shoeybi, and B. Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag, 2024. URL (ttttttts://arrx orgg org//bs//////ee/eeeeooooee abs/2401 10225
[33] X. Ma, X. Zhang, R. Pradeep, and J. Lin. Zero-shot listwise document reranking with a large language model, 2023. URL https:://///// //ap:ii////xvivv/// /awxivxxivvv..i wwtp:/iiv.rr///oo.. abs/abbs///3// 02156
[34] X. Ma, L. Wang, N. Yang. F. Wei, and J. Lin. Fine-tuning Ilama for multi-stage text retrieval. In G. H. Yang. H. Wang, S. Han, C. Hauff, G. Zuccon, and Y. Zhang, editors. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 2421-2425 ACM, 2024. fattps/s::///oi org/ 10. 01111111/331771 366767227777777711111
doi: (0.1145/36267723367777
[35] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2381-2399 Association for Computational Linguistics, 2018
[36] T. Nguyen, P. Chin, and Y.-W. Tai. Reward-rag: Enhancing rag with reward driven supervision, 2024. https:::::// //tp///aavvvvva org/ awaaaa///...../ 22000/370000000
[37] R. F. Nogueira, Z. Jiang, R. Pradeep, and J. Lin. Document ranking with a pretrained sequence- to-sequence model. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computatioonal Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACI, pages 708-718. Association for Computational Linguistics, 2020. doi: 10. 1/653//1/2220FFNDIIII FIN.NINNNNNNSS-NNLG....... URL httpss/ts://doi... ortp://////oi..o/11......... 18653/ v1/2020. findings ewiggg-eemll.... 63
12

---

[38] OpenAI, :, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, et al. Gpt-4o system card, 2024. URL pttpss///aaax org abs //s/ 21276
[39] OpenAI, Achiam, Adler, S. Agarwal, L. Ahmad, I. Akkaya, et al. Gpt-4 technical report, 2024. L
org/ http://wiii..rgaa////2233/33 att.///b////////////// 08774
[40] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder,
P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, 2022. URL httts:://aaaat org/ ab2//233..2..... 02155
[41] F. Petroni, A. Piktus, A. Fan, P. S. H. Lewis, M. Yazdani, N. D. Cao, J. Thorne, Y. Jernite, V. Karpukhin, Maillard, V. Plachouras, T. Rocktäschel, and S. Riedel. KILT: a benchmark for knowledge intensive language tasks. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani- Tür, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2523 2544. Association for Computational Linguistics, 2021.
[42] R. Pradeep. S. Sharifymoghaddam, and J. Lin. Rankvicuna: Zero-shot listwise document reranking with open-source large language models, 2023. URL http:: org/ abs/ 2309. 15088.
[43] Z. Qin, R. Jagerman, K. Hui, H. Zhuang, J. Wu, L. Yan, J. Shen, T. Liu, J. Liu, D. Metzler, X. Wang, and M. Bendersky. Large language models are effective text rankers with pairwise ranking prompting. In K. Duh, H. Gomez, and S. Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 1504-1518, Mexico City, Mexico, June 2024. Association for Computational I Linguistics. doi: 10.1 001886531112222 h/tp://///222...nggggaaaaaaa lttitp:://acaantollggeoo or0/
2024. www.ifinnnnnaaaaii 977
[44] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[45] O. Ram, Y. Levine, Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In-context retrieval-auggmented language models. Transactions of the Association for Computa- tional Linguistics, pages 1316-1331, 2023.
[46] S. E. Robertson and H. Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4):333-38, 2009. doi: 10.1561/1500000019, URL https: org/10. 1....1166111000000011
[47] D. S. Sachan, M. Lewis, M. Joshi, A. Aghajanyan, W. tau Yih, J. Pineau, and L. Zettlemoyer. Improving passage retrieval with zero-shot question generation, 2023. URL htttt:t//ssaiiii org/ /b////////a 220...224..000 07496
[48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/ 003//777.063444 2017. URL httt:://arxiv.org/abbe /707 06347
[49] Z. Shao, P. Wang, Q Zhu, R. Xu. J. Song, X. Bi. H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models,
2024. https:ttts//:attt
org/ abs/ 0bbs2402.. 2402. 03300.
[50] E. Song, S. Kim, H. Lee, J. Kim, and J. Thomne Re3val: Reinforced and reranked generative retrieval. In Y. Graham and M. Purver, editors, Findings of the Association for Compu- tational Lingnistics: EACL 2024, St. Julian's ss Malta, March 17-22, 2024, pages 393-409 409. Association for Computational Linguistics, 2024. URL httpt:ttt//aaaaatat findings-ean eacl. 27
org/2022
[51] I. Stelmakh, Y. Luan, B. Dhingra, and M. Chang. ASQA: factoid questions meet long-form an- swers. In Y. Goldberg, Z. Kozareva, and Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 8273-8288. Association for Computational Linguisties, 2022.
13

---

[52] W. Su, Y. Tang, Q. Ai, Z. Wu, and Y. Liu. Dragin: Dynamic retrieval augmented generation based on the information needs of large language models, 2024. https://://aam org/ abs/2403.1 10081
[53] W. Sun, L. Yan, X. Ma, et al. Is ChatGPT good at search? investigating large language models as re-ranning agents. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 14918-14937, Singapore, Dec. 2023, Association for Computational Linguistics. doi: 10. 10.18653/v1/2023 umnlp-main.99 URL https::/aclantttoeeee/ www.olloogoorgg/22222 www.0023...2.....o mmnnp-main...... mww..
[54] R. Taori, Gulrajani, T. Zhang. Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-followig llama model. https://githie cwm/tatsttttt lab/
stanford stanford_alpaca 2023.
[55] Thome,, A. Vlachos, C. Christodoulopoulls, and A. Mittal, FEVER: large-scale dataset for fact extraction and verification. In M. A. Walker, H. Ji, and A. Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationa Linguistics: Human Language Technologies, NAACI-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pages 809-819. Association for Computational Linguistics, 2018.
[56] H. Touvron, L. Martin, K. Stone, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: htap:://arse //wrxv/rxvvvv/// org/ lirg/abb abs/ 2207 09288
[57] H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In A. Rogers, J. Boyd- Graber, and N. Okazaki, editors, Proceedings of the 6/st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014-10037, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 11..8863/11/2222..cllooogg acl URL https://aclaathoooogy org acl- long. 557/
[58] Z. Xi, Y. Ding, W. Chen, B. Hong, H. Guo, J. Wang. D. Yang. C. Liao, X. Guo, W. He, S. Gao, L. Chen, R. Zheng, Y. Zou, T. Gui, Q Zhang. X. Qiu, X. Huang, Z. Wu, and Y.G. Jiang. Agentgym: Evolving large language model-based agents across diverse environments, 2024.
/ //tp///axvvvv//e////v//eeeeawvvv org/ abs/2406. 2406. 04151
[59] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2369-2380. Association for Computational Linguistics, 2018.
[60] S. Yao, Zhao, D. Yu, N. Du I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing rea- soning and acting in language models, 2023. URL uttps///://axxx org abs//bb//2211 03629
[61] T. Yu, S. Zhang, and Y. Feng. Auto-rag: aettonoomousrtttiiivalaaaaagg language models, 2024. URL
generation for large
[62] Y. Yu, W. Ping, Z. Liu, B. Wang, J. You, C. Zhang, M. Shoeybi, and B. Catanzaro. Rankrag: Unifying context ranking with retrieval-augmenned generation in 1lms, 2024. URL https //arxiv. org/ abs /407 024885
[63] A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang. Agenttuning: Enabling generalized agent abilities for llms, 2023. https /tttpp:://araaa //wpxv/./vvvv//////////////vaav org/oagggoooooo www.oogg//////ooo/oo//ooo/oooee/ abss/331..12
[64] T. Zhang, V. Kishore, F. Wu. K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[65] Zhang, S. G. Patil, N. Jain, S. Shen, M. Zaharia, I. Stoica, and J. E. Gonzalez, Raft: Adapting language model to domain specific rag, arXiv preprint arXiv:2403. 1003,
14

---

A Appendix
A.1
Limitations
While DynamicRAG demonstrates strong performance across multiple benchmarks, several limita- tions warrant discussion.
Our current work exclusively employs list-wise ranking methodologies, where the reranker pro- cesses all documents collectively. Point-wise and pair-wise ranking methods, which may offer different computational and quality trade-offs, remain unexplored and will addressed in future research.
optimization approach treats document selection holistically without considering fine- ine-grained step-level decisions. Exploring step-wise optimization, such as progressive refinement or adaptive termination criteria during the reranking process, remains as future work and could potentially lead to more efficient document processing and better interpretabilityy.
We adopt DPO 44 as the sole reinforcement learning algorithm in our main experiments due to its training stability and efficiency. While DPO's off off-policy nature provides practical advantages, our framework's 's well-defined reward function makes it compatible with various on-policy algorithms which we leave for future exploration.
such as PPO 48 and GRPO 49
A.2
Related Work
A.2.1
Retrieval-Augmented Generation
RAG boosts LLM performance by adding external knowledge, enhancing factual accuracy and
context [26] 14]. Research includes retrieval-baat based next-token prediction [22] 45 and end-to-end
Asai et [ optimizes RAG using special tokens for adaptive retrieval and reflection, fine-tuning with critic model. In related work, Ke et al. [21 optimized RAG by training a bridge model to refine the retriever-LLM connection. While sharing similarities, our approach differs in key ways: (1) We optimize the reranker by treating it an agent (2) We jointly train reranker and generator, finding that shared parameters improve adaptation to downstream tasks.
fine-tuning for better integration 3
that interacts with the generator to collect training trajectories.
A.2.2 LLM for Reranking
LLMS are increasingly used for passage reranking, with methods generally being point-wise (assessing individual relevance via relevance or query generation (27 (7)) pair-wise (comparing passage pairs for relative relevance [4])), and list-wise (holistically ranking passages like Learning Rank
5)). Recent advancements like zero-shot reranking with fine-tuned open-weight LLMs and logit-based methods 7] aim to solve issues but often need specific fine-tuning have scalability limits. Our approach treats the LLM reranker as an agent, initially using behavior cloning to imitate expert behavior, then interacting with the generator to create trajectories for further optimization.
53
33
42
30
A.3 Algorithm
The algorithm of our main method is shown in Algorithm D
B Preliminaries
B.1 Retrieval Augmentation Generation
B.1.1
Retrieval Phase
The first step in the RAG framework is retrieve relevant documents or passages from a large corpus. This is typically done using information retrieval (IR) system. Given query q, the IR system selects a set of relevant documents D1, D2, .., D3 based on some retrieval method, such as BM25, dense retrieval with embeddings, a hybrid approach. Formally, the retrieval process can be
15

---

Algorithm DynamicRAG
Require: Expert dataset De, environment G. iterations N, Normal dataset Drrain
1: INITIALIZE reranker Te. and generator =0,
2: STEP 1: BEHAVIORAL CLONING
3: for each sample 7) D. do
4: Update reranker via:
T
Inc(Or) E
log ne, i,
5: end for
STEP 2: GENERATOR TRAINING
7: for each sample (q. D, Ugt) Dtrain do
8: Optimize generator To, via:
T
(0g)
=
log pe, (Vr 11 (..
t=1
9: end for STEP 3: INTERACTIVE. LEARNING
11: for = to N do
12:
Collect trajectories and compute rewards:
EM 3 SSS TFF+ A LP + -LLM-Eval
13:
Optimize reranker via DPO:
(0,) log o (3(lg no, log no. (T-)))
end for represented as:
Retriever(q C), where C is the document corpus, and D represents the set retrieved documents.
(13)
To quantify the relevance score s( Di, q) of each document D, with respect to the query we define:
s sD,, 9) = Score(D;,g))
(14)
where Score(-) is a function specific to the retrieval method employed (e.g., BM25 score, cosine similarity for dense embeddings).
B.1.2 Encoding Phase
Once the relevant documents have been retrieved, both the query and the documents encoded into fixed-size vectors using neural encoder such BERT (})
(Di, D2..... ... D RoBERTa [31], or other transformer-based models. The goal to obtain a dense representation of both the query and documents. Let qenc and D; represent the encoded query and the encoding of document D,, respectively:
qenc Encode(q), (q),
(15)
D, Encode D;)) for i i= 1,2,......
B.1.3 Generation Phase
After encoding, the next phase is to generate an answer or response based on the query and retrieved documents. The generation model takes both the query and the retrieved documents as input and generates a response ý. The generation process can be framed as maximizing the conditional probability:
argmax, p(y q, D1, D2...., Dz),
(16)
16

---

where ply q. D1. D2....... D2) is the likelihood of generating the output y given the query q and the retrieved documents. To incorporate the encoded representations, we model the conditional probability as:
ply q. D,.....,D = Decode(q, DI,..., D&).
(17)
Finally, the generation model is trained to maximize the likelihood of generating the correct response given the query and the retrieved documents. The loss function be expressed
-og poy q, D1, ....... Dx),
(18)
where is the ground-truth response and q. Dı,..., De are the input query and retrieved documents.
B.2 Learning from Environment
We model the Reranker as player and the generator reranker's task in the environment as partially observable Markov decision process (Z, S, A, T, R), where Z is the instruction space, S is the state space, A is the action space, T S A S is the deterministic state transition function, and R S x A + R is the reward function. In our design, we exclude explicit observations O here since we focus on the overall reward of the reranker's complete episode in the environment, rather than step-wise rewards. We leave observation-bant optimization for future work.
environment G. We can formalize the
Given task instruction in environment G, the Reranker generates an action sequence 0 02. ........ dT based on its policy wa. where each action d, (-(-19, i, ......, a1-1) is determined by the history of previous actions. The trajectory represented as:
(aI...., aT) ~ na (1
(19)
T
no (T
II t=1
To (a:l9, h4-1)
(20)
where T' is the number of steps in interaction, and h4-1 (a1,.... (a-1)) represents the history of action sequences up to step t 1. The final reward (G) E [0, 1] is computed based on the quality of the generator's 's response.
C Further Analysis
C.1 Performance with different retrievers
We conducted experiments to compare the performance of Vanilla-RAG and DynamicRAG on three different benchmarks: NQ, HotpotQA, and ASQA, using different retrievers (DPR (20), Contriever, MonoT5), shown in Figure 6 It can be observed that the retriever models improve, both approaches exhibit better performance on downstream tasks. This also demonstrates the robustness of our model.
C.2 Effect of Top-N Documents on DynamicRAG Performance
The Figure 7 presents the performance of DynamicRAG with different numbers of top-K documents (from 50 to 500) used for reranking across three benchmarks: NQ, HotpotQA, and ASQA. We adopt
VanillaRAG
DynamklAAG
@VanllaRRAG
DynamkHLAG
= VaanllaRAG
DynamkRAG
35
50
80
25
40
10
0
DPR
Contrisser
MonoT5
DPR
Contrince
MonuT5
DPR
Contriver
MunoT5
(a) Performance on NQ.
(b) Performance on HotpotQA.
(c) Performance on ASQA.
Figure 6: Performance with different retrievers between Vanilla-RAC -AI and DynamicRAG.
17

---

the same technique as Sun et al. [53 where we use sliding window to handle large document sets which allows the model to process larger corpora efficiently. First, as the number of top documents increases, the performance improves, reaching its peak around Top- 10-1000 or Top-150, with the highest EM scores recorded for NQ and HotpotQA. However, as more documents are introduced (Top-200, Top-300, Top-500), the performance starts to decline slightly, with the average EM dropping from 56.8 (Top-50) to 55.3 (Top-500). This trend indicates that beyond a certain threshold, increasing the number of documents introduces irrelevant misleading information that negatively impacts the model's performance. The challenge of processing and filtering through a larger set of documents, which can introduce noise and reduce the model's 's ability to focus on the most relevant content.
NQ
HotpotQA
ASQA
50-
45-
4
Zero-Shot
Top-50
Top-100
Top-150
Top 200
Tap-300
Top-500
Figure 7: The impact of varying the number of Top-N documents (Top-50, Top- 100, Top-150, Top-200, Top-300, and Top-500) used for reranking on DynamicRAG performance across different benchmarks (NQ, HotpotQA, ASQA). We use Exact Match as the metric.
C.3 Impact of Different Reward Functions on DynamicRAG Performance
The Table 5 presents analysis of DynamicRAG's performance under various reward model configurations, incorporating key components such as EM (Exact Match), SS (Semantic Similarity), TF (Textual Fluency), LP (Length Penalty), and LLMEval (LLM-Based Evaluation). The EM component is found to be critical for open-domain QA tasks, as its removal results a significant performance decline, particularly on benchmarks such NQ, HotpotQA, and ASQA. However, its impact on long-text generation tasks, like ELI5, is comparatively less pronounced. Conversely, the removal of SS or TF functions yields opposing effects: removing SS and TF has more pronounced negative impact on long-text generation tasks, while their effect on open-domain QA is relatively modest. This suggests that SS is essential for improving the model's generalization capabilities, while TF plays crucial role in enhancing text relevance. The LP function, when excluded, results in a slight but consistent drop performance across all benchmarks, indicating its influence on overall model balance by regulating output length and maintaining response coherence. LLMEval, exhibiting effects similar to those of SS and TF, contributes moderately to performance optimization, underscoring its supportive role. Overall, the consistent trends across different reward function configurations highlight that the model's success predominantly driven by the synergy of EM, SS, TF, and LP, with LLMEval serving supplementary component in refining performance.
Table 5: The performance of SynamicRAG with different reward function designs across various benchmarks (NQ, HotpoiQA, ASQA, ELI5). We use Exact Match as the metric for NQ. HotpotQA and ASQA and use Rouge-L the metric for EL15.
Reward Function
NQ
HotpotQA EM
ASQA EM
ELI5
Avg.
EM
Rg
DynamicRAG
48.4 4
36.7
56.3
24.6
41.5
w/o EM
39.6
22.3
35.4
24.4
30.4
w/o SS
47.7
36.0
55.6
22.6
40.5
w/o TF
47.8
36.2
55.7
22.0
40.4
w/o
48.0
36.2
55.8
24.0
41.0
w/o L.ME
48.1
36.3
56.0
22.7
40.8
To further prove the robustness of our reward functions, we additionally experimented with the following configurations (a, 8, X, and 8):
18

---

Hyperparameter 1 (0.3, 0.25, 0.2, 0.15, 0.1) Hyperparameter (0.4, 0.15, 0.15, 0.15, 0.15)
Hyperparameter (0.1, 0.25, 0.2, 0.15, 0.3)
Table 6: Performance comparison across different settings and datasets
Setting Paper Setting
NQ
HotpotQA
ASQA
ELI5
Avg.
48.4
36.7
56.3
24.6
41.5
Hyperparameter 1
48.5
36.6
56.2
24.3
41.4
Hyperparameter 2
48.5
36.7
56.5
24.4
41.5
Hyperparameter 3
48.2
36.6
56.2
24.4
41.3
The results show that different parameter settings yield similar performance, indicating that framework is robust and largely insensitive to these hyperparameters.
C.4 Dynamic Reranker Module Enhances Closed-Source Model RAG Performance
Many studies 62 have demonstrated that increasing the number of input documents introduces more irrelevant information, ultimately degrading model performance. Therefore, dynamically adjusting k remains crucial, even for strong models. In fact, since our reward mechanism leverages responses and ground-truth outputs, our approach fully applicable to closed- source, robust models such GPT-4o. Experimental results validating this claim are presented below:
Table 7: Performance Improvement Using Dynamic Reranker on Closed-Source Models.
Model
NQ
HotpotQA
ASQA
GPT-4o
40.0
36.1
74.1
GPT-4o w/ RAG Top-20
40.4
34.2
73.2
GPT-4o w/ Dynamic Reranker
42.3
36.9
74.8
The results show that the Dynamic Reranker module consistently improves GPT-4o's performance across all datasets, with gains of 2.3, 0.8, and 0.7 percentage points on NQ, HotpotQA, and ASQA, respectively. In contrast, the standard RAG Top-20 approach shows inconsistent results, even degrading performance on two datasets. The Dynamic Reranker module effectively addresses this issue by dynamically adjusting the number of reranked documents k, demonstrating that even powerful closed-source models like GPT-4o can benefit from our approach.
C.5 Training Data Scaling for DynamicRAG
Our main experiments were conducted using a training dataset of 150k samples. To investigate the impact of training data scale, expanded the dataset incrementally 160k, 180k, and 200k samples from the same training corpus, primarily augmenting the training data used the RI. phase. The results of this scaling analysis presented in Table 8
Table 8: Performance comparison with increasing training data size
Model
Training Data
NQ
HotpotQA
ASQA
ELI5
Avg.
150k
48.4
36.7
56.3
24.6
41.5
160k
48.6
36.8
56.6
24.6
41.7
DynamicRAG-8B
180k
48.7
37.0
57.0
24.8
41.8
200k
49.0
37.0
57.2
24.9
42.0
As demonstrated by the results, the performance of our model consistently improves as the training data size increases. This trend is particularly noteworthy given that in our main experiments, DynamicRAG did not outperform AG on three of the benchmarks. However, it is important to highlight that our model was trained with less than one-third of the training data used for RankRAG. The steady improvement in performance with increased data volume suggests that with comparable training data sizes, DynamicRAG has the potential match exceed the performance of RankRAG.
19

---

Table 9: Full list of instructions used during our evaluations. We use the same prompt when eval Open- Open-domain QA (NQ, TriviaQA, HotpotQA, 2WikimQA, ASQA.)
Dataset
Instruction
ARC FEVER
Please answer the following questions and directly output the answer options. Please answer the question with "SUPPORTS", "REFUTES" "NEI" based what you know.
ELJ5
Please answer the question with paragraph.
Open-domain QA
Please answer the question with short phrase.
D Experiment Details
D.1 Training Details
We train our models on Nvidia A100 GPUs, each with 80GB memory. Fine-tuning is performed for I epoch with an effective batch size of 4, achieved by setting a per-device batch size of 2 and accumulating gradients over steps. The learning rate is configured at 1e-5, with a 10% warmup ratio and a cosine learning rate schedule. Maximum token lengths are set to 4,096 for LLaMA2 and 8,192 for LLaMA3, adhering to the training configuration. Multi-GPU distributed training is efficiently handled using DeepSpeed Stage 3, with B precision to optimize memory usage. FttshAttention enhances the efficiency long-context training. For behavior cloning, we employ MonoTS (37 as the expert model, set 0.8 and constrain the number of documents to maximum of 15, since many works [I [22 only use the top-10 10 as the input for the generator, we aim to obtain relatively fair comparison,
we do not set k too large. For 3, X, and 5 in reward functions, we simply use (0.2, 0.2, 0.2, 0.2, 0.2). For sampling trajectories, we use temperature to 1.0 and top to 0.9. For DPO training, we used the following configuration: learning rate of 5e-6, epochs, a 10% warmup ratio, and a cosine learning rate schedule. The batch size was set to 1, and accumulating gradients over 4 steps. For inference, we leverage the same A100 GPUs and utilize vLLMs to accelerate inference time.
D.1.1 Retriever Setting
To construct the training data, we retrieved the top 45 documents using Contriever-MS MARCO from official 2018 English Wikipedia embeddings. These documents were used to create training datasets for both the reranker and generator components. Unlike other works, do not use retrieval results from external sources, such as Google Search. Instead, all evaluations are strictly based on retrieval results from the same retriever used for training data construction, ensuring consistency and comparability.
D.1.2 Training Dataset
We utilized the Alpaca and specific KILT benchmark datasets, including NQ, FEVER, HotpotQA, ELI5, and TriviaQA, as well as ASQA and OpenBookQA, totaling approximately 150k data instances. And we employ 20k for cold-stan start reranker training, 100k for supervised fine-tuning of the generator, and 30k for DPO training.
D.1.3 Evaluation Setting
For ELI5, we set the maximum token length to 256 to accommodate the benchmark's requirement for long-answer responses. For all other benchmarks, the maximum token length is set to 50. Table 9 shows the list of the instructions used during evaluations.
D.2 Qualitative Examples
Tables and 11 present two distinct examples illustrating the effectiveness of our approach. In the
first example, the dynamic reranker produces reordered sequence and selects a different number of retrieved documents compared to the expert model, yet it successfully generates the correct answer. This demonstrates the reranker's superior ability to identify the most relevant information. In the second example, no additional documents are selected. This suggests that the reranker recognizes the
20

---

query as sufficiently straightforward for the generator, eliminating the need for external information and thereby enhancing the overall efficiency of the generation process.
D.3 DynamicRAG Prompt
DynamicRAG includes three prompts: the prompt used for constructing dynamic reranker data, the prompt used for constructing retrieval-based generator data, the prompt for the reward function, and the prompt for GPT and LLaMA baselines as shown in Tables
13 14
and T6 respectively.
21

---

Table 10: Case Study for DynamicRAG. We compare method with Vanilla-RAG with Reranker (monoTS). The initial retrieved content the same for both methods. We take Top-40 documents as the input.
Question
What is the name of the American Neoclassical new-age music group that released the "Ambience" collection in 2001?
GPT-4o
The Ambient.
Vanilla-RAG
(Reranker Results:)
w/Reranker
Doe 2: Pure Moods Pure Moods was the first United States release a series of compilation albums of new-age music released by Virgin Records. The original was titled "Moods A contemporary Soundtrack" and released in the UK in 1991. This followed by "Moods 2" in 1992. The series focuses on the genres of new-age, ambient, world music, and to a lesser extent, downtempo, trip-hop and smooth jazz.
Doc Christina Lux, Alquimia and Gleisberg: and several compilations of meditative and world-influenee New Age music including "Musical Healing" in 2001 and "Perfect Balance" in 2006. In 2006, the Gordon brothers released "Soothing Sanctuary", ambient music with nature sounds, re-version of the earlier release "Sanctuary" (so stated in the liner notes) with new arrangements and remastering.
Doc 14: number that expanded to 200 as of March 2010 with the release of "Shaman", duo recording featuring Jean Derome and Joane Hétu. Presently DAME has a catalogue of over 450 titles. Ambiances Magnétiques began in 1982 when René Lussier, guitarist, and Jean Derome, who plays saxophone and flute, performed live as a duo in Montreal under the name Ambiances Magnétiques
(Total 40 Documents)
Response: Ambient
X
DynamicRAG
(Reranker Results:)
Doc 5: over the fact that the album was not released of May 2018, in the group's 2018 Mannheim Steamroller Christmas Tour announcement, founder Chip Davis announced that Exotic Spaces would be officially released upon the start of the tour, being sold at all concert locations as well on and Amazon.com. "Billboard's Top New Age Albums chart became the New Age Albums chart in June 2009. Mannheim Steamroller is an American Neoclassical new-age music group founded by Chip Davis that is known primarily for its "Fresh Aire" series of albums, which blend classical music with elements of
www.mannheinscaanollle..co
Neoclassical new-age music content: Neoclassical new-age music Within the broad movement of new-age music, neoclassical new-age music, or instrumental pop, is influenced by and sometimes also based upon early, baroque or classical music, especially in terms of melody and composition.
Doc 17:
Doc 3: to an increase CD sales and eventually awards with Westemn Canadian Music Awards (WCMA) (best dance) and Juno Award nomination (best instru- mental). Canadian success and a trip to MIDEM in France led to deal with Bay Area indie label XDOT25 which released the EP in the USA.
(Total 7 Documents)
Response: Mannheim Steamroller /
Ground Truth
Mannheim Steamroller
22

---

Table 11: Case Study for DynamicRAG. We compare our method with Vanilla-RAG with Reranker (monoT5). The initial retrieved content the same for both methods. We take Top-40 documents as the input.
Question
where is arachidonic acid found in the body
GPT-4o
Cell Membranes,
Brain,
Liver,
Platelets,
Eicosanoid Production Sites.
Vanilla-RAG
(Reranker Results:)
w/Reranker
Doc I: present in the phospholipids (especially ghoxphatidyllchhaokkminn, phos- phatidylcholine, and phosphatidylioositdee) of membranes of the body's cells, and is abundant in the brain, muscles, and liver. Skeletal muscle is especially active site of arachidonic acid retention, accounting for roughly 10-20% of the phospholipid fatty acid content typically. In addition to being involved in cellular signaling as a lipid second messenger involved in the regulation of signaling enzymes, such PLC-, PLC, and PKC, and isoforms, arachidonic acid is key inflammatory intermediate and can also act as vasodilator. (Note separate synthetic pathways, described in section below.)
Doe 3: Arachidonic acid Arachidonic acid (AA, sometimes ARA) is a polyunsat- urated omega-6 fatty acid 20:4(-6), 20:4(5,8,11, It is structurally related to the saturated arachidic acid found in cupuaçu butter ("L. arachis" peanut). In chemical structure, arachidonic acid is carboxylic acid with 20-carbon chain and four "cis"-donn" double bonds; the first double bond is located at the sixth carbon from the omega end. Some chemistry sources define árachidonic acidfo desig- nate any of the eicosatetraenoic acids. However, almost all writings in biology, medicine, and nutrition limit the term to all 1 Arachidonic acid a polyunsaturated fatty acid...
1.ei.. acid.
Doc 5: arachidonic acid supplementation for Alzheimer's patients are needed. Another study indicates that air pollution is the source of inflammation and arachidonic acid metabolites promote the inflammation to signal the immune system of the cell damage. Arachidonic acid is marketed as an anabolic body- building supplement in variety of products. Supplementation of arachidonic acid...
(Total 40 Documents)
Response: brain, muscles, and liver /
DynamieRAG
(Reranker Results:)
None
(Total 0 Documents)
Response: brain, muscles, liver /
Ground Truth
brain, muscles and liver.
23

---

Table 12: Prompt template for dynamic Reranker.
Dynamic Reranker Prompt
You are an expert at dynamically generating document identifiers to answer a given query.
will provide you with set of documents, each uniquely identified by a number within square brackets, c.g.. [1]. [2], etc.
Your task is to identify and generate only the identifiers of the documents that contain sufficient information answer the query.
Stop generating identifiers as soon as the selected documents collectively provide enough information to answer the query.
If no documents are required to answer the query, output "None".
Output the identifiers comma-separated list, e.g.. [1]. [2] "None" if documents are needed.
Focus solely on providing the identifiers. Do include any explanations, descriptions, or additional text.
Query: Question
Retrieved Content:
Title: title Content: content }
Top-N. Title: title Content: content
Table 13: Prompt template for retrieval-basee generator.
Retrieval-basee Generator Prompt
You are an intelligent assistant that uses retrieved knowledge answer user queries accurately and concisely. Follow these rules:
1. Task:
Use the provided [Retrieved Content] to generate responses.
If the Retrieved Content is None, you should generate an answer based on your own knowledge.
If the information is insufficient or you don't know the answer, state, "I cannot fully answer based on the available information. Please provide more details."
2. Requirements:
Accuracy: Base your answers on the retrieved content.
Conciseness: Keep answers brief and relevant.
Context Awareness: Ensure your responses align with the user's query.
3. Input Format:
Query: [User Query]
Retrieved: [Retrieved Content]]
4. Output Format:
structured, clear response tailored to the query.
Always prioritize clarity and reliability.
Query: / Question }
Retrieved Content:
Title: title J Content: content }
Top-K. Title: title Content: { content /
24

---

Table 14: Prompt template for our designed reward function.
Reward Function Prompt
Use the following criteria evaluate the quality of the model's response knowledge- intensive task, considering the provided ground-truth answer. Assign a score between 0-100 based on the overall quality, relevance, and correctness of the response:
1. Relevance to the Prompt (20 points):
Award up 20 points the response aligns well with the user's query, even minor errors are present. Deduct points if the response lacks focus or deviates significantly from the query.
2. Accuracy of Factual Information (20 points):
Grant up to 20 points for correct factual details aligning with the ground-truth answer. Penalize for inaccuracies, missing essential elements, presenting incorrect knowledge.
3. Handling of Temporal and Logical Reasoning (20 points):
Award up to 20 points for demonstrating correct temporal and logical reasoning. Deduct
points temporal reasoming is flawed or logical consistency is missing.
4. Clarity and Coherence of Response (20 points):
Assign up to 15 points for clear, coherent, and well-structured responses. Reduce points for ambiguity, confusion, poor organization.
5. Potential Misleading Nature or Misconceptions (20 points):
Award up 10 points if the response avoids being misleading. Penalize responses that could confuse or mislead the user, even if partially relevant. After evaluating the response based these criteria, provide total score in the format: "Score: points". Few-Shot Examples
User: Instruction
Ground-True Answer: { Answer Model Response: response
Table 15: Prompt template for GPT baselines.
GPT Baseline Prompt
Below is question, directly generate the answer. Your answer should be as concise as possible, which can be word phrase.
Question: { Instruction ) Response:
Table 16: Prompt template for Llama baselines.
Llama Baseline Prompt
Below is an instruction that describes a task.
Write response that appropriately completes the request. Paragraph: { Retrieved documents )
Question: Instruction }
Response:
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: answerYes
Justification:
The main claims can be found in the abstract and introdeuction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We acknowledge the limitations of our current approach, which the Limita- tion Section in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Comprehensive theoretical assumptions and complete proof' are presented in both the preliminary section and the methodology section.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our main experiments, ablation studies, and comprehensive analyses are detailed in the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the complete codebase for training and evaluation procedures, along with the associated datasets, the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Detailed descriptions our training and inference procedures are presented in the implementation details of the experimental section and in Appendix B.4.1. training details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include comprehensive statistical experiments, demonstrated in Sections 4.5.3 and 4.6.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide a comprehensive discussion of the computational resources utilized during training Appendix B.4.1, specifically in the training details subsection.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt.pp/e//arrrpp...///bbb//eeeee
Answer: [Yes]
Justification:
Our research adheres the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All components of this research are based exclusively on open-source models and datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All open-source models and datasets utilized in this work are properly cited at their respective usage locations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) http:p//eeerrrppccc//ooooeeeeeer :ctec///eeeeeeeee/////////////// for what should or should not be described.
32


Reviews:
[{"review_id": "8vLve03z4L", "reviewer": "Reviewer_VGiT", "summary": "DynamicRAG enhances a standard retriever by adding a dynamic reranker that adaptively selects both the order and the number kk of documents retrieved for each query. This reranker is initially trained via supervised fine-tuning to mimic expert behavior and then optimized through reinforcement learning using Direct Preference Optimization (DPO), treating the language model as the environment. The reward signal integrates multiple components, including Exact Match, BERTScore, ROUGE, length penalties, and LLM-based evaluations, with ablation studies revealing that Exact Match is especially influential. Using LLaMA-3-8B, the model achieves state-of-the-art results for its size, outperforming even GPT-4o in a non-retrieval setting.", "questions": "Which generator-independent reward signals could substitute for, or complement, Exact Match so that DynamicRAG remains reliable on tasks where the correct answer cannot be determined by straightforward token overlap?", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:** DynamicRAG adapts the passage budget *k* to the difficulty of each query, trimming irrelevant documents and spotlighting the most salient evidence. Its reinforcement-learning training optimises directly for answer quality, making the system more robust than fixed-threshold or static-score methods and less sensitive to hyperparameter tuning.\n\n**Weaknesses:** DynamicRAG’s reward is computed directly from the generator’s output, so the reward distribution itself shifts whenever the generator is replaced or its decoding parameters are altered. For instance, coupling the trained reranker with a stronger external LLM such as GPT-4o yields an immediate jump in EM score, yet attaining peak performance still demands RL fine-tuning tailored to that new pairing. This reveals that the current reward scheme is overly tied to a particular generator and lacks true model-agnostic generality. Moreover, although the framework nominally combines five reward components, Exact Match is overwhelmingly dominant: removing it lowers average performance by more than eleven points. Consequently, the system is effectively optimised for token-level answer overlap, and its robustness on tasks requiring long chain-of-thought reasoning, open-ended generation, or other settings where correctness cannot be judged by simple string matching remains unverified.", "was_revised": true, "final_justification": "Overall, I find the proposed approach to be practical and well-motivated. However, since my concerns have not been fully resolved, I am assigning a weak accept.", "rebuttal": ""}, {"review_id": "WlyXAXk5nY", "reviewer": "Reviewer_Mhs5", "summary": "This paper addresses a core limitation of traditional RAG rerankers—the use of a fixed number of retrieved documents (k), which often fails to balance the trade-off between information loss (when k is too small) and noise (when k is too large). To overcome this, the authors propose DynamicRAG, a novel framework that jointly optimizes both the number of retrieved documents and their ranking order. Built upon the SFT+DPO training paradigm, DynamicRAG significantly improves reranker performance, achieving higher generation quality with less training data compared to strong baselines.", "questions": "In addition to the concerns raised under Weaknesses, I have two further questions: \n1. A significant portion of the main experimental results appears to be missing. Is there a specific reason for this? \n2. What is the rationale behind the choice of hyper-parameters? Additionally, how sensitive is the method to these hyper-parameters? Asking because the cost of tuning them on different datasets could be substantial and non-negligible. Although Appendix C.3 lists three hyper-parameters, is there any theoretical or empirical justification for their selection?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. Clear motivation: The authors highlight that relying on a fixed number of retrieved documents (k) inherently fails to balance the trade-off between information loss (when k is too small) and noise introduction (when k is too large). This is a valuable and often overlooked insight in existing research.\n2. Targeted method design: The paper proposes a dynamic reranking mechanism built on a reinforcement learning framework, allowing the reranker to adaptively adjust both the number and order of retrieved documents based on feedback from LLM-generated outputs.\n3. Strong performance validation: Extensive experiments on seven knowledge-intensive datasets demonstrate that DynamicRAG achieves state-of-the-art performance on most benchmarks, while maintaining high efficiency and consistency across different backbone sizes. The comprehensive experimental setup and reproducibility of results further strengthen the credibility of the findings.\n\nWeaknesses：\n1. The core premise of the paper is that dynamic determination of k  enhances generation quality, yet the ablation experiments lack explicit validation of k's impact. For instance, fixing k during RL training to isolate its effect—e.g., comparing performance with static k vs. dynamic k —would better demonstrate the value of adaptive selection. \n2. The paper lacks a direct comparison with RankRAG, despite their similar focus and comparable performance. Though Appendix C.5 shows the trend of DynamicRAG’s performance, it does not substitute for direct comparative experiments—e.g., scaling DynamicRAG to 470k data or reducing RankRAG to 150k—to conclusively demonstrate efficiency advantages. \n3. Key definitions in the framework require clarification: \n(i) In Figure 3, the threshold condition \"s_i > T“ for document filtering lacks explanation—including how T is set and handling cases where no documents meet the threshold.\n(ii) The \"expert model\" used to score documents in Figure 3 is undefined. Whether it relies on human annotations, LLM evaluations, or another metric (e.g., MonoT5 reranking) and its reliability criteria are unspecified, leaving the training signal's validity ambiguous. These omissions obscure the framework's technical details for readers.", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "cmM0ZLan9o", "reviewer": "Reviewer_tviR", "summary": "This paper introduces DynamicRAG, a RAG framework where a reranker is treated as an RL agent that dynamically selects both the order and number of retrieved documents per query. In this way, the irrelevant information in RAG could be mitigated and thus improve the overall performance. The proposed method includes two stages. First, some expert trajectories are used to cold-start the reranker via behavior cloning. Then, the reranker refines its policy via DPO over multiple sampled trajectories, with rewards derived from a multi-dimensional evaluation of the LLM’s generated output. Experiments are conducted on 7 datasets, focusing on llama3 8B and llama2 13B LLM backbones, demonstrating good empirical results over baselines.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The two-stage training idea is straightforward to improve RAG systems with irrelevant noise. The ablation studies verify the effectiveness of each component in DynamicRAG\n- Leveraging LLMs' feedback as a supervision for reranking is interesting, and the use of DPO makes sense.\n- The empirical results look good, with many popular datasets being included.\n\n### Weaknesses\n- Relaying on relatively small LLMs (LLaMA2-7B, 13B, LLaMA3-8B) for evaluation limits generality to larger, more popular models.\n    - Have the authors tried DynamicRAG with larger LLMs (e.g., Llama-70B, GPT-4) or shown whether the proposed strategy still yields gains at scale?\n    - In addition, have the authors tried reasoning models? Do they also suffer from the issue caused by irrelevant info? Can the proposed method also be paired with reasoning models and improve their performance?\n- Engineering-heavy reward design (five weighted metrics) complicates reproducibility and may require extensive hyperparameter tuning.\n    - How sensitive is DynamicRAG to the choice of weights in Eq.7?\n- While Figure 4 highlights LLM call efficiency, what is the end-to-end training cost (compute hours) compared to static rerankers?", "was_revised": true, "final_justification": "My major concerns were on the experiments on larger models, training costs, and hyperparameter sensitivity. From the response to me, I think my concerns have been resolved.", "rebuttal": ""}, {"review_id": "UbzHNyK9P1", "reviewer": "Reviewer_mLW6", "summary": "This paper introduces DynamicRAG, a retrieval-augmented generation framework in which the reranker is treated as an agent that dynamically selects both the order and the number of retrieved documents for each query. Training proceeds in two phases: first, behavior cloning from expert trajectories provides a supervised initialization; second, reinforcement learning refines the reranker’s policy using multi-dimensional rewards derived from the quality of the LLM’s generated outputs.", "questions": "1. Have the authors considered applying DynamicRAG to tasks beyond question answering, and if so, what modifications might be necessary?\n2. How robust is performance to the relative weights of the five reward components?", "limitations": "See weakness above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of framing reranking as an interactive agent that learns from the generator’s feedback brings a principled integration of retrieval and generation objectives.\n2. The two-stage training strategy leverages behavior cloning to bootstrap the policy and reinforcement learning to optimize for downstream quality, balancing sample efficiency and task alignment.\n3. The reward design captures multiple facets of answer quality, from factual correctness to linguistic fluency, offering a rich supervisory signal.\n\nWeaknesses:\n1. The focus remains on question answering; it is unclear how readily the method extends to other retrieval-augmented generation tasks such as summarization or code generation.\n2. While the efficiency comparison is promising, end-to-end latency numbers under realistic deployment conditions are not reported.", "was_revised": true, "final_justification": "I have read the rebuttal, which has fully resolved my concerns. I decide to keep my score positive.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces DynamicRAG. The approach treats the RAG system as a RL agent. The system dynamically selects the optimal number (and order) of retrieved documents using the model response quality as a reward signal.

The reviewers noted the following strengths of the paper:
* The paper addresses a critical limitation (i.e., fixed choice of k) in modern RAG systems
* Framing the reranker as an RL agent that learns from e2e task performance is novel and well-motivated
* The method demonstrates SOTA across a wide range of datasets and model backbones, outperforming competitive baselines with less training data
* The list-wise reranking strategy is more efficient than commonly used point-wise reranking methods

The reviewers also identified a number of weakness with the paper, including:
* The initial version of the paper lacked a direct ablation study comparing a dynamic k strategy vs. a fixed k strategy.
* Reviewers initially raised concerns about the method's generalizability to SOTA closed models
* The complexity of the reward function raised questions about its robustness
* The initial focus on QA tasks made it unclear how well the framework might extend to other RAG applications

The authors' rebuttal was exemplary. It included new experiments that addressed the key weaknesses identified by the reviewers. This included a direct comparison between a dynamic and static k selection strategy, evals using SOTA frontier models like GPT-4o, and experiments on new tasks like slot-filling, all of which supported (and strengthened) the paper's claims. As a result of the strong rebuttal, two reviewers raised their scores and a third kept their already positive rating. This is a nice, well-executed paper that should be accepted for publication.
Decision: Accept (poster)