GRO-RAG: Gradient-aware Re-rank Optimization for Multi-source Retrieval-Augmented Generation
Authors: Siyuan Chen
Hang Ding
Kangxiaoyu
Jiechao Gao
Venue: ICLR 2026 Poster (2026)
Keywords: Retrieval-Augmented Generation
LLM
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Retrieval-Augmented Generation (RAG) systems often rely on information retrieved from heterogeneous sources to support generation tasks. However, existing approaches typically either aggregate all sources uniformly or statically select a single source, neglecting semantic complementarity. Moreover, they commonly employ re-ranking models to obtain Top-k documents, without accounting for actual contribution to generation objective.
In this paper, we propose GRO-RAG, a training-free, gradient-aware re-ranking framework for multi-source RAG. 
Our method performs Top-k document selection by reading gradients from the language model, estimating each document’s contribution to the generation loss through a single backward pass. 
This enables re-ranking not by heuristic relevance, but by direct feedback from LLM's generation objective. 
At the source level, we incorporate inter-source redundancy and query relevance to select source combination prior to re-ranking.
Theoretically, we prove that this gradient-based Top-k selection approximates the optimal subset minimizing the generation loss, and aligns with minimizing the leave-one-out loss upper bound.
Experiments across multi-source QA and open-domain generation tasks demonstrate consistent improvements in generation quality, highlighting the importance of generation-aware retrieval selection in multi-source RAG.

Published as a conference paper at ICLR 2026
GRO-RAG: GRADIENT-AWARE RE-RANK OPTIMIZA- TION FOR MULTI-SOURA RETRIEVAL-AUGMENTED GENERATION
Siyuan Chen
Ding Hang?
Xiaoyu Kang*
Jiechao Gao 4.t
1 University of Bristol
2 Shhnngaai Jiaotong University
IIntiitte of Information Engineering, Chinese Academy of Sciences
*Stanford University
ABSTRACT
Retrieval-Augmenter Generation (RAG) systems often rely on information re- trieved from heterogeneous sources to support generation tasks. However, existing approaches typically either aggregate all sources uniformly or statically select a single source, neglecting semantic complementarity. Moreover, they commonly employ re-ranking models obtain Top-k documents, without accounting for actual contribution to generation objective. In this paper, we propose GRO-RAG, training-free gradient-aware re-ranking framework for multi-source RAG. Our method performs Top-k document selection by reading gradients from the language model, estimating each document's contribution the generation loss through a single backward pass. This enables re-ranking not by heuristic relevance, but by direct feedback from LLM's generation objective. At the source level, we incorporate inter-source redundancy and query relevance to select source combi- nation prior to re-ranking. Theoretically, we prove that this gradient- based Top-k selection approximates the optimal subset minimizing the generation loss, and aligns with minimizing the leave-one-out loss upper bound. Experiments across multi-source QA and open- peen-domain generation tasks demonstrate consistent im- provements in generation quality, highlighting the importance of generation-awar retrieval selection in multi-source RAG.
INTRODUCTION
Retrieval-Augmented Generation (RAG) (Lewis et al. 2020 has emerged as a powerful paradigm for enhancing large language models (LLMs) by grounding their outputs in external knowledge. typical RAG pipeline (Chen et al.] [017] Das et al. 2019) first retrieves set of supporting documents from a corpus, and then conditions the generation process on both the query and the retrieved context. In practice, especially in open-domain and multi-hop settings, information is often distributed across multiple heterogeneous sources such as encyclopedias, web documents (Komeili et al., 2022; Dinan al. 2019), or community forums. This gives rise to the challenge of multi-source retrieval, where the system must identify not only relevant documents, but also determine which sources to trust, combine, ignore (Yan et al. 2024 Yao al. 2023 Wang et al. 2024 Zhao et al. 2024a).
By integrating multiple retrieval sources, Multi-Source RAG mitigates the capability ceiling limitation
inherent in Single-Source RAG. Recent studies (Yan et al. 2024 Yao et al. 2023 Wang et al. 2024
Zhao et al. 2024a) demonstrate that leveraging multiple retrieval sources dynamically and controllably improve retrieval accuracy, thereby enabling the generation of more comprehensive and high- quality mnwweedee-goondde responses. For instance, ReAct (Yao et al. (023) introduces an iterative multi-source retrieval approach, whereas UniMS-RAC (Wang al. (2024) uniquely integrates source selection, retrieval, and generation into a unified model enhanced with action and evaluation tokens,
"Equal contribution.
1 "ooresponniig author; j siecho@staanoordd

---

Published as a conference paper at ICLR 2026
Vanilla RAG
GRO-RAG
query
d2
da
42
V.
LLM
L
Retriever
Ranker
LLM
selection
Retriever
query
selection
Source
Source
Figure 1: Comparison between vanilla RAG and our proposed GRO-RAG. Vanilla RAG pipelines retrieve documents from certain source and rank them based on query-document relevance. GRO- RAG selects subset of sources that jointly balance query relevance and semantic diversity. And it re-ranks retrieved documents using a single backward pass over the generation loss, estimating each document's contribution via the alignment between its representation and the loss gradient.
enabling the language model to dynamically invoke and filter sources based on real-time demands. Additionally, PrefRAG (Zhao al.) 2024a) advances RAG further by employing a preference- driven adaptive retrieval mechanism coupled with self-reflection, supporting in-depth and controlled exploration across diverse retrieval sources. Despite these advances, most existing RAGs treat source-level retrieval in simplistic manner-either aggregating all sources uniformly or statically selecting single source-thereby neglecting the semantic diversity and redundancy inherent across sources. Furthermore, even after document retrieval, many systems (e.g., BM25 (Robertson et al.) 2009)) rely on re-ranking models that score documents solely based on retrieval- signals such as -occmmd similarity, without considering their actual utility to the downstream generation objective. This creates a mismatch between what is retrieved and what is ultimately needed for high-quality generation.
To address these limitations, we propose GRO-RAG, a training- freee optimization framework tailored for Multi-source
Generation (MS-RAG) Yaan et al. 2024 Yao et al. 2023 Wang et al. 2024 Zhao et al.) 2024a). GRO-RAG introduces a principled mechanism to dynamically select both which source combinations to retrieve from" and "which documents
as context"
Firstly, we select source combinations by optimizing a relevance-redundancy tradeoff that jointly considers query relevance and inter-source semantic overlap, rather than uniformly aggregating or statically selecting sources. This allows GRO-RAG to identify a diverse yet relevant set of sources that provide complementary information, improving both recall and retrieval quality at the source level. Subsequently, we introduce a gradient-aware document re- re-ranking strategy that directly leverages the feedback from large language model. Specifically, after retrieving candidate documents from the selected sources, we perform forward pass to compute the generation loss and then a single backward pass to obtain the gradient of this loss with respect to the input representations. For each document, we compute importance score the inner product between its hidden representation and the loss gradient. This score reflects how much the document contributes to reducing the generation loss. Selecting the Top-k documents based these scores enables posterior- aware re-ranking process that is tightly aligned with the final generation objective, rather than heuristic query-document similarity. GRO-RAG is fully training-free and compatible with frozen language models. It introduces no additional model parameters or training phases, and requires only one forward-hackwardd pass per query. Theoretically, we show that the gradient-based Top-k selection approximates the solution to an underlying utility maximization problem and aligns with minimizing a leave-one-out loss upper bound.
Our main contributions are summarized as follows:
We introduce novel, training-free free approach that estimates each document's contribution to the generation loss using single backward pass, enabling posterior- aware Top-k selection based model feedback.
2

---

Published as a conference paper at ICLR 2026
We formulate source selection relevance-redunde tradeoff over source subsets, abling dynamic source combination and improving retrieval diversity and complementarity.
We theoretically show that our gradient praddent-basedd selection approximates an underlying utility maximization objective and aligns with minimizing a leave-one-out loss upper bound. Exten- sive experiments on multi-source QA and open-domain generation benchmarks demonstrate consistent improvements over strong retrieval and re-ranking baselines.
RELATED WORKS
2.1
RETRIEVAL-AUGMENTTR GENERATION
eetrieval-augmentee generation (RAG) is increasingly recognized effective method to mitigate
several limitations of large language models (LLMs) (Wu et al. [025 Tang et al.) (023), notably
hallucinations (Shuster et al. 2021), factuality issues (Wang et al. 22023 and the lack of loog-term
memory (Xu et al., 2022). Following the established "Retriever-and-Ree Reader" paradigm (Chen et al.
2017 Das al. 2019), RAG first employs an external retriever to select relevant textual information from knowledge sources(e.g, Wikipedia). These retrieved passages subsequently serve as external context for a reader/generrator, enabling the model generate knowledge grounded response (Lewis et al. 2020). Initial retrieval methods (e.g., BM25 (Robertson et al. 2009)) used sparse retriever for relevance scoring, but often fail to capture deeper semantic information (Guo al.) 2022). To dense retrieval approaches have been developed,
overcome this limitation, Banguage-model-basee encoding documents and queries into dense vectors to effectively represent the semantic feature of text content (Karpukhin et al. 2020 Li et al.] [023 Bruch et al. (2023). Recently, researchers have explored leveraging LLMs as retrievers (aang et al. [2024] Asai et al. 2023 Jiang et al. 2023 Yu cet al. 2024| Zhao et al.) 2024a). For instance, Self-RAG introduces reflection tokens, enabling the
model to dynamically retrieve supporting passages and self-correct its outputs (Asai et al. 2023)
2.2 MULTT--OURCC RAG
Based on the retrieval sources, recent advances in retrieval -wwprrll--ggmmeeeddd generation (RAG) can be categorized into Single-Source RAG (SS-RAG) (Asai et al. 2023) Jiang al., 2023) Yu et al. 2024
Cui et al. 2023) and Multi-Source RAG (MS-RAG). SS-RAG methods inherently limit RAG system performance due to reliance single knowledge source. MS-RAG addresses this limitation by integrating heterogeneous knowledge sources, including specialized databases, structured archives, and the open web (Yan al., 2024). Common MS-RAG implementations perform sequential or parallel retrieval across multiple indices. For example, CRAG treats the web as a fallback source Yan al. 2024), while ReAct coordinates retrieval and reasoning within an agent- based framework ( Yao et al. 2023). However, naively concatenating evidence from diverse sources can enlarge context windows and introduce noise or conflicting information, thus necessitating adaptive source selection (Wang et al., 2024 Zhao al.) 2024a). UniMS-RAC RAG [Wang et al.) (024) addresses these issues by unifying source selection, retrieval, and generation into single scquence-to-sequenee lo-sequence model. introduces specific action and evaluation tokens, allowing the LLM to dynamically invoke and filter sources as needed. PrefRAG (Zhao et al. (024a) further advances MS-RAG with preference-die adaptive retrieval approach that employs self-reflection, enabling more in-depth and controllable exploration across multiple retrieval sources.
3 METHOD
3.1 TASK DEFINITION AND NOTATION
Let Q be the space of user queries and A' the space of target answers. We are given a query q E Q and seek generate answer a" with the assistance of external evidence. Evidence is organized in a set of heterogeneous sources S = (81, ... 81 Each source E S exposes a (possibly dynamic) document collection D. = {ds.1..... | where a document d a sequence of tokens from the vocabulary V. Our goal is to choose both (i) subset of sources A C S and (ii) size-k document set (*) S U.e D, that minimizes the generation loss. A frozen Transformer (Vaswani et al. 2017) language model Mo receives the query q together with small context set D* and produces
3

---

Published as a conference paper at ICLR 2026
Source Combination Selection
Gradient-aware Re-ranking
dc
Ranking by
dπ
V.
d3
h2
d
2
2
ht
d
h4
d
TL
L
S3
LLM
Source
Retriever
Candidate
Figure 2: Model architecture of GRO-RAG
best answer a". Because enumerating all subsets is infeasible, the following two subsections describe tractable, training-free freee procedure-fise first selecting sources, then ranking documents with gradient feedback-the approximates the optimum of generation objective.
3.2 SOURCE COMBINATION SELECTION
Suppose geography query asks "Which European river flows through both Vienna and Bratislava?". news feed, Wikipedia, and a travel blog may each contain statements answering the question. If we indiscriminately merge all three sources the model will read many redundant sentences about the Danube, wasting context budget. Conversely, if we keep only Wikipedia, we indeed cover the Danube, but we throw away the travel-blog anecdote that might help the model answer follow-up questions (e.g. travel time, boat service). Our goal is therefore to retain sources that bring new information while discarding those that merely repeat what has already been covered.
Selecting which source combinations to retrieve from is crucial in multi-source RAG (Wang et al. (024|/Zhao al.) 2024a): querying too many sources balloons the candidate pool with near- duplicupls,,sss. whereas restricting to single source squanders complementary evidence that may be indispensable for multi-hop reasoning. We formalize this decision as small yet expressive serevance-redundane optimization and solve it with greedy algorithm that enjoys provable approximation ratio.
Our first step is to choose a subset of sources A C S with the given query g. For each source s S we invoke fixed recall engine and obtain the top-mn candidates (g) !d,,,..... da. q 9enc (4) is the query embedding of A source representation Rd is formed by averaging the frozen document embedding of its candidates:
8 = m
d
ds.j
Jone (ds.j)-
==1
We define scoring function
f: P over subsets of sources for a candidate subset ACS as:
f(A:)) Σ cos( q.
x Σ cos(s,
where the first term measures query relevance and the second quantifies inter-source redundancy, and e (0, 1) trades off relevance and redundancy. The relevance term favors sources containing content aligned with the query semanties; the redundancy term penalizes sources whose content is highly similar one another. can be read as reward the marginally informative, punish the already-covered.
Any sparse or dense retriever is admissible; the choice does not affect downstrcam optimization.
We centence-BERT: other encoders yield similar behavior.
4

---

Published as a conference paper at ICLR 2026
We now turn the sub-modularity of equation equation The first term is modular by construction, as it is a sum of independent contributions. The redundancy term is pairwise and symmetric, and under cosine similarity, it induces uub-modular structure due to inrrassing overlap with more sources. Since the sum of modular and a sub-modular function remains sub-modular, the full objective f is submodular (Carbonell & Goldstein 1998) when À is small. This structure allows
employ greedy algorithm for subset selection with provable approximation guarantee. We initialize Ao Ø. For = O,.... |S] - we compute the marginal gain (s = ((A A) f(As; A) for every SES and set A++1 farg max, (s)) The loop stops once |Ar+1l = [S] or no positive gain exists.
We denote the final set by Agreedy, classical results on submodular (Carbonell & Goldstein 1998) maximization imply the following guarantee:
fAAgredyy: A) 2 (1 - 1/e)
max ASS
((A
The union of their candidate sets (4) UseAmmac (g) serves the input to the gradient-aware re-ranking stage. This two-level selection retains complementary evidence while sharply reducing cross-source redundancy.
3.3
GRADIENT-AWARE RE-RANKING
Our source selection stage has already removed most irrelevant duplicated corpora, leaving a mixed candidate pool C²(g)) U.EA (g(g) = {d1,-... ..... dn }. The final bottleneck is the frozen language model's context limit: can absorb most passages. The usual remedy is to train a cross-encoder or to reuse query gucry-document similarity scores, but both approaches ignore how the generator itself reacts to each passage. GRO-RAG therefore lets the LLM "vote": we ask, "If I boost passage i, will my loss decrease?" and we obtain the answer from single hrtward-bacwwerr pass.
For each candidate document d₃ E (q) we already possess the frozen contextual embedding h;- Selecting exactly k passages is combinatorial. We relax the binary choice (keep / drop) to soft non-negative weight vector π E A" (T, E [0, 1), || || = 1), we write the mixture representation h(=) = Σ== nh; Next, we construct soft prompt (q. h(x)). and compute the generation loss [(T) L(a" q, h(x)) with respect the reference answer a'. Although this relaxation converts the discrete Top-k selection into a continuous problem, the resulting loss function 2(-) remains non-convex and analytically intractable due to the non-linear behavior of the generator. To obtain a tractable approximation, we apply a first-order Taylor expansion around the uniform mixture (I/n..... 1/n). This yields:
((T) R C(R) + V,C.M(M) m()))
Considering ((T) = EE n, and T((TT) = 1 Σi= h; we obtain:
[(T) R ((T) + Σ (V,CCh,,
(5)
Minimizing the loss is therefore approximately equivalent minimizing a linear weighted sum over document scores. Since π is constrained to lie in a k-sparse simplex, the approximate optimal solution obtained by selecting the documents with the largest negative inner products (h,, - -V.)) We define the ranking score of document d; accordingly:
6i (h., -v
(6)
This value estimates the sensitivity of the generation loss the presence of de in the prompt, and serves as posterior-aware signal for document selection. The larger di. the more sharply the loss would drop if document received additional weight. We keep the documents with the highest di set all other weights to zero, and re-normalize. The selected passages D* (q) are finally pre- pre-pended to the query. In practice we simply concatenate their raw text, but one could alternatively keep the mixed vector h to save context tokens. That is, only need one newward-backward pass sort the candidate documents. This strategy requires no additional training, and leverages L.M--nttrnal gradients to estimate document utility with respect to the actual generation objective, not just similarity.
natural way to measure how much a single document d, actually helps the generator is to remove from the context, run the model again, and observe how much the loss increases. We call this
5

---

Published as a conference paper at ICLR 2026
leave-one-out (LOO) loss. Formally, starting from the uniform mixture म (2..... ... 1 over all candidates, we define
(de)
C ee
all documents
document removed
where e; is the i-th basis vector. Computing this quantity for every passage would require n n++ forward passes paassss-rroobbitiee slow. Below we show that single backward pass yields a gradient score &e that upper-bounds LOO loss, thereby providing safe ranking surrogate.
Proposition 3.1 (Gradient inner product upper-bounds leave-one-out) Let = Vi.(m) and Φi hi,-g)- If the one-dimensional function l((t) A( + tes) is convex on the interval t E n for some >> O, then for every documents d;
(d.) S -Ø..
Thus Φi upper-bounds the true marginal utility, so ranking by is guaranteed prioritize passages whose absence would hurt the loss the most.
The gradient-based scoring described above is single-step heward-backward pass: we linearize the loss landscape around a uniform mixture म = (2...., 1)) and select the top-k documents accordingly. Moreover, we can extend this process into multi-step optimization routine. At each iteration we maintain soft weight vector n², use it to form the context h(T²) Σ. n/,; and compute generation loss and its gradient a forward-backwar g' (V,((((T)) which projected back onto the k-sparse simplex to maintain feasibility. In effect, this defines an iterative refinement process over document mixtures, allowing the model itself guide the selection toward increasingly informative subsets. Each iteration re-evaluates the loss under the current document mixture and refines the weights accordingly no new model parameters introduced, and the only computation is an additional aewward-backward pass of the frozen LLM.
pass. This yields the descent direction to update x' via gradient descent. The updated weights are then
Proposition 3.2 (Linear convergence of the iterative loop) Let f(T)) = L(d' q, (,,(()) denote
the generation loss evaluated at h(x)) = E i= T,,;, Assume p-strong convexity and T-smoothhess of C the sub-space span (h,,.... hn) Starting from the uniform vector के (1/n,.... 1/m). repeat
π
2+1
VV L(TT),
-1- IIa-- (=
0 O<nS
where MA--i onto the probability simplex A = 2 || = 1}. Then for all 0:
C( 1( L s - (1) [I(TT)- -*
with optimal C min L(T). Hence each additional iteration contracts the sub-optinality by the factor (1 (u) and therefore never worsens the one-step solution. A proof is provided in Appendix B
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Dataset Following previous works Yao et al.) 2023 Trivedi et al. 2022a Zhao et al. 2024b), We evaluate our method on four widely used question answering benchmarks that span both open-domain and multi-hop reasoning settings: HotpotQA (Yang et al. 2018). 2WikiMulthhpQAA (Ho et al. 2020), and MuSiQue (Trivedi et al., 2022b). These datasets each provide set of ground-truth documents (typically 10-20) for each question as well as ground-trute answers.
Metries We assess model performance from both retrieval and generation perspectives. For retrieval evaluation,
nDCG A (Normalized Discounted Cumulative Gain) (Burges al. 2005), which measures the ranking quality of selected documents based on graded relevance and position. A higher nDCG indicates that relevant documents are ranked closer to the top. Ground-truth relevance annotations where standard QA metrics (Gao et al. 2023): Exact Match (EM), which measures the proportion of
where available- are used as supervision. For generation evaluation, we adopt two

---

Published as a conference paper at ICLR 2026
Table Results (%) of GRO-RAG and baselines three datasets, "Bold" and "Underlined" denote the highest absolute values and second highest values, respectively.
HotpotQA
2WikimQA
MuSiQue
LLM
RAG methods
EM
F1
EM
EM
w/ Retrieval
27.8
23.1
19.7
13.9
8.4
3.5
Local
34.2
28.2
24.1
19.4
13.1
8.4
Vanilla RAG
Web
31.5
24.8
20.4
15.3
10.5
5.1
Both
36.0
29.7
27.3
21.8
15.9
9.2
Llama3. 1-8B
Self-RAG
32.3
26.4
21.1
17.4
14.8
8.9
Single-Source RAG
FLARE
34.5
28.6
28.5
23.0
17.3
10.7
RankRAG
31.9
24.3
25.7
20.8
13.6
7.9
CRAG
34.2
25.5
22.6
17.9
16.2
9.2
Multi-Source RAG
GRO-RAG
39.1
30.9
28.9
22.8
18.6
10.3
w/ Retrieval
29.4
23.6
18.6
13.5
10.3
4.1
Local
36.8
29.8
25.3
20.1
13.0
8.2
Vanilla RAG
Web
30.4
23.9
19.5
14.8
9.4
4.3
Both
39.3
31.5
28.2
22.4
16.5
9.6
GLM-4
Self-RAG
34.4
28.3
22.8
18.3
17.8
10.5
Single-Source RAG
FLARE
38.6
30.7
29.7
23.8
20.2
11.6
RankRAG
33.2
27.3
27.4
21.6
15.8
8.8
CRAG
38.1
30.3
24.8
20.4
17.4
9.6
Multi-Source RAG
GRO-RAG
42.8
33.6
30.3
23.7
21.1
12.4
generated answers that match ground-truth strings exactly, and F1 score, which captures the overlap between predicted and reference answers based on precision and recall. Retrieval metrics are computed over the Top-k ranked documents, while generation metrics reported over answers produced by the language model conditioned on the selected document set. Here, we set l = 10.
Retrieval settings To support multi-source retrieval, we leverage both local corpora and web sources. Specifically, for web search, we employ the DuckDuckGo API-a publicly available interface- to access large-scale online information. We compare with three representative retrievers: BM25 (Robertson al. 22009, sparse keyword-baase method; E5-base (Wang al., (022). dense dual-encoder trained via contrastive learning; and BGE-M3 (Chen et al. 2024). dense multilingual and multitask-aligned retriever.
Generation settings We compare GRO-RAG against four categories of baselines. (1) w/o-Retrieval:
LLM directly answers questions without access to any external documents. (2) Vanilla RAG:
Standard RAG methods that retrieve documents from either local corpus, a web corpus, or both, and concatenate them with query input to the LLM. (3) Single-source RAG: Methods such as Self-RAG (Asai al. 2023 FLARE (Jiang et al. 2023), and RankRAG Yu al. 2024), which rely a single retrieval source (e.g., local corpus only). y. (4) Multi-source RAG includes CRAG Yan et al.) 2024) conduct one-time retrieval from the primary source, followed by a one-time supplementary retrieval from secondary source. For all methods, we conduct experiments based
built-in Meta AI 2024) and GLM4 (GLM et al.
LLMs, including I.lama311-8 1-8B (Grattafiori et al. 2024
2024
4.2
MAIN RESULTS
Performance Generation Table presents comprehensive comparison across three represen- tative QA -ottarrkrssotttAAAA.... 2WikiMQA, and MuSiQue- under both LLaMA3.1-881 and GLM-4 language models. We begin by observing the general trend that retrieval substantially boosts performance over generation-only settings. Across all datasets and models, methods with access to external documents consistently outperform the No-Retrieval baseline, confirming the necessity of knowledge augmentation for multi-hop reasoning tasks. Among vanilla strategies, retrieving from the local corpus tends to be more effective than the web corpus alone, likely due to better domain alignment and lower noise. Concatenating both further improves accuracy, suggesting local and web

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of NDCG@ 10 for different re-ranking methods on three QA datasets using only local corpus. GRO-RAG uses gradient-basse based re-ranking from frozen LLMs (LLaMA3-8B and GLM-4), while BM25, BGE-M3, and E5-base serve as heuristic or dense retrieval baselines.
Different rerankers
HotpotQA
2WikimQA
MuSiQue
Average
BM25
0.6237
0.5760
0.3453
0.5150
BGE-M3
0.6892
0.6273
0.3922
0.5696
E5-base
0. 0.7013
0.6749
0.4180
0.5981
GRO-RAG with Llama3. 1-BB
0.6442
0.6345
0.4039
0,5609
GRO-RAG with GLM-4
0.6538
0.6382
0.4156
0.5692
sources offer complementary coverage. However, simply merging top documents from multiple sources is suboptimal. Vanilla RAG (Both) and CRAG, while better than single-source retrieval, rely on static combination rules and lack an understanding of which sources are truly informative. GRO-RAG addresses this through a source combination module that jointly considers query rele- vance and semantic redundancy, selecting a subset of sources that are diverse and overlappa....gg. This leads to improved retrieval precision while avoiding wasted context on wear-duplicate content. Compared to adaptive single-source methods such as FLARE and Self-RAG, GRO-RAG consistently achieves stronger performance across all datasets. These methods typically depend on fallback heuristics or self-generated queries, which may help in certain cases but do not explicitly model the final generation objective. In contrast, GRO-RAC performs gradient- based re-ranking directly using the LLM's generation loss, enabling the model to "vote" on which passages are most useful via single backward pass. This strategy aligns document selection tightly with the downstream objective and proves especially effective under tight context budgets. On more challenging datasets such MuSiQue, which feature higher document entropy and require more subtle reasoning, the advantage of generation-aware selection becomes even more evident. GRO-RAG not only main- tains high accuracy but also demonstrates greater stability across LLMs of varying capacities. For instance, while many baselines suffer performance drops when moving from GLM-4 to the smaller LLaMA3. 1-1.....1.,, 1-8B, GRO-RAG's relative improvements remain consistent, highlighting its robustness and model-agnostic nature. Lastly, we emphasize that GRO-RAG achieves these gains without any fine-tuning or additional model parameters. All retrieval and scoring steps are conducted with frozen models, making GRO-RAG readily deployable in practical settings.
Performance of Retrieval To ensure fair comparison and isolate the impact of re-ranking, we constrain retrieval to single local corpus and use identical candidate pools for all methods in single source setting. The results are shown in Tab We compare GRO-RAG against three common baselines: BM25, sparse term-matching method; BGE-M3, modern dense retriever trained with contrastive supervision; and E5-base, strong general-purpose embedding model. GRO-RAG uses training-free re-ranking approach that computes gradient-based importance scores from the frozen language model. Despite not using any retrieval supervision or document-query similarity learning, GRO-RAG consistently improves over BM25 and performs on par with close to strong dense retrievers. In particular, on the most difficult dataset (MuSiQue), GRO-RAG achieves higher nDCG than BGE-M3, suggesting that gradient signals from the generation objective capture nuanced relevance signals beyond static embeddings. These results demonstrate that even without additional training, GRO-RAG can effectively identify useful documents for the LLM, narrowing the gap to supervised retrievers and providing à principled, efficient alternative for ranking in retrieval- augmented generation. We emphasize that GRO-RAG is not designed to be standalone retriever; its goal is not to maximize retrieval metrics, but to identify documents most useful for generation. In subsequent experiments, we will show that this strategy leads consistent improvements in final answer accuracy with minimal computational overhead.
4.3 IN-DEPTH ANALYSIS
Ablation study To examine the contribution of each component in GRO-RAG, we conduct an ablation study by removing (1) the Source Combination Selection (SCS) module and (2) the Gradient- aware Re-ranking (GR) module. The results are summarized in Table Removing SCS means retrieving documents independently from all sources without optimizing for sevvance-redundange
8

---

Published as a conference paper at ICLR 2026
Table Ablation study of GRO-RAG, Removing souree selection (w/o SCS) or gradient re-ranking (w/o GR) leads to performance degradation across models and datasets.
HotpotQA
2WikimQA
MuSiQue
LLM
Methods
F1
EM
F1
EM
F1
EM
GRO-RAG
39.1
30.9
28.9
22.8
18.6
10.3
Llama3. 1-8B
w/o SCS
38.0
30.6
26.4
21.3
17.0
10.2
w/o GR
37.5
30.2
23.3
19.6
16.2
9.3
GRO-RAG
42.8
33.6
30.3
23.7
21.1
12.4
GLM-4
w/o SCS
40.1
31.4
28.6
22.5
20,0
11.5
w/o GR
37.6
28.7
25.3
20.9
16.8
9.4
0.83
1p4
1xp4
0.0
Dotoooa Mu-id
0.6
039
lick
020
044
045
049
043
1500
Dutnel
Daienl
(a) Llama3.1-8B
(b) GLM-4
(c) Llama3.1-8B
(d) GLM-4
Figure 3: (a)(b) show the effect of optimization steps on retrieval performance INDCG@10); (c)d)
show the effect of Top-k values on generation performance (FI score).
tradeoff. While this variant still benefits from having access to multiple corpora, it introduces redundant or semantically overlapping documents into the context, leading to modest but consistent performance drop across datasets. This suggests that carefully selecting a compact and complementary set of sources helps improve information coverage while minimizing wasteful input. On the other hand, removing GR has a more pronounced effect. This variant replaces gradient-based scoring with a naive per-document loss, ignoring how documents interact when presented jointly. As result, the model is more likely to prioritize individually informative but contextually redundant documents. Performance declines most noticeably on complex datasets like MuSiQue, indicating that gradient-infomme scores better reflect generation utility under constrained context windows.
Multi-step optimization Figures 3a and 3b illustrate the NDCG@ 10 of GRO-RAG under multi- step optimization three QA datasets using LLaMA3.1 and GLM-4 as backbone LLMs, respectively. In both settings, the curves show consistent and steady improvements with an increasing number of optimization steps. Importantly, the gain in each step diminishes at a near-constant rate, forming a clear linear improvement pattern. This trend empirically supports the theoretical expectation of linear convergence, where the optimization error decreases proportionally across iterations. The convergence behavior is stable across datasets and model scales, with the total gain around (3-5%), validating the efficiency and robustness of GRO-RAG's gradient-based refinement process.
Impact of different We investigate how varying the number of retrieved documents (k) affects the final answer quality. As shown in Figures 3e and 3d increasing & does not always lead to better performance. While a larger l provides more information. may also introduce irrelevant or noisy content, which can interfere with the generation process. We observe that F1 scores generally improve from 3 to & 5, but further increases sometimes yield diminishing or even negative returns. This highlights importance of carefully selecting & to balance completeness and relevance in RAG.
Computation times Computing a single-step gradient using LLaMA3.1-8P takes 882ms on aver- age, while dense retrievers like BGE-M3 complete retrieval in just 39ms. Compared to traditional retrievers, gradient-based retrieval with GRO-RAG introduces significantly higher -tticalytypically 1 to orders of magnitude slower. This motivates our design choice of using GRO-RA -RO--AAG- as a re-ranker, where the initial retrieval already provides strong prior. In such cases, one gradient step is needed to refine the document selection, making the additional computation cost manageable while still improving answer quality.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, we introduced GRO-RAG, training-free and gradient-aware framework for document re-ranking in multi-soure Rerrieval-Augmented Generation. By directly leveraging generation loss gradients, GRO-RAG identifies Top-k documents that most effectively contribute to the model's output, moving beyond traditional query-based relevance scoring. the source level, our approach selects complementary sources by jointly modeling query relevance and inter- redundancy. Theoretically, we show that our re-ranking objective approximates the solution to a loss minimizing subset selection problem and aligns with minimizing a leave-one-out upper bound.
REFERENCES
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self self-reftection. In The Twelfth International Conference Learning Representations, 2023.
Sebastian Bruch, Siyu Gai, and Amir Ingber. An analysis of fusion functions for hybrid retrieval.
ACM Transactions on Information Systems, 42(1):1-35, 2023, doi: 10.1145/3596512.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pp. 99, 2005.
Jaime Carbonell and Jade Goldstein. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 335-336. ACM, 1998.
doi:
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1870-1879, Vancouver, Canada, 2017. Association for Computational Linguisties.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi culti-fuuctioonality multi granularrtylrrttyy text embeddings through self-knowledge distillation. arXiv preprint arXiv:240..03216, 2024.
Chao Cui, Shisong Tang, Fan Li, Jiechao Gao, and Hechang Chen. Calibrating video watch-time predictions with credible prototype alignment. In Forty-second International Conference Machine Learning, 2025.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. Multi-step retriever- reader interaction for scalable open-domain question answering. In 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. Proceedings of the 7th International Conference Learning Representations (ICLR), 2019.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint 2rXiv:2312.10991 2:1, 2023.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-1300 to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The la herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
10

---

Published as a conference paper at ICLR 2026
Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, and Xueqi Cheng. Semantic models for the first-stage retrieval: A comprchensive review. ACM Transactions on Information Systems, 40(4):1-42, 2022. doi: 10.1 10.1145/3486200.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint 2.10100 2020.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969-7992, 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781, 2020.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8460-8478, Dublin, Ireland, 2022. Association for Computational Linguisties.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. retrieval-augment genera- tion for knowledge- www.ldggiinnensiveee nlp tasks, Advances neural information processing systems, 33: 9459-9474, 2020.
Hang Li, Ahmed Mourad, Shengyao Zhuang, Bevan Koopman, and Guido Zuccon. Pseudo rele- vance feedback with deep language models and dense retrievers: Successes and pitfalls. ACM Transactions on Information Systems, (1(3):62:1622:444 2023. doi: 10. 10.1145/3570724
Meta AI. Llama 3 model card, 2024.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval. 3(4):333-389 2009.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguis- tics: EMNLP 2021, pp. 3784-3803, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.1 N0.18653/v1/2000 1t533/////222...nnnnneeee/////// findings-emnlp.320.
Shisong Tang, Qing Li, Dingmin Wang, Ci Gao, Wentao Xiao, Dan Zhao, Yong Jiang, Qian Ma. and Aoyang Zhang. Counterfactual video recommendation for duration debiasing. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4894-4903, 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint 22t::2212.10000 2022a.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pp. 5998-6008, Curran Associates, Inc., 2017.
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue. Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao. Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang. Jindong Wang. Xing Xie, Zheng Zhang, and Yue Zhang. Survey on factuality in large language models: Knowledge, retrieval and domain-speciffeiry. 2023.
Hongru Wang, Wenyu Huang, Yang Deng. Rui Wang, Zezhong Wang, Yufei Wang, Fei Mi, Jeff Z Pan, and Kam-Fai Wong. Unims-rag: A unified multi-source retrieval- auggenttd generation for personalized dialogue systems. arXiv preprint 2r01 2024.

---

Published as a conference paper at ICLR 2026
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-superrve contrastive pre-training. arXiv preprint arXiv:2212.03531, 2022.
Binrui Wu, Shisong Tang, Fan Li, Bing Han, Chang Meng, Jingyu Xiao, and Jiechao Gao. Aligning and balancing and multimodal representations for recommendation. In Proceedings of the 31st ACM SIGKDD Conference Knowledge Discovery and Data Mining V. 2, pp. 5029-5038, 2025.
Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng- Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. Long time no see! open-domain conversation with long-term persona memory. In Findings of the Association for Computational Linguistics: ACL 2022. pp. 2639-2650, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10. 18533v1/20022.../2 (ttp:/00222.ninnngaac...22.))
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Hua Ling. Corrective retrieval augmented generation, 2024. URL htt.:ttp///aaxv..or///// http://aa//222111.1/
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint w 0900009600, 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with aemrental-aggmentee generation llms. Advances in Neural Information Processing Systems, 37:121156-121 184, 2024.
Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, and Jie Tang. Prefrag: Preference-driven multi-source retrieval-angmenned generation. arXiv preprint 14.06924,,.0068, 2024a.
Qingfei Zhao, Ruobing Wang. Xin Wang, Daren Zha, and Nan Mu. Towards multi-source retrieval- augmented generation via synergizing reasoning and preference-driven retrieval. arXiv preprint arXiv:2411.00689 2024b.
A PROOF FOR PROPOSITION 3.1
VnL(a) and 6, (h,,-g) the one-dimensional function lx(t) [(( + te;) is convex on the interval t t [-] +, E] for some O, then for every documents d,
Proposition A.1 (Gradient inner product upper-bounds leave-one-out) Let g
(ioo(di) < -di.
Express LOO as a directional difference. From the definition of h(T)) =E, nyhy;
H( eee)) h(()) - 1 h..
Hence L100 (d.) (((1
Apply convexity of l. For a convex function f.f(v) 2 f())+(()())=))) Setting x 0, y - gives
4(-1) 2 400) + (-))
Rearrange. Subtract the right-hand side from (,))):
()) f( ((-1) 4) < (()).
Convert the directional derivative. Chain rule yields (,(0) = (h,, Vi.(T)) = (ba.g). Substitute and flip sign. Therefore
(o) < (b., < -
The factor cancels because it is positive and identical for all passages, leaving the desired inequality.
12

---

Published as a conference paper at ICLR 2026
B PROOF FOR PROPOSITION 3.2
Proposition B.1 (Linear convergence of the iterative loop) Let C(T) = C(at | q, q[T)) denote
the generation loss evaluated at h(x)) = Ei= TzH; Assume p-strong convexity and L-smoothness of L in the sub-space H. span ( (hi,..., h. Starling from the uniform vector (1/n, 1/n), repeal
5 # मK x(T')
n²' IIA--1(**11))
0 o<
where A projects onto the probability simplex A = 2 | = 1}. Then for all 2 0:
(( L S (1 m/4 -
with optimal L -in L(m). Hence each additional iteration contracts the sub-optimality by the factor (1 TJJ)) and therefore never worsens the one-step solution.
Proof:
We have:
A n-1 { E R" n. 2 O, ||]|| = 1}, I1a= = arg
min || x 0112.
(8)
TEA n-1
For any iterate nO define the mixed-context hidden vector h(TT) = Ei मे. h, and G V,
Because (((T) M with M [hi,.., hn.. the chain rule gives
G = MM V (
(9)
T-smooth on H. hence for any n. u R.:
L 10 -
[(v) < [(T) (V.((T), T)
(10)
2
Apply equation 10 with n¹ and v म 1- ग¹ n
Li²
( (G Gy) || C(a') 11111
1G
(11)
Let n² be a minimizer of L on A n- Using the Pythagorean property of Euclidean projection,
|| S -|
(12)
Because L is p-strongly convex,
((T) - L(xx) |T x 111
VT A n-1
(13)
Combine equation] 12 and equation 13 to get
((1 [(TT) & ((1(1) -(x²))
+1)
(14)
Another consequence of strong convexity
||G' [1 > 2μ [[(m') ((]
(15)
Insert equation 15 into the descent lemma equation and use < 7 5 1/L= 2
C(# (1)) S L(T') mo -
(16)
Subtract L(**) from both sides and combine with equation 14
((a L(xx') < (1 m))) [C(=²) -(x==((=)))))) This proves the claimed inequality with L = L(x*). Iterating the contraction yields
(17)
L(*)) L' (1 (1- [((T) - C*],
(18)
which is geometric linear convergence with rate Tp (< 1).
13

---

Published as a conference paper at ICLR 2026
C LIMITATIONS
While GRO-RAG offers a lightweight and training-free alternative for improving document selection in multi-source RAG, it also presents several limitations. First, the gradient-based re-ranking procedure requires a backward pass through the frozen LLM, which, although computationally modest compared fine-tuning, still introduces latency compared to purely retrieval-side methods. Second, GRO-RAC 's ranking relies on local linear approximations of the generation loss, which may become inaccurate when the true loss landscape is highly non-linear or when documents exhibit strong interaction effects. Finally, our source selection module is limited to fixed document embeddings; future extensions could explore query-aware or adaptive source encoders further enhance selection.
D BROADER IMPACTS
GRO-RAG contributes to the growing field retrieval-augmentie generation by proposing more interpretable, modular, and training-free alternative to dense scoring models. Its emphasis gradient- based document utility aligns document selection directly with the language model's generation behavior, offering greater transparency and controllability in the retrieval pipeline. By avoiding task-specific fine-tuning, GRO-RAG lowers the resource barriers to deploying RAG systems in practical applications, particularly in low-resource privacy-sensitive settings.
14


Reviews:
[{"review_id": "KtasN58N4Z", "reviewer": "Reviewer_HXps", "summary": "The paper backs this with theory (a leave-one-out upper bound and a simple iterative refinement view) and shows the setup is training-free for frozen LLMs. It first picks which sources to use by balancing query relevance with cross-source redundancy so you don’t haul in a pile of near-duplicates. It then ranks candidate passages by how much each would drop the LLM’s loss, using one forward-backward pass and an inner-product score.", "questions": "1. How sensitive is source selection to the redundancy weight and to the encoder choice used to summarize each source\n\n2. Is there a simple recipe for the number of refinement steps that balances marginal gains against extra backward passes.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Ranking is tied to the generator’s actual objective instead of just query similarity, which is exactly what RAG needs. \n2. The source router uses a clean relevance-diversity trade-off with a greedy submodular pick that has a standard approximation guarantee. \n3. It slots into frozen models with no extra training and keeps the compute to a single backward pass per query.", "weaknesses": "1. A backward pass per query adds real latency compared to retriever-only pipelines, so the deployment fit needs clearer bounds.\n\n2. The submodularity claim relies on a small trade-off weight and cosine redundancy, which may not hold up uniformly across domains.\n\n3. Source selection builds on fixed document embeddings rather than a query-adaptive encoder, which can undercut routing on niche topics", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YZlKlch0Oz", "reviewer": "Reviewer_Kxfs", "summary": "The paper proposes a training-free re-ranking framework that selects sources by a relevance–redundancy tradeoff and then ranks documents by how strongly each aligns with the LLM’s generation-loss gradient. The re-ranking score is the inner product between a passage embedding and the loss gradient, which approximates loss reduction under a first-order expansion and yields Top-k selection in one backward pass. A theoretical result upper-bounds leave-one-out utility by the gradient score and a second result shows linear convergence for an optional iterative mixture update under smooth-strongly convex assumptions. Experiments on HotpotQA, 2WikiMultihopQA, and MuSiQue with Llama-3-8B and GLM-4 report consistent EM/F1 gains over vanilla, single-source, and multi-source baselines.", "questions": "(1)How stable are gradient scores across decoding temperatures or prompt formats and do these scores correlate with true leave-one-out gains on held-out queries. \n\n(2)What is the end-to-end latency profile broken down into retrieval, gradient computation, and generation, including tail percentiles under web-augmented scenarios.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The gradient-aware scorer ties ranking directly to the generator’s objective rather than query similarity, which better matches end-task utility. \n\n(2) The method is training-free and uses a single forward–backward pass per query, so it plugs into frozen LLMs without extra parameters. \n\n(3) As a re-ranker over a fixed pool, it approaches dense retrievers in NDCG and surpasses them on the hardest dataset (MuSiQue) in at least one backbone configuration.", "weaknesses": "(1) The linearization around a uniform mixture can misestimate interactions among passages when the loss landscape is highly non-linear. \n\n(2) The convergence guarantee assumes smooth-strongly convex behavior in the span of passage embeddings, which is unlikely to hold strictly for modern LLMs. \n\n(3) Latency increases materially because a backward pass is required, which is orders of magnitude slower than dense retrieval and may constrain interactive use. \n\n(4) The limitations section acknowledges dependence on local linear approximations and fixed document encoders, which could cap performance in complex settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ZtMTm4NpP", "reviewer": "Reviewer_S2EC", "summary": "This paper presents **GRO-RAG**, a gradient-aware and training-free framework for improving document selection in multi-source Retrieval-Augmented Generation (RAG). The method addresses a key limitation of existing RAG systems that rely on heuristic or similarity-based document ranking, which often fails to reflect how retrieved passages affect the downstream generation objective. GRO-RAG introduces a two-stage approach: first, a source combination selection module balances query relevance and inter-source redundancy through submodular optimization; second, a gradient-based re-ranking mechanism estimates each document’s contribution to generation loss using a single backward pass through a frozen language model.  \n\nThe framework is theoretically grounded. The authors show that gradient-derived importance scores approximate a leave-one-out loss bound and empirically validate the approach on multi-hop QA benchmarks. The method consistently outperforms standard RAG and multi-source baselines without requiring fine-tuning or additional parameters. The overall goal is to align retrieval selection more directly with the LLM’s generation behavior in a model-agnostic and computationally efficient manner.", "questions": "1. Could the authors provide more intuition about how the gradient-based re-ranking signal behaves in practice? For example, how should one interpret documents with small or negative gradient alignments in terms of their contribution to generation quality?  \n2. The paper mentions that a backward pass adds latency but remains training-free. Can the authors contextualize this cost relative to other optimization-free enhancements such as reflection-based or reranking-only pipelines?  \n3. How was the relevance–redundancy trade-off parameter λ chosen? Is it generally robust, or does it require tuning across datasets?  \n4. The theoretical analysis relies on convexity and smoothness assumptions that may not hold strictly for transformer-based models. Could the authors clarify how these assumptions should be viewed, primarily as guiding intuition or as formal guarantees?  \n5. The gradient linearization is a key design choice. Could the authors discuss under what conditions it might fail, and how practitioners should interpret its limitations when applying GRO-RAG in different settings?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work provides an elegant and principled perspective on retrieval optimization by connecting document ranking directly to the generation loss. \n\n* The central idea of using LLM gradients as feedback for re-ranking is conceptually novel yet intuitive, bridging the gap between retrieval relevance and downstream utility. \n\n* The proposed method remains entirely training-free, introducing no additional parameters and operating seamlessly with frozen LLMs, which makes it appealing for practical deployment in resource-limited or privacy-sensitive contexts.  \n\n Overall, the study introduces a solid and thought-provoking idea that advances the understanding of generation-aware retrieval.", "weaknesses": "* While the core contribution is technically sound, the evaluation and discussion could be broader. All experiments focus on question answering, leaving uncertainty about how well the method generalizes to other forms of retrieval-augmented generation such as summarization or dialogue where grounding is less explicit. Although the method itself is task-agnostic, the paper would benefit from more explicit reasoning about its broader applicability.  \n\n* From a practical standpoint, the backward pass required for each query may raise latency concerns in large-scale applications. Although the discussion briefly mentions this issue, a clearer contextualization of the trade-off between computational cost and quality improvement would make the work stronger.  \n\n* The gradient linearization assumption may also be optimistic given the non-linear nature of LLM loss landscapes. The theoretical discussion on convexity and smoothness provides a mathematical foundation but may not fully hold for transformer architectures. Clarifying how these assumptions should be interpreted in practice would improve the overall balance between theory and application.  \n\nIn summary, the idea is creative and meaningful, but the empirical scope and certain discussions fall slightly short of the depth expected for ICLR acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4pkVojt7C", "reviewer": "Reviewer_qyTo", "summary": "The authors propose GRO-RAG, a training-free framework for improving multi-source RAG by integrating a gradient-aware re-ranking mechanism. The framework selects a subset of sources by optimizing for the minimization of redundancy, and performs a re-ranking of retrieved documents based on their contribution to the final generation output. The method is evaluated across several multi-hop QA datasets, showing consistent improvements over several baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method introduces a gradient-based re-ranking approach backed by solid theoretical analysis, with proofs showing that the gradient-based selection approximates the optimal utility-maximizing solution, providing strong guarantees for its effectiveness.\n2. GRO-RAG is training-free,    making it practical for deployment in existing systems without requiring additional model training.\n3. The experiments and ablation studies show consistent performance improvements.", "weaknesses": "1.  The gradient-based relevance estimation relies on the first-order Taylor expansion (Eq. 5), implicitly assuming local smoothness of the loss landscape. The paper does not provide quantitative evidence on the approximation error when the loss function exhibits strong nonlinearity.\n\n2.Proposition 3.2 shows linear convergence under strong-convexity and L-smoothness assumptions. However, the generation loss of a frozen LLM is generally non-convex. It remains unclear whether monotonic improvement still holds empirically when these assumptions are violated.\n\n3.The scoring function (Eq. 6) assumes additive document contributions, while semantically dependent passages may only be useful jointly. The treatment of such higher-order interaction effects is not addressed or evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)