RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models
Authors: Jiarui Zhang
Xiangyu Liu
Yong Hu
Chaoyue Niu
Fan Wu
Guihai Chen
Venue: NeurIPS 2025 poster ()
Keywords: LLM Routing
Retrieval-Augmented Generation
Large Language Models
Primary area: deep_learning

Abstract: Retrieval-Augmented Generation (RAG) significantly improves the performance of Large Language Models (LLMs) on knowledge-intensive tasks. However, varying response quality across LLMs under RAG necessitates intelligent routing mechanisms, which select the most suitable model for each query from multiple retrieval-augmented LLMs via a dedicated router model. We observe that external documents dynamically affect LLMs' ability to answer queries, while existing routing methods, which rely on static parametric knowledge representations, exhibit suboptimal performance in RAG scenarios. To address this, we formally define the new retrieval-augmented LLM routing problem, incorporating the influence of retrieved documents into the routing framework. We propose RAGRouter, a RAG-aware routing design, which leverages document embeddings and RAG capability embeddings with contrastive learning to capture knowledge representation shifts and enable informed routing decisions. Extensive experiments on diverse knowledge-intensive tasks and retrieval settings, covering open and closed-source LLMs, show that RAGRouter outperforms the best individual LLM and existing routing methods. With an extended score-threshold-based mechanism, it also achieves strong performance-efficiency trade-offs under low-latency constraints. The code and data are available at https://github.com/OwwO99/RAGRouter.

RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models
Jiarui Zhang'
Xiangyu Liu?
Yong Hu?
Chaoyue Niu
Fan Wu
Guihai Chen
Shanghai Jiao Tong University
Weehat,, Tencent Inc
Abstract
Retrieval-Augmennte Generation (RAG) significantly improves the performance of Large Language Models (LLMs) on knowledge-intensive tasks. However, vary- ing response quality across LLMs under RAG necessitates intelligent routing mechanisms, which select the most suitable model for each query from multiple retrieval-augmente LLMs via dedicated router model. We observe that external documents dynamically affect LLMs' ability answer queries, while existing rout- ing methods, which rely on static parametric knowledge representations, exhibit suboptimal performance in RAG scenarios. To address this, we formally define the new retrieval- augmented LLM routing problem, incorporating the influence retrieved documents into routing framework. We propose RAGRouter, a RAG-aware routing design, which leverages document embeddings and RAG capa- bility embeddings with contrastive learning to capture knowledge representation shifts and enable informed routing decisions. Extensive experiments on diverse knowledge-intensss intensive tasks and retrieval settings, covering open and closed-source LLMs. show that RAGRouter outperforms the best individual LLM and existing mechanism, also
routing methods. With an extended wore-threshold-i achieves strong performance-efficieney trade-offs under low-latency constraints.
Introduction
The rapid advancement of large language models (LLMs) has led to an increasingly diverse model landscape, with significant heterogeneity in parametric knowledge stemming from variations in training data, architectures, and learning objectives [45]
35 36
57 53). However, LLMS
remain limited by outdated knowledge, hallucinations, and insufficient domain coverage [17] 22 [29]. Eetrieval-Augment Generation (RAG) [27 addresses these issues by injecting external knowledge at inference time, effectively reducing hallucinations and improving performance
knowledge-intensivg tasks 5
20 41)
While RAG enhances LLM performance by incorporating external knowledge, different LLMs exhibit substantial variation in their ability to utilize retrieved content. Prior studies 66 [T] show that, given identical documents, LLMs differ in information extraction, integration, and robustness to noise-refle inherent heterogeneity in RAG capabilities stemming from differences in ar- chitecture, training data, and optimization. Such diversity suggests that combining multiple models can yield complementary strengths, enabling performance that surpasses any single model. LLM routing, which pre-selects the most suitable model for each query from pool of LLMs without invoking them all, offers an efficient and effective fusion strategy [9]. Under dual heterogeneity in parametric knowledge and RAG capability, intelligent query routing presents promising direction for leveraging RAG to achieve superior performance, motivating the central question: How can we effectively route queries to the most capable LLM under the RAG paradigm?
Chaoyuu Niu the corresponding author (rvince @sw.eeeusssu....n........
"The code and data are available at tattps://githun cww wwtmm//ww099//AAooor..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

PA
Amram In the Book of Exodus, Amram the husband inchebed and father of Auron, Moses and Mirim. Alternative spallings of the name include. in addition being marned
1515
Retriesal
10.0
Ganvation
Who the fathor Moses?
Amram
Aaron
Route
O
Router
Jacob
Amram
i0cco Bo00 $1.00 40.00
3000 0.00 2006 40:30 00.00
$3.00
10000
Rodinggsccecell (w/o RAG)
Rodinn Foiled (w RAG)
Figure 1: Left: Accuracy of various LLMs on the PopQA task before and after RAG. Right: An example query where retrieved documents improve Qwen's 's response (unanswerable - answerable) but impair Llama's s (answerable -> unanswerable), illustrating how existing routing methods fail under RAG due to their inability to capture such dynamic shifts.
Current multi-model routing methods primarily match queries to LLMs based on their inherent parametric knowledge [40 [99 31 43
38
10.
7 33 58 [2]. Several approaches 7 33 58 [2]
construct compact vector representations of LLMs to enable efficient query-model compatibility estimation. These methods all assume static knowledge representations for non-RAG scenarios.
However, these approaches face critical limitations in RAG settings, as they fail to account for the dynamic impact of knowledge injection. As illustrated in Figure 1 RAG dramatically shifts the distribution of response quality across input sveries-exterie documents can reverse a model's ability to answer question, rendering routing strategies designed for non-RAG scenarios obsolete. The core issue lies the Static Knowledge Assumption: existing approaches assume fixed LLM knowledge, ignoring how retrieved content dynamically reshapes their capabilities. In practice RAG response quality depends on the interplay between a model's internal knowledge and external information. This leads to shortcomings: Missing Doc Interaction-exisei queries and model embeddings, overlooking document features and their interaction with models; and Ignoring RAG Capability- -pior work captures only static knowledge differences, neglecting LLMs' differing ability to leverage documents. These gaps highlight shortcomings of current LLM routing strategies- they fail to adapt the dynamic relationship between LLM and external knowledge.
methods focus on
To tackle this issue, we propose RAGRouter, a contrastive learning- based routing framework that explicitly models knowledge shifts in RAG scenarios. RAGRouter is designed to route queries across LLMs by modeling key factors that affect post-retrieval performance. At the architecture level, RAGRouter incorporates a document encoder and cross encoder to capture document semantics and query interactions, thereby addressing missing document interaction, and assigns each LLM a RAG capability embedding a learnable vector representing its proficiency in utilizing retrieved con- tent-to mitigate ignoring RAG capability. However, directly optimizing such a router is challenging due to inherent variations introduced by retrieval. To address this, at the optimization level, employ a contrastive learning objective, where positive and negative samples-iag i.e., representations of LLMs that correctly incorrectly respond to a query- are drawn from both Cross Pross-Setting (between Ron and RAG settings) and Intra-Setting (within each setting). Taking the query representation an anchor, the objective encourages alignment between answerable model-query pairs while pushing apart unanswerable ones. This allows RAGRouter to effectively model atrrevall-nducee behavior shifts, moving beyond the static knowledge assumption.
We evaluate RAGRouter on a suite of Beowledge-intensive tasks 13 34 25 2 23 and retrieval
settings. Experimental results show that RAGRouter surpasses the performance of the best individual LLM, highlighting its ability to leverage the complementary strengths of multiple models in retrieval- augmented scenarios. Furthermore, RAGRouter substantially outperforms existing non-RAG-aware routing methods, validating the effectiveness of modeling retrieval-induce knowledge shifts.
Our main contributions are summarized as follows: (i) To the best of our knowledge, this is the first work exploring LLM routing in the RAG setting; (ii) We propose RAGRouter, contrastive learning- based routing mechanism that is aware of knowledge shifts, incorporating RAG capability and document-aware representations to effectively address the failure modes of existing routing strategies in RAG; (iii We validate the effectiveness of our method on five knowledge-intensive tasks

---

under local and online retrieval settings, using open goun-s LLMs such as the Qwen and LLaMA series scales from 0.5B to 72B and closed-source LLMs including GPT-4o. Qwen2.5-Max, and DeepSeek-R1. R1. Results show that RAGRouter outperforms the best individual LLM and existing non-RAG-awarr routing methods by
(iv) We apply an extended score-threshold-oee based mechanism to RAGRouter, and results show that its accuracy-latency curve generally lies above those of all baselines, indicating superior performance--fficienny trade-offs under low-latency constraints.
2 Related Work
Auggeented Generation. RAG enhances language models by integrating retrieved in- formation from external databases [27 15]. It typically follows round of retrieval and sequential generation pipelines, where documents are retrieved based on the input query and concatenated with it for generation. Prior work has improved RAG by optimizing retrieval components [56] 51 42
enhancing the generator's ability to utilize retrieved content [21 49 48). Recent studies highlight heterogeneous LM capabilities in processing external information, both in utilizing retrieved content
and tolerating retrieval noise I 39]. These heterogeneous capabilities reveal optimiza- tion opportunities for ensemble approaches that strategically leverage multiple LLMs within RAG scenarios. In this work, we study the routing problem under the RAG setting.
28 6
LLM Routing, Existing LLM routing approaches 7 10 12 33
58 31 primarily focus non -ARG
settings, where routing relies solely on the input query and each model's parametric knowledge, without incorporating external retrieved documents. For example, RouterDC 7 uses dual contrastive learning to model query-model compatibility, while EmbedLLM (58) and RLu 33 apply matrix factorization to learn compact model embeddings for scalable routing. GraphRouter [12] constructs a heterogeneous graph with nodes for tasks, queries, and LLMs, and encodes their interactions as edges capture contextual alignment between query needs and model capabilities. However, in RAG scenarios, retrieved documents induce dynamic shifts in model knowledge, which existing methods overlook. In contrast, our proposed RAGRouter models both the documents and LLMs' RAG capabilities, enabling more effective routing under retrieval- strieval-augmeented settings.
3 Problem Formulation
RAG enhances LLMs by integrating external knowledge through two-stage process: given a query q. the retriever Ret(D, q) selects relevant documents from an external corpus D, and the model M (g. d) generates a response based on both the query and the documents d, i.e y = My, Ret(D, 4)).
We formulate a LLM routing problem under RAG setting. Let M {M1, (M, MN } be set of candidate LLMs. A routing policy R Q x D - {I,..., N} selects the most suitable model M MR(g.d) for each input pair (g, d). To evaluate response quality, we define an oracle scoring function a (M;, q, d) {0, 1). where a (M; q, d) 1 if the response from M, given q and d matches the reference answer y" Importantly, using fixed model can be suboptimal, as different LLMs excel on maximize the expected routing performance:
different query-document pairs. The objective
max Eg [( 9, q.4]]
R
Notably, when no external documents are available (i.e., d = 0), the LLM routing problem under RAG setting naturally degenerates into the conventional L.LM routing problem. In this setting, the routing policy simplifies to R : Q + (1,.... N}. and the oracle scoring function becomes a (M, q). which assesses the response based solely on the query. The objective becomes:
max R
Eax [a [o( 4)]
(2)
Thus, the conventional routing problem can be seen as a special case of LLM routing under the RAG setting, corresponding to the boundary condition where = 0.
4 RAGRouter
4.1
Routing Model Architecture Design
We establish a conceptual framework by constructing an intuitive explanation of knowledge rep- resentation and LLM-query matching under RAG and non-RAG settings. In non-RAG settings
3

---

7, each LLM is typically associated with compact knowledge representation vector E Rdiim which implicitly reflects its parametric knowledge; and meanwhile, query is encoded as vo E Ii representing the knowledge needed to answer it. A proxy metric, like similarity sim(ng, Uk) is then used gauge the LLM'' s ability to respond, guiding non-RAG routing process.
However, in RAG settings, the LLM is augmented with retrieved documents that provide non- parametric knowledge. This additional information influences the model's response generation, rendering the original knowledge representation insufficient. The effective knowledge of the LLM shifts due to the integration of external information, resulting in a new representation:
1/4
+ vf
where by is the fused knowledge representation derived from the documents. Consequently, in RAG scenarios, the similarity between the query and the updated knowledge representation sim(u, vi)) should serve as the new routing criterion, as it more accurately reflects the model's ability to respond.
RAGRouter is designed with this insight in mind and explicitly models the fused knowledge vy to dynamically update the LLM's knowledge rep- resentation. We identify three core factors that influence vy: (1) the non-parametric knowledge provided by the documents; (2) the LLM's ability to process external information, including knowl- edge extraction and robustness to noise; and (3) the query's role in guiding knowledge retrieval. Based on these, RAGRoueer consists of the fol- lowing modules, with its architecture illustrated in Figure
(b) Knowledge Update
(c) Similarity-Bassed Routing
S 00
Fy
8 emilenty
LUM Inowledde Erebediding Lager
RAG Capabiitty Emebedring Laper
Cross Enceder
Decament Eecoder
Querr Erooder
Representing Parametric Knowledge. To ob- tain the original parametric knowledge represen- tation uk we introduce the LLM Knowledge Em- bedding Layer Φx. which takes the LLM ID M and outputs us = Φx (M), capturing inter-model variability in parametric knowledge, For query representation, we employ a Query Encoder ΦQ- which encodes the query q as 04 = @Q(().
M
b DO
LINS
(a) Input Embedding
Figure 2: The inference pipeline of RAGRouter: (a) Encode query, document, cross interaction, LLM knowledge, and RAG capability; (b) Fuse RAG capability, document, and cross embeddings to update knowledge representation; (c) Route based on similarity with the query embedding.
Representing RAG-Aware Factors. To com- pute the fused knowledge vy. RAGRouter inte- grates signals from three perspectives. First, the non-parametric knowledge provided by the docu- ments is captured by the Document Encoder ΦD. which encodes a document d into 2y = dpp (D(d). (d). In practice, the document and query encoders share parameters ensure consistency in the embedding space. Second, the LLM's ability to process external information is captured by the RAG Capa- bility Embedding Layer ΦR. which maps each candidate model M to embedding v,- = ()) representing its intrinsic capacity to utilize retrieved evidence. Third, the query's role in guiding knowledge retrieval is represented by the Cross Encoder Φc, which processes the query-document pair (d, q) to produce an interaction representation
éc q).
Representation Update for Similarity- Based Routing. The fused knowledge representation By is then derived via a multi-head attention mechanism that integrates these signals:
ff Attention(vr, Ud, Ve)
With the fused knowledge computed, the RAG-aware knowledge representation of the model becomes v/g = Ui + vy. The final routing decision based on the similarity between the query and the updated knowledge representations of candidate models:
R(g, d) = arg max
{sim( )
(5)
ie (T......
This formulation allows the routing policy to explicitly account for knowledge shifts introduced by document retrieval, thus maintaining accurate assessment of each LLM's ability in RAG settings.

---

Cross Setting Contrast Intra-Settin Contrast
Pull Close
RAG Setting
Away
Who plays Mary Jane in spiderman 2?
Non-RAG Setting
(a) Construct Positive and Negative Samples for One Query
(b) Contrastive Learning
Figure 3: (a) CSC constructs positive and negative samples under different settings based on response quality (e.g., Llama RAG (V) vs. Llama w/o RAG (X)), while ISC constructs them under the same setting (e.g., Llama w/ RAG (V) vs. Qwen w/ RAG (X)); (b) By combining CSC and ISC, contrastive learning pulls positive samples closer to the query representation and pushes negative ones away.
4.2 Optimization
In RAG settings, the incorporation of retrieved documents often leads to significant changes in LLM answerability- L.LMs become able to answer queries they previously could not, while others fail after retrieval. These shifts in answerability, effectively label transitions, reflect corresponding changes in the model's knowledge representation. Such transitions naturally yield structured positive and negative pairs across different knowledge states. This setting aligns well with the principles of contrastive learning [8. 18], which is particularly well-suited for capturing and optimizing the knowledge representation shifts induced by external knowledge injection in RAGRouter.
To this end. we design the Cross-Setting Contrast (CSC) mechanism to model representation differences between the non-RAG and RAG settings, and introduce the Intra-Setting Contrast (ISC) mechanism to model representation differences within the same setting. As shown in Figure 3 using the query representation ty as the anchor, CSC constructs positive and negative samples by selecting knowledge representations with different response qualities from the non-RAG and RAG settings (blue arrows). ISC, on the other hand, selects positive and negative samples from models with different response qualities within the same setting (orange arrows). This enables CSC help RAGRouter distinguish between different knowledge transfer patterns induced by documents, while ISC enhances the model's s discriminative ability across LLMs within same setting.
Combining CSC and ISC, we construct comprehensive set of positive and negative samples to train the RAGRouter. For given query q. we define the positive and negative sets follows:
V+ (1k, (M, 9) (v IMM, d. 1}
(6)
V_ = {ux. o q) = 03 u (0% a(Mj, d, g) = 0}
The corresponding contrastive loss is defined as:
exp(sim(L,, Uk+)/T)
(CT(O)=
Σ llog exp(sim(Ug. )/T) E' exp(sim(Vg uk- 1/T)
VV+
where is temperature hyperparameter. This loss encourages the query embedding e, to be closer to positive samples and further from negative ones, enabling the learning of representations that sensitive to both knowledge shifts and model heterogeneity. When retrieved documents alter model's 's response ability-e.g. from unanswerable to answerable- the mechanism captures these dynamic transitions, enhancing routing accuracy and knowledge adaptability in RAGRouter.
To further enhance LLM discrimination, we introduce a binary classification loss. For the original model M, define MM. Sigmoid sim(v,, vy)); for the RAG-enhanced model M', define s
Sigmoid (sim(1) )))) . Let UM.g = a (M, q) and yM'.g = a (M, d, q) be the ground-truth labels. The
classification loss is:
LCLS(g) MEMUMM
Σ
[yM.q log SM. + (1 yM.q) log(1 sM.g)l
(8)
The total loss is the weighted sum of the contrastive loss and classification loss, with À > 0 as balancing hyperparameter:
C(g) Lcr(g) + XLCLs(g)
(9)
5

---

4.3 Latency-Aware Extended Design
While RAGRouter does not explicitly model LLM's latency, outputs a relevance score for each can- didate LLM given a query, which can be exploited to support flexible trade-offs between performance and efficiency. To this end, we introduce a score- www.tthhrehhhbbeeeeee -bssed routing mechanism. Concretely, we pre-sort the N available LLMs [M1 M2, ................................ My] based on their prior efficiency profiles, such as smaller parameter sizes and lower latency meaning that M1 is the most efficient and My the least.
Given a query, suppose M; receives the highest predicted score from RAGRouter (i.e., it is the ieformancc-opiimaae model). Instead of routing directly M,, we traverse the list from Mj to M; and select the first LLM M, whose score satisfies $M.. $ I 6, where @ is a user-defined score margin threshold. This mechanism sacrifices a small amount of accuracy for significantly improved efficiency, making RAGRouter adaptable to latenny-constrained or essource-limiter scenarios.
Experiments
5.1 Experimental Setup
Datasets. We select queries from five different wowwlede-inttnsivy tasks: (i) PopQA [32 is an open-domain gunstion-answering benchmark covering diverse factual topies from broad knowledge domains; (ii) MedMCQA [34] is a multiple-choice benchmark focused on biomedical knowledge
and clinical reasoning: (iii) Natural Questions (NQ) 25 is an open-domain benchmark based on real-world search queries requiring span-level answer retrieval from Wikipedia; (iv) WebQuestions (WebQ) 2 is a knowledge base-driven benchmark grounded in Freebase relations, designed to evaluate entity-centric factual reasoning: and (v) TriviaQA (TQA) [33 is an open-domain benchmark centered on factoid-style questions sourced from trivia enthusiasts and web documents. Following 42.. we adopt Cover Exact Match the evaluation metric for PopQA, NQ, WebQ, and TriviaQA. Further data processing details and dataset statistics are summarized in Appendix A.2
Candidate LLMs. We selected 15 mainstream LLMs with parameter size ranging from 0.5B to 72B, Comprehensive statistics on model scales and latency are presented in Table and implemen- tation details are provided in Appendix A.1
Table 1: Statistics of different LLMs and their latency.
[52
14
44
55
LLM
Latency (ms)
Qwen2.5-0.55 5-0t-l-tetB
0.494
24.54
Llama-3.2 1iiaa-3.-1B-Instnen
1.240
20.47
Qwen2.5-1.5B..0 Ins-irrrttt
1 1,500
24.79
Retrieval Settings. Following 42 we adopt both local and online retrieval strategies for PopQA and MedMCQA reflect realistic RAG scenarios. Local retrieval uses the 2018 English Wikipedia dump [24] with BGE-large-en-v1.5 (50) as the dense retriever. On- line retrieval leverages the DuckDuckGo Web Search API 9l to access up-to-date external content. For NQ, WebQ, and TriviaQA, we follow [II and construct re- trieval contexts from Wikipedia passages augmented with synthetic noise (e.g., irrelevant distractors, coun- terfactual noise) to simulate imperfect retrieval. This setting enables evaluate the effectiveness of the routing model under noisy conditions.
gemma-2-2b-it
2.614
31.80
Mlama-3.2-3B-Insttne
3:213
81.82
Qwen2.5-30-Instme
3,000
24.39
Yi-1--66B-hht
6.061
142.67
Qwen2.5.7B-Ineee I-Istuutt
7,616
80.83
Ministral 8B SB-Instruct-2410
8.020
26.13
Jama-3. Metn-IJamm-3. 1----.1-----ssm-m-
Metn-I
8.030
177.37
Vi-1.5-9B-hha
8.829
199.61
Qwen2.5.14BB Instrurttuctt
14.770
175.42
Bwen2.5-32B-Insmmen
32,764
156.26
Dwenn.5-72BB. Instruct
72,706
1610.00
Llama 3.3. 70B Instruct
70.554
1970.00
Baselines. We compare RAGRouter against range of baselines. Existing routing methods were not RAG-aware and exploited only query-LLM compatibility, ignoring the impact of retrieval augmentation. This series of baselines include Prompt LLM [2]. which employs GPT-4o for model selection via meta-prompts: GraphRouter [12,, which models queries, tasks, and LLMss in a heterogeneous graph; RouterDC [7]. which aligns query-LLM embeddings through contrastive leamiig; KNN Router, which [99 relies on historical performance of similar queries; and Matrix Factorization (MF) [33 (58) which reconstructs LLM correctness patterns via low-rank latent spaces. We also introduce some rule-based routing methods, including Single Fixed LLM for all queries; Oracle Single Best that ideally selects the best-performing single L.LM per dataset; Random L.M
b Latency refers to the average time taken by an LLM to complete single query, including both inference time and potential network delays. https: http::://ddcxcccco.oooom

---

Table 2: Performance comparison of RAGRouter with rule-based and non-RAG -on-RAG-aware baselines across different knowledge- intenssvee tasks and retrieval settings. Testing accuracy (%) reported. "Ret." indicates whether the method is retrieval-aware (v) or not (X). The best results are shown in bold, and the second-best are underlined.
PopQA
MedMCQA
Method
Ret.
NQ
WQ
TQA
Avg
Local
Online
Local
Online
Wwen2.5-0.5B-Instuue
45.19
51.11
25.93
34.81
27.08
38.75
39.17
37.43
1amma-32--1B-Instan
39.26
46.67
17.78
36.67
25.42
31.67
43.33
34.40
Qwen2.5-1.55 1.5.-...tttt
44.44
48.89
30.00
42.59
30.00
35.42
46.67
39.72
gemma-2-2b-it
41.11
50.74
20.37
37.04
22.92
27.92
40.83
34.42
1lama-3.2-3B-Instte
45.56
51.48
27.41
44.81
36.67
45,42
65.00
45.19
Qwen2.5-3B-Instten
41.11
47.41
40.37
49.26
30.42
36,25
53.33
42.59
Yi-1.5-6B-Chr
46.67
51.48
31.11
40.00
35,83
43.33
56.67
43.58
Qwen2.5577B-nntrruu
42.96
48.15
35.93
43.33
29.58
35.42
55.00
41.48
Ministral-8B IB-Inntruct-2411
41.48
46.30
50.74
62.22
38.33
42.08
63.75
49.27
Meta-I.lama-3.1-88
46.67
51.85
41.85
52.59
39.58
46.25
69.58
49.77
Yi- Yi-1.5-9B-Chs
46.67
52.59
50.74
57.78
38.33
47.08
58.75
50.28
Qwen2.5 1emn2.55-4B-Instrue
46.30
50.00
57.04
64.07
42.92
47.08
72.50
54.27
Swven.5-32B-Imma
45.93
48.52
43.33
49.63
44.58
50.42
80.42
51.83
Owen2.5-72BB 5-7n--nrut
44.81
47.78
67.04
70.00
40.00
48.75
79.17
56.79
Llama-3.3 3.mma-3.3-70-- IOB-Instruct
46.30
50,37
68.89
70.37
51.67
50.42
87.92
60.85
Oracle Single Best
46.67
52.59
68.89
70.37
51.67
50.42
87.92
61.22
Random
44.30
49.56
40.57
50.35
35.56
41.75
60.81
46.13
Weighted
46.35
50.53
68.31
70.18
46.87
48.39
86.09
59.53
Prompt LLM
46.67
51.48
61.85
65.93
39.58
49.17
71.25
55.13
GraphRouter 12
47.41
51.48
68.89
70.37
51.67
50.42
87.92
61.17
RouterDC 7
X
44.81
50.37
67.04
68.89
40.00
48.33
77.50
56.71
KNN Router 19
X
46.67
52.22
68.15
71.48
52.08
46.25
86.25
60.44
MP 131
58
X
46.30
52.59
68.89
71.48
49.17
50.42
82.92
60.25
RAGRouter (Ours)
48.52
52.59
71.48
74.44
56.67
56.67
90.83
64.46
Oracle
54.44
57.41
91.85
90.37
69.17
77.92
96.25
76.77
Weighted routing to different LLMs according to RAGRouter's 's empirical probability distribution. To show the ideal upper bound of routing performance, introduce the Oracle baseline by routing each query to its optimal LLM using ground- truth performance data.
assignment:
Implementation Details. For the RAGRouter architecture, we use all-mpnet-base-22 37 as the encoder for both queries and documents, and ms-marco- mns-marc-MMinLMML MiniLM-LL12v2 [46] as the enoseer, resulting in total parameter size of approximately 136M. Both the knowledge representation vector and the RAG capability vector are set to dimensionality last two transformer layers in the query/document encoder and the cross-encoder are frozen during training. The classification loss weight is set to 2.0, and the contrastive learning temperature r to 0.2. The router is optimized using Adam [30] with learning rate of 5e-5, batch size of 64, for 10 epochs. All experiments are conducted on a single NVIDIA RTX 4090D GPU.
768. To mitigate overfitting, all but the
5.2 Main Results
Comparison with Single Non-Routed LLMs. Table 2 presents the test accuracy across variety of knowledge-intene -inegge-intensive tasks and retrieval settings. RAGRouter consistently achieves the highest perfor- mance, with average accuracy of 64.46%. It surpasses the best- berf-peeforming single RAG-enabled LLM, LLaMA-3.3 3..MA--33-700 3----.3----1-- 70B-Instruct (60.85%), by +3.61%, %, demonstrating the effectiveness of routing in RAG. Notably, RAGRouter also outperforms the Oracle Single Best baseline (61. (61.22% w selects the optimal single model for each dataset- by +3.24%, indicating that routing enables the integration of multiple RAG-enhanced LLMs to achieve performance beyond any individual model.
Comparison with Non-RAG-Aware and Other Baselines. RAGRouter significantly outperforms all non-retrievall-wwar routing baselines, including GraphRouter (61.17%, +3.29%), MF (60.25%, +4.21%), KNN Router (60.44%, +4.02%), and RouterDC (56.71% %,, +7.75%) These results underscore the limitations of methods that do not explicitly model the interaction between LLMs and retrieved

---

knowledge. Without capturing retrieval-induced capability shifts, such approaches struggle to make effective routing decisions in RAG. RAGRouter also surpasses rule-based strategies such Random 13%, +1 +18.33%) and Weighted (59.53%, +4.93%). Together, these findings support our core claim: modeling RAG-specific capabilities and knowledge shift is essential for accurate L.LM routing.
Inference Cost of RAGRouter. The inference characteristics are evaluated on a single NVIDIA RTX 4090D GPU. Peak GPU memory usage 4147 MiB. With a batch size of 64, RAGRouter processes 270 instances in seconds across batches, yielding an average inference time of 0.011 seconds per instance. These results clearly show that RAGRouter lightweight and efficient during inference.
5.3 Routing Performance Involving Closed-Source LLMs
To verify that RAGRouter can still deliver performance gains with strong closed-source LLMs. we augment the candidate LLMs set set-compri Qwen2.5-32B- Instruct, Swen2.5-72B-Instruu and 1lama-3.3-70B- Instruct with closed-souree models Qwen2. Max 1 GPT-4o and reasoning- based DeepSeek -R
Table Testing accuracy (%) of RA-
GRouter and baselines with the inclusion of closed-source LLMss (Qwen2.5-Max. GPT- 4o, and DeepSeek-RI) on NQ and WebQ.
bold indicates best results.
Method
NQ
WQ
Avg
As shown in Table 3 RAGRouter outperforms the strongest model GPT-4o by +1.67%, while invok- ing GPT-4o in only 44.79% of samples. It also sur- passes Qwen2.5-Max (+7.71%) and the reasoning- based DeepSeek-RI (+4.17%). Furthermore, RA- GRouter significantly outperforms non-RAG-aware baselines such as KNN Router (+4.17%) and MF (+2.92%). These results demonstrate that RAGRouter can effectively integrate closed-source models, lever- aging their complementarity to achieve simultaneous improvements in both performance and efficiency.
Qwen2.5-32B 22s--ssttcct
44.58
50.42
47.50
Qwen2.5-728- Instruct
40.00
44.38
Llama 1a3na-3.3770 3.3-70B- Instructt
51.67
50.42
51.05
Qwen2,5-Max
51.25
52.08
51.67
GPT:40
58.33
57.08
57.71
DeepSeek-RI
56.67
53.75
55.21
KNN Router
58.75
51.67
55.21
MF
55.42
56.46
RAGRouter
59.58
59.17
59.38
5.4 Extended Results on Latency-Aware Routing
Metrics. Three metrics are used for evaluate latency- aware routing: Area, which measures the proportion of the area under the accuracy- curve within 1 second, reflecting overall time- efficiency; Peak Ace, the highest accuracy achieved within second; and Latency Gap-to-Match, defined as the latency of the best-performing single L.M minus the minimum latency required by method to match its accuracy, indicating the efficiency margin obtained through routing.
Table 4: Area (%), Peak Accuracy (PA, %), and Latency Gap-to-Match (G, for RAGRouter and baselines using serr-thresholl-baae routing on MedMCQA (Local/Online) and TriviaQA "L" indicates failure to match the best single-LLM performance; bold denotes the best result.
McdMCQA (Local)
McdMCQA (Online)
TQA
Method
Area
PA f
G()
Area f
PA t
G Gs)
Area
PA f
Gs) f
RouterDC
55.87
62.22
57.77
65.56
70.99
75.83
MF
46.42
59.63
0.10
54,85
65,56
0.45
66.84
80.00
GraphRouter
47.96
59.30
0.18
57.82
64.67
0.32
65.42
73.15
0.01
KNN Router
52.26
62,30
60.18
66.10
0.33
72.61
81.65
RAGRouter
57.12
62.59
0.24
63.12
67.78
0.51
73.78
87.50
0.76
Quantitative Results and Visualization. We apply the score-threshodd-based routing mechanism to RAGRouter and baselines across all tasks. Results on MedMCQA (Local/Onnine) and TriviaQA are reported in Table 4 and Figure 4 with supplementary results Appendix C RAGRouter consistently achieves the highest Area and Peak Accuracy across all tasks. Specifically, it outperforms basclines
https://chat.qwett https://platfon wttt:/cccccooeccc//c///oooc gotte////oolsgggt-4400
hstps://www.depsseee
8

---

x Qward.5 SB-lnerrac
Owam2.5-21B--mmaa
Waightl
KNN Roase
Men-UJaree 1.1 fnooncc
Qwen7. 5-211-ttsttee
Pronge LLM - RourerD
Qwent Si-hndracl
SaB--hi
Liama-3 S-100-Inomm
Grggfluntzz
peemmo 2 20-n
Qwen2.5-78 -i-eeeeeett
Qwee? 5- Tall dvvnac
Kondren
EmbedLiM
RAGRourt (Oue)
TWragA
90
Texting Accuncy (56)
80
80-
KAGR
Tosting Accuncyy (96)
Accunccy
70-
64-
20-
60-
40-
-0-
30
20
1000
1500
2000
1500 Latency per Query (ms)
Latency per Query (ms)
Latency per Query (ms)
Figure 4: Accuracy-latency curves on MedMCQA (Local), MedMCQA (Online), and TriviaQA.
by 4.86%-10.75 2.94% 8.27% and 1.17%-8.33. 8.36% in Area, and by 0.29% 3.29%, 1.68% 3311 1%, and 5.85% Peak Accuracy on MedMCQA (Local/Online) and TriviaQA, respectively. As shown in Figure its accuracy-latency curve consistently dominates those of baselines, indicating superior accuracy under low-latency constraints. RAGRouter also achieves the largest positive Latency Gap-to-Match margins, demonstrating it can match the accuracy of the best single LLM with substantially lower latency. These results highlight RAGRouter's s ability to exploit the optimization space enabled by retrieval augmentation, efficiently leveraging smaller, faster models to achieve the performance of larger ones without incurring unnecessary latency.
5.5 Sensitivity Analysis and Ablation Study
Table 5: Ablation study of Cross Encoder and RAG Capability Embedding Layer.
PopQA
MedMCQA
Configuration
NQ
WQ
TQA
Avg
A
Local
Online
Local
Online
RAGRouter
48.52
52.59
71.48
74.44
56.67
56.67
90.83
64.46
0.00
Cross Encoder
47.78
52.22
70.74
74.44
55.42
55.00
88.75
63.48
w/o RAG Capubility Embedding Layer
48.52
52.22
71.11
74.44
55.42
54.17
90.00
63.70
-0.76
Table 6: Ablation study of contrastive learning objectives.
PopQA
MedMCQA
ISC
CSC
NQ
WQ
TQA
Avg
A
Local
Online
Local
Online
/
48.52
52.59
71.48
74.44
56.67
56.67
90.83
64.46
0.00
X
48.15
52.22
71.11
73.33
53.75
56.25
89.58
63.49
-0.97
48.52
51.85
71.48
74.07
55.00
56.25
89.58
63.82
-0.64
X
47.78
51.48
69.26
70.74
53.75
53.75
89.17
62.28
-2.18
Ablation Study on RAGRouter Architecture. We conduct an ablation study to evaluate the contri- butions of the Cross Encoder and the RAG Capability Embedding Layer in RAGRouter. As shown in T removing the Cross Encoder reduces performance by 0.98%, highlighting the importance of query-document interactions for deriving fused knowledge representations. Removing the RAG Capability Embedding Layer reduces performance by 0.76%; in this configuration, we removed the explicit modeling of RAG capabilities and instead assigned each LLM single embedding capturing its overall behavior, without distinguishing between parametric knowledge and RAG capabilities. This result demonstrates that explicitly modeling RAG capabilities is essential for effective routing that accounts for each LLM's capacity to exploit external information.
Effects of Positive and Negative Sample Selection in Contrastive Learning. To assess the effectiveness of contrastive learning, we perform an ablation study on the two positive-negative sample construction strategies used in our method. As shown in Table removing either Intra Intra-Setting Contrast or Cross-Setting Contrast results in performance drops of 0.64% and 0.97%, respectively. When both components are removed-ete effecii-efiectively disabling contrastive learning accuracy decreases
Ceaning-accuracy

---

Table 7: Effects of different candidate LLMs sets.
PApQA
MedMCQA
Candidate Set
Method
NQ
WQ
TQA
Avg
A
Local
Online
Local
Online
Oracle Single Best
45.56
51.48
40.37
49.26
36.67
45.42
65.00
47.68
0.00
Small
RAGRouter
45.56
51.85
40.37
51.48
36.67
47.92
65.00
48.41
+0.73
Oracle Single Best
46.30
50.37
68.89
70.37
51.67
50.42
87.92
60.85
0.00
Large
RAGRouter
47.41
50.74
71.11
73.33
54.58
53.33
90.42
62.99
+2.14
Oracle Single Best
46.30
51.48
68.89
70.37
51.67
50.42
87.92
61.01
0.00
Small & Large
RAGRouter
48.15
52.22
71.48
73.33
53.33
55.00
89.58
63,30
+2.29
6:00
0.00
0.00
PopgA SLoal)
D001
01.00
40.76
41.75
4475
4.77
PopOA
6.00
0.31
PopDA (Eivter)
2.00
0.70
0.03
1101
0.10
0.70
0.70
Toizg Aramoo Ohagg-lech
MedNCCOs
0.00
-0.00
0.00
NU
0.31
2:31
3.84
231
4,2
2.11
01:00
1600
0.2%
2.26
3:00
0.74
Wg
6.00
1.00
0.70
1:45
6.76
wQ
0.00
0.00
3.14
67
1.81
10%
TQ4
0.00
0.94
1.42
.42
3331
2:37
T1A4
2A81
11:00
1600
0.45
4.45
1000
4.46
Axg
0.00
1/13
01:65
214
2.32
0.46
Avg
2100
0.08
0.76
0.63
1 10
1:04
0.1
1758
766
164
1051
04
20
Dinomwke
Figure 5: Effects of dimension (baseline: 128).
Figure 6: Effects of À (baseline: À 0).
by 2. 18%. These findings underscore the importance of contrastive learning in capturing knowledge representation shifts and LLM heterogeneity, which is essential for effective routing in RAG settings.
Effects of the Dimension of LLM Knowledge and RAG Capability Embeddings. We investigate the impact of the dimensionality of both knowledge representation and RAG capability vectors. As shown in Figure we observe that performance improves as the dimensionality increases, peaking at 768, after which it declines. Accordingly, we adopt dimensionality of 768 in the main experiments. Detailed results are provided in Appendix D
Effects of A. We investigate the impact of the classification loss weight À in the overall loss function (Eq. [9) on test accuracy. As shown in Figure 6 we observe that combining contrastive and classification losses yields better performance than using contrastive loss alone (i.e., X = 0, achieving only 63. 76% on average). As increases, accuracy improves, peaking at À = 2 with 64.46% on average, before experiencing slight decline. Based on this, set À 2 in all experiments. Detailed results are provided in Appendix [
Effects of Diffferntt Candidate LLMs Sets. We investigate how the composition of candidate LLMs affects routing performance. To this end, we form two subsets of models-Smmi Small (<3B parameters) and Large (232B)-a0 and evaluate three configurations: Small only, Large only, and heterogeneous set combining both. As shown in Table 7 RAGRouter consistently outperforms the Oracle Single Best in all settings, demonstrating its ability coordinate models and achieve cumulative gains, Two key insights emerge. First, the routing upper bound is largely determined by model strength: the Large set yields significantly higher Oracle Single Best average (60.85%) than the Small set (47.68%), with RAGRouter following the same trend (62.99% vs. 48.41%) Second, combining heterogeneous models yields the best performance: the Small & Large setting achieves the highest RAGRouter average (63.30%) and the largest gain over its Oracle Single Best (+2.29%), suggesting that model diversity improves complementarity and enables more effective routing.
Conclusion
In this paper, we have studied the problem of LLM routing in Retreeval-Augmented Generation (RAG) for the first time and propose RAGRouter, the first RAG-aware routing method. By leveraging contrastive learning, RAGRouter captures knowledge representation shifts induced by external documents, enabling effective routing decisions. Experiments on diverse knowledge-intensvve tasks demonstrate that RAGRouter outperforms existing non-RAG-aware methods and achieves strong performance-effiiinnny trade-offs under low-latency constraints.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by National Key R&D Program of China (No. 2022 190.D00119000 China NSF grant No. 62025204, No. 62202296, No. 62272293, No. 62441236, No. U24A20326, and No. 62572299, Tencent WeChat Research Program, and SITU-Huawei Explore X Gift Fund. The opinions, findings, conclusions, and recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agencies or the government.
References
[1] Q Jiang Albert, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, and Devendra Singh Chaplot. Mistral 7b, arXiv, 2023.
[2] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answerr pairs. Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544, 2013.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Sébastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
[5] Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. Recent advances in aurrivall-augmette text generation. In Proceedings of the 45th international ACM SIGIR conference on reseurch and development in information retrieval, pages 3417-3419, 2022.
[6] Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language mod- els in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754-17762, 2024.
[7] Shuhao Chen, Weisen Jiang, Baijiong Lin, James T Kwok, and Yu Zhang. Routerde: Query- based router by dual contrastive learning for assembling large language models. arXiv preprint 2409v 19886, 2024.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1559-1607. PmLR, 2020.
[9] Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang, Hailong Sun, and Philipp S Yu. Harnessing multiple large language models: A survey Ilm ensemble. arXiv preprint 2502 18036, 2025.
[10] Dujian Ding, Ankur Mallick, Chi Wang. Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid llm: Cost-efficient and quality- aware query routing. arXiv preprint arXiv:2404.145 146188 2024.
[11] Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. Enhaneing noise robustness of eetrieval-augment language models with adaptive adversarial training. arXiv preprint 24XX::2405.20978 2024.
[12] Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: graph- router for Iim selections. In The Thirteenth International Conference on Learning Representations, 2024.
[13] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval- -aumeeete generation for large language models: A survey. arXiv preprint 23 10997, 2, 2023.
[14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

---

[15] Kelvin Guu, Kenton Lee. Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning. pages
PMLR, 2020.
[16] Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. Wikiasp: A dataset for multi-domain aspect-based summarization. Transactions of the Association for Computational Linguistics, 9:21 9:211-225, 2021.
[17] Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. arXiv preprini arXiv:2301.00300 2022.
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[19] Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang. Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403. 12031, 2024.
[20] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020.
[21] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yoe Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint Siti::2200.032999 1 1())4, 2022.
[22] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1-38, 2023.
[23] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017.
[24] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov. Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769-6781, 2020.
[25] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
[26] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[27] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intene intengsveetetiiveeeee nlp tasks. Advances in neural information processing systems, 33:9459-9444 9474, 2020.
(28) Kuan Li, Liwen Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Shuai Wang, and Minhao Cheng. Lara: Benchmarking retrieval-augmenner generation and long-context llms-no silver bullet for Ic or rag routing. arXiv preprint arXiv:2502.09997 2025.
[29] Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? study on several typical tasks. arXiv preprint Stti::2305008862 2023.
[30] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXiv:1711.01101, 2017.
12

---

[31] Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient -rrrrd-guided ensemble of large language models. arXiv preprint (tt.::3/1.000992 2023.
[32] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprini arXiv:2212. 2022.
[33] Isaac Ong. Amjad Almahairi, Vincent Wu, Wei Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route Ilms from preference data. In The Thirteenth International Conference on Learning Representations 2024.
[34] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 260. PMLR, 2022.
[35] Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language understanding by generative pre-trane training. OpenAl blog, 2018.
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[37] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1908. 10084, 2019.
[38] Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 606-615, 2024.
[39] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210-31227. PMLR, 2023.
[40] Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thomp- son, and Mikhail Yurochkin. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309. / 15789, 2023.
[41] Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 01-2566--25888 2021.
[42] Huatong Song, Jinhao Jiang. Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in lims via reinforcement learning. arXiv preprint arXh::2503.05592 2025.
[43] Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. Tensoropera router: A multi-model router for efficient 11m inference. arXiv preprint arXiv:2408. 240.22200....20 2024.
[44] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.0m 001 18, 2024.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in newral information processing systems, 33:5776-5788, 2020.
[47] Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wa, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, et al. Searching for best practices in retrieval- augmented generation. arXiv preprint arXiv:2407.01219, 2024.
13

---

[48] Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, al. Speculative rag: Enhancing retrieval augmented generation through drafting. arXiv preprint arXiv:2407.08223, 2024.
[49] Zhepei Wei, Wei-Lin Chen, and Yu Meng. Instructrag: Instructing retrieval -agreeeall-gmeeeted genera- tion via self-synthesized rationales. arXiv preprint wwXXv:2406.13629, 2024.
[50] Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C- pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641-649, 2024.
[51] Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May D Wang, Joyce C Ho, Chao Zhang, and Carl Yang. Bmretriever: Tuning large language models as better biomedical text retrievers. arXiv preprint arXiv:2404.18443; 2024.
[52] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1.11:2412..5115.. 2024.
[53] Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng Ann Heng, and Wai Lam. Unveiling the generalization power of fine-tuned large language models. arXiv preprint arXiv:2403.09/63 2024.
[54] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 2018.
[55] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024.
[56] Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jin Nie. Retrieve anything to augment large language models. arXiv preprint arxiv:2310.07554, 2023.
[57] Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. Llmeval: A preliminary study how evaluate large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19615-19622, 2024.
[58] Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramehan- dran. Embedllm: Learning compact representations of large language models. arXiv preprint arXiv:2410.02223, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
Yes, the paper discuss the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided detailed training settings and datasets information in the Experiments section and Appendix.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use public dataset to conduct experiments, and our data and source code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/giiee//eeeuuubiiiiiin
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the experimental setting and details in the Experiments section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All experiments are conducted on single NVIDIA RTX 4090D GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpge///aarrsss.cppbbbb///heeit
Answer: [Yes]
Justification: Yes, the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the broader impacts discussed in the conclusion of this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
19

---

Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: have cited all referenced papers and open-source assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, thhors thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We construct data suitable for LLM routing under RAG based public datasets, and relevant details are presented in the Experiments section and Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Yes, the paper describes the usage LLMs for LLMs are important component of experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM for what should or should not be described.
21

---

Appendix Contents
Experimental Setup Details
23
A.1 Details of Candidate LLMS
23
A.2 Details of Datasets
23
Impact of RAG on LLM Performance
23
C Full Results on Latency-Aware Routing
25
D Full Results of Sensitivity Analysis and Ablation Study
26
Case Studies of Failures and Successes
26
Quantitative Failure Case Studies of Routing Decisions
26
E,2 Qualitative Success Case Studies of RAGRouter
27
Routing Performance under Cross-Domain Settings
28
Routing Performance under Noisy Retrieval
28
Routing Performance on Summarization Task
29
22

---

A Experimental Setup Details
A.1 Details of Candidate LLMs
The responses of - Instruct,,, Llama-3-700 3-700---ntrruuue and closed-source LLMs were ob- tained via API calls, while other open-source LLMs were locally deployed using the vLLM framework [26] for high-speed inference from Huggingface
8 Notably, latency calculations for API-based models incorporated both average network latency and inference time, whereas latency measurements for locally deployed models exclusively accounted for inference time.
A.2 Details of Datasets
Table 8: The statistics results for different tasks.
PopQA
MedMCQA
Query Num
NQ
WQ
TQA
Local
Online
Local
Online
Train
2000
2000
2000
2000
1000
1000
1000
Test
270
270
270
270
240
240
240
As shown in Table g
we randomly sampled queries from five knowledge-intensiv tasks (PopQA
NQ (25). WebQ 22, and TriviaQA (23)) and partitioned them into training and test sets. For PopQA and MedMCQA, we retrieved documents through both local and online search engines following 42). while for NQ, WebQ. and TriviaQA, we constructed artificially noised documents based on [TI] with an equal proportion of four types: Golden Context, Relevant Retrieval Noise, Irrelevant Retrieval Noise, and Counterfactual Retrieval Noise. Each query-documen pair was processed by the 15 LLMs described in Section 5. generate responses. These responses were then evaluated against ground-trume answers to derive binary scores.
32]. MedMCQA 34
B Impact of RAG on LLM Performance
Performance Shift with and without RAG. We analyze how LLM performance changes before and after RAG across different models and settings. As shown in Figure 7 on real retrieval settings such as PopQA (Online), MedMCQA (Local), and MedMCQA (Online), RAG improves overall performance and reduces the accuracy gap among models. Notably, small-scale models benefit more (e.g., Qwen2.5-0055B 5-0...--nntt achieves 17.57% improvement on MedMCQA (Online)) compared to larger models (e.g., -llma-..3-70B-Innse-
improves by only 0.88%). In contrast, under noisy retrieval settings (NQ, WebQ, TriviaQA), the impact of RAG varies-some models improve while others degrade (e.g.. on NQ, Llama-3.3 -.3-a-3..-70---- -.3-tuB-Intttuct performs worse, while Owen2.5-0.5BBB ---ntrutt performs better). These results highlight inconsistent performance shifts across models, indicating that RAG significantly alters the distribution of response quality, which undermines the assumptions of non-RAGG-wware routing strategies.
A Analysis of Response Quality Reversal. We further investigate how RAG causes quality reversals for the same query. To quantify these reversals, we define two metrics at the task level. The positive gain rate is the proportion of queries that were unanswerable without retrieval but became answerable with retrieved documents, calculated as:
Positive Gain Rate = #(Incorrect w/o RAG - Correct w/ RAG)
(10)
#Incorrect w/o RAG)
The negative interference rate measures the opposite effect-queries that were answerable without retrieval but became unanswerable due to retrieved content:
Negative Interference Rate #(Correct w/o RAG + Incorrect w/ RAG)
(1))
#(Correct w/o RAG)
Figures g and report these metrics across 15 LLMS and multiple settings, On average, across all
models and tasks, the positive gain rate is 30.86%, negative interference rate 31.46%. These results confirm that response quality reversals induced by external documents are common suggests that RAG-induced knowledge shifts are widespread among LLMs.
httpe://hugginntaee.coo
23

---

(a)
(e)
Figure 7: Accuracy of 15 LLMs before and after RAG under various tasks and retrieval settings: (a) PopQA (Online), (b) MedMCQA (Local), (c) MedMCQA (Online), (d) NQ, (e) WebQ, (f) TQA.
Owee 4458 doecee
Uheen 25 18 Anvirecr
Clowe 2.2 16 Jourer
Orves Aweent
Owend 5.78 butees
Mosuee Mone-u---00 hostess-311
Maka Kaann 488 devdrtr
Owost AH dwerst
Owoot uZN laeess
Chood lovirst
gemman Zhvr
Are
36,20
36.47
35.76
34.25
34.6
33:23
37.12
33.50
91.2
32.21
37.13
33.92
$2.55
30,00
24.15
PoyDA (O (OXiae))
45:90
42:33
44.55
45/05
41:77
41.22
45
14
37.19
37.34
45.29
91.67
38.9
9613
26.45
40:35
ModMCRA (Las!))
13.06
10.32
11.22
18:59
16
31
M16
11:11
37:43
17:36
5.51
20.57
MeMMCOA (Ohtlse)
11.6
15.06
15
15:24
33.55
15.00
4181
43.11
40.11
46:21
3::11
4073
4091
18:45
23,99
22.01
24211
23.24
26.92
21.74
22.88
26.13
26.04
27:31
22.34
22.34
20
23:70
25,87
20,99
23,00
23377
20.17
30.00
29.39
13.35
22.33
31.32
26.45
16:92
1812
16:20
21.2
20.29
35.07
30.43
33.05
55:83
Podlir Gata Rtas (B)
Figure& 8: Positive Gain Rates of 15 candidate LLMs across tasks.
Masieet AR tovens tveres--341 2409
Mesia Luma 3.1 bestoot
Qleat 10.90 hoarrct
Limoo foaated
Limu foatted
Omeez Abcmnc
Omiu Auains
Q N-448 loerect
Useat h-ta fostect
Dseat feated
Lior teeteet
Ni MassihCCe Che
V-. Clu
TryDA (oeatt
22:5
21.36
16.22
28.57
30.78
21.11
34
25.18
21.38
24.61
18:00
19:75
20.37
24.09
27.29
22.10
Popti
13,90
11:65
6.6
96)
9.13
16.66
6.22
9.41
11.00
13.66
5.65
8.58
12:04
11:27
16:09
10:6
McMMCDA daaa))
1:.7
67.17
5005
10
1T
15:11
51.11
5171
3161
15
31.11
26.33
15:0
11:00
(4:00
MeDICO (rntion)
SU02
41,35
43.00
4434
11:11
76.14
12.53
40.32
10.00
#125
27.54
22.97
37.36
15.33
15:44
35.00
NQQ
40,55
31,99
2033
6351
29.28
47.60
43
$1.25
37.18
35.06
31322
34.21
31.41
56.40
27.85
41,00
MQQ.
<lUS
43:99
47.15
34:72
18.26
47:57
45:11
43:21
31.54
41.33
30.35
34.25
24.52
28.35
30017
5032
TOAA
1.33
12.65
44.04
23:55
31.00
64
1.54
2059
19
24.44
15.89
11.36
15.51
1007
36/6
1.15
11.00
18.00
7178
37.04
77.64
23.13
30.7
30/77
Negator Rate
Figure 9: Negative Interference Rates of 15 candidate LLMs across tasks.
24

---

C Full Results on Latency-Aware Routing
Setting Details. Based on LLMs' profiles, the 15 candidate models in Section 5.4 were ranked as follows in ascending order: Qwen2.5-0.5B-Imsttee Llama-3.2 1-tmaa-3.2-111 1-..-..----mmsst BIIntruuct, Owen2.5-1.3B-IIsttue gemma-2-2b-it,
Qwen2.5-3B-Instrrech Vi-1.5-6B-Chat, @0wen2.5-7B-Instruus Meta-Llama-3, 1-8---nttttttttt Yi-1.5-9B-Chat, Swwen..5-14BInnsiie Qwen2.5-32B 1-102.5--2B-nnstruc. Qwen2.5-72B Instruct, Llama-3.3. 1.mma-3.3-7BBB. 70B-Innttuuct Substitution models were selected from immediate predecessors of the highest- rrootin--soore score model within the threshold. For quantitative analysis, we discretized the threshold parameter over [0, 1] with step size of Ic-4 to trade-off curves.
generate complete high-precision accuracy-
Results. Figure T0 illlsttrats the securac-laatnne curves of RAGRouter and baseline methods on PopQA (Local/Online), NQ, and WebQ, while Tables present their quantitative results on Area, Peak Acc, and Latency Gap-to-Match metrics. RAGRouter achieves the highest scores Area and Peak Acc, with its accuracy-latency curve mostly surpassing the baselines, demonstrating strong performance-efficing trade-offs under low-latency constraints.
Qwen22 1-OL5B--assa
Llasta 3.2 Jastuctt
Minisnal 4B Busrrct2410 2410
X
Qwen2.5 328 Instxt
Weiglted
KNN Router
L.aam-3322
5-3B
Mat-- listnat
5-222 5-725-5-2---na
Qwan? All-breract
x 11mm-111-WWhhomm
Chack
CinquBouster
genina 2biit
0.ea255:7 lnstort
Quei2. Insruct
Randor
EntedLLM
RAGRoster (Dus)
PopQA (Local)
PopQA (Online)
RACGRouuer
RAGRouttr
Tostns Accraay (R)
54
48-
52
80-
48
X
140
500
750
1000)
1750
1900
1750
2000
1250
1509
1750
2000
NQQ
WQ
RAGRoouter
70-
60-
Doalig Kounay (1)
30
250
750
1500
Lateney per Query (mm)
Lauency per Quary
Figure 10: Accuracy-latency curves on PopQA (Local/Online). NQ and WebQ.
Table Area (%), Peak Accuracy (PA, %), and Latency Gap-to- Mattthh-atcch (G, for RAGRouter and baselines using score-thhe score-threshole threshold--aasee routing on PopQA (Local/Online), NQ and WebQ. in Latency -M-Mchh indicates failure match the best single- performance; "." Area denotes maximum routing latency below 1s, excluded from comparison; bold denotes the best result.
Method
PopQA (Local)
PopQA (Online)
NQ
WQ
Area 1
G
Area
PA
G
Area
PA
G(s)
Area
PA
G
RouterDC:
44.58
47.04
0.02
49,80
51.85
33.10
42.92
46.39
49.17
MF
46.30
49.96
52.22
-0.91
37.34
44.17
50.42
1.66
GraphRouter
47.41
51.48
35.68
38.35
0.02
0.01
KNN Router
46.67
-0.06
52.22
41.13
50.63
0.72
50.42
1.72
RAGRouter
45.13
48.52
-0.48
50.13
52.59
0.15
43.63
55.83
1.13
58.33
1.84
25

---

D Full Results of Sensitivity Analysis and Ablation Study
Table 10 presents the impact of the dimensionality of LLM knowledge embeddings and RAG
capability embeddings test accuracy, the best average accuracy is observed dimension 768.
Table 10: Effects of dimension.
PopQA
MedMCQA
Dimension
NQ
WQ
TQA
Avg
Local
Online
Local
Online
128
48.52
52.22
71.11
73.70
54.17
53.33
87.92
63.00
256
48.52
53.33
70.37
73.33
55.42
55.83
88.75
63.65
49.26
52.59
70.37
72.59
56.25
53.75
89.17
63.43
512
48.89
52.59
70.37
73.33
55.42
56.25
89.17
63.72
768
48.52
52.59
71.48
74.44
56.67
56.67
90.83
64.46
1024
48.52
52.59
71.11
73.70
55.42
51.67
90.00
63.29
Table 11 presents the impact of the loss weight X on test accuracy, the best average accuracy is
observed at À = 2.
Table 11: Effects of X.
PopQA
MedMCQA
NQ
WQ
TQA
Avg
Local
Online
Local
Online
0.0
48.89
52.59
70.37
74.44
55.42
$4.17
90.42
63.76
0.5
48.89
52.96
70.74
74.07
55.42
54.17
90.42
63.81
48.52
52.59
71.48
73.70
55.42
55.42
90.42
63.94
1.5
48.89
52.59
71.48
74.07
55.83
55.42
90.83
64.16
2.0
48.52
52.59
71.48
74.44
56.67
56.67
90.83
64.46
2.5
48.52
52.96
71,48
74.44
57.08
56.25
90.42
64.45
3.0
48.52
52.96
71.85
74.44
55.83
55.83
90.00
64.21
E Case Studies of Failures and Successes
E.1
Quantitative Failure Case Studies of Routing Decisions
We conduct a guantitative analysis cases where RAGRouter and -on-RA--aaare routing methods fail to select the correct LLM or route correctly on the PopQA (Local), MedMCQA (Local), and NQ datasets. particular, we categorize non con-irivially unsolvable failures (i.e., excluding cases where all LLMs fail) into four types as follows:
F1: Failure to perceive performance degradation caused by RAG (i.e., cases where LLM's performance decreases after RAG applied).
F2: Inherent task difficulty, where the majority of LLMs (e.g., >80%) fail, leading to routing confusion.
F3: Overconfident selection of stronger LLMs for cases where high-capacity LLMs chosen but still fail, outside the conditions F1 and F2.
F4: Other factors, such as outliers or ambiguous inputs.
Table 12: Type-wise failure rates (as percentage of all cases) and the overall failure rate (FR).
Method
F1 (%)
F2 (%)
F3 (%)
F4 (%)
FR (%)
KNN Router
5.00
8.33
1.28
1.54
16.15
MF
5.51
8.21
1.41
1.79
16.92
RAGRouter
3.85
7.05
1.41
0.64
12.95
26

---

As shown in Table 12 we can observe that compared with non non-RAG-aware routing methods, including KNN Router and MF, our RAGRouter achieves the lowest overall failure rate and the lowest F1-type failure rate. This suggests that RAGRouter more correctly captures performance shifts in LLMS introduced by RAG.
We additionally analyze cases where RAGRouter fails while non- non-RAG-aware routing methods succeed. We find that such cases are less than 2% of all the evaluated cases. Notably, more than 50% of these cases fall into the F2 type, where the performance gap between candidate LLMs is inherently small.
E.2 Qualitative Success Case Studies of RAGRouter
We further illustrate RAGRouter's ability to perceive changes in LLM knowledge states under RAG conditions through the two case studies shown in Table 13 and Table 14 In Table [3] the retrieved document contains both correct answer information and certain distracting content. In this case, RAGRouter identifies that the document provides significant performance gains for Qwen2.5-14B- Instruct and accordingly selects it as the responder. Although this model produces an incorrect response in the non-RAG setting due to confusion between two French departments, it successfully corrects the answer when the document is incorporated. In contrast, Meta- --ae---lma...----mmmmme exhibits greater sensitivity to distracting content in the document, where the introduction of RAG leads to reverse interference and impairs its reasoning, and thus it is not prioritized by RAGRouter. However, traditional routing strategies without RAG awareness, such as MF, rely solely on static model performance and fail to perceive the document-induced performance shifts, ultimately routing to Deeta-Llama-33 3.1 3.1-8B- 11--BB-Insruue Instruct and resulting in routing failure for this sample.
Table 13: A case from PopQA (Online), demonstrating RAGRouter's 's ability to perceive LLMs' RAG capability and knowledge shift.
Query
What is Agen capital of?
Document
Agen, located in the Nouvelle- Aquitaine region of Southwestern France, serves as the prefecture of the Lot-ct-Garonm department. Known for its geographical positioning along the river Garonne, the city lies approximately 135 kilometers southeast of Bordeaux. It has rich cultural heritage, featuring various historical buildings such as the twelfth-century Agen Cathedral and numerous museums, including the Muséc des Beaux Arts. Agen is also colloquially referred to as the čapital of the prune, hosting a popular prune festival every August. The town, with population of 32,485 in 2021, has its own Roman Catholic diocese, adding to its historical significance within the region.
Ground truth
Lot-et-Garonne
Router
RAGRouter
MF
Selected L.M
Qwen2.55-14B-Istrrcc
Meta-L.ama-3.. 1 1-8t-rntrutctt
Response w/o RAG
Lot department. (X)
Lot-et-Garonne department. (V)
Response w/ RAG
Lot-ct-Garonne department. (V)
The prune capital. (X)
As shown in Table 14 the query itself is challenging, and neither Llama-3.3- 3.3- nor Qwen2.5577BB-Instree is able to provide a correct answer without access to external documents. However, when the retrieved document containing key information is provided, Llama-3.3-70B- Instruct, benefiting from its stronger capabilities in information extraction and comprehension, successfully identifies "Poreotics" as the correct answer. contrast, Dwen2.5-72B-Inntrrue fails to effectively utilize the document and still produces an incorrect response. In this scenario, RAGRouter able to sense the differential capabilities of the candidate LLMs leveraging retrieved content and routes the query to the model with stronger information extraction ability, leading to a correct response. By contrast, -nn-RrG-aarre routing methods based on static modeling assumptions, such as RouterDC, fail to capture dynamic performance shifts induced by retrieval and result in incorrect routing and response failure. This case further highlights the advantage of R. RAGRuuter in perceiving capability differences among LLMs in aatreval--uugentee settings.
27

---

Table 14: A case from NQ, demonstrating RAGRouter's ability to perceive LLMs' RAG capability and knowledge shift.
Query
who the dancers in the lazy song video?
Document
tenth the chosen. The official video directed Mars and Cameron Duddy, produced by Nick Tahri and Dara Siegel, and features Poreotics wearing chimpanzee masks; was released on April 15, 201 1. The whole video presented in as a lone continuous and uninterrupted shot, it begins with Mars singing and hanging out in bedroom with five dancers. they all wear monkey masks and Mars dresses in black sunglasses and flannel shirt. While Mars sings what he feels on a day off, he and the monkeys perform dance moves typical of boy-band,
Ground truth
Poreotics
Router
RAGRouter
RouterDC
Selected LLM
Llama-3.3-7088 1.ma-3.3-70B-Instrue Instruct
Breen2.5-72B-nstteee
Response w/o RAG
Bruno Mars and dancers. (X)
Five dancers. (X)
Response w/ RAG
Porcotics dancers. (V)
Bruno Mars and his backup dancers. (A)
F
Routing Performance under Cross-Domain Settings
To evaluate RAGRouter's generalization across domains, we design two cross-domain settings:
Setting 1: Trained on MedMCQA (Local) that falls into medical domain, but tested on a new dataset HotpotQA [54] for Wikipedia-basee multi-hop QA.
Setting 2: Trained on NQ that contains real-world search queries, but tested on MedMCQA (Local).
Table 15: Testing accuracy (%) of RAGRouter and baselines under two cross-domain settings.
Setting
Setting
Method
McdMCQA
NQ NQ- (Local)
KNN Router
18.60
63.33
MF
20.20
58.89
RAGRouter
21.20
68.89
As shown in Table 15| RAGRouter outperforms non non-RAG-aware routing methods by 1.00% and 5.56% in two cross-domain settings, demonstrating strong generalization. This performance advan- tage can be attributed RAGRouter's is contrastive learning framework, which effectively captures relative RAG capability differences among candidate LLMs, even when transferred across domains.
G Routing Performance under Noisy Retrieval
We further partition the TriviaQA dataset into four subsets with manually injected retrieval mrise-Golden Context, Relevant Noise, Irrelevant Noise, and Counterfactual Noise-to evalu- atc the routing effectiveness of RAGRouter under different noise conditions, in comparison with various baseline methods. As shown in Table 16 RAGRouter consistently achieves the best perfor- mance across all subsets, outperforming both the Oracle Single Best and other routing strategies, demonstrating strong robustness to different types of retrieval noise.
Notably, on the Relevant Noise, Irrelevant Noise, and Counterfactual Noise subsets, non-RAG-awen baselines exhibit significant performance gaps compared to RAGRouter, highlighting their limitations high-noise retrieval scenarios. We hypothesize that this due to knowledge shift induced by noisy retrieval, which affects different LLMs in heterogeneous ways. As result, routing methods that rely on fixed model representations and ignore RAG capabilities struggle to accurately model routing strategies under such conditions.
28

---

Table 16:
Test accuracy (%) of RAGRouter and baselines on TriviaQA subsets with different types of retrieval noise, bold indicates best results.
Method
Golden Context
Relevant Noise
Irrelvvnt Noise
Counterfactual Noise
Avg
Oracle Single Best
95.00
83.33
90.00
83.33
87.92
KNN Router
96.67
78,33
83.33
86.67
86.25
GraphRouter
83,33
90.00
83,33
RouterDC
78.33
73.33
78.33
77.50
MF
95.00
76.67
80.00
80.00
82.92
RAGRouter
98.33
86.67
90.00
88.33
90.83
H Routing Performance on Summarization Task
To verify that RAGRouter is also applicable RAG tasks beyond question answering, conduct experiments on WikiASP [16]. a summarization task commonly used in RAG-related research. Specifically, WikiASP aims generate aspect-based summaries for Wikipedia entities and naturally supports the use of retrieved evidence. We adopt ROUGE-L as the evaluation metric.
For continuous performance metrics such ROUGE-L. extend RAGRouter's contrastive learning by score-based probabilistic sampling. Given each candidate LLM's s normalized score s E [0, 1]. we label it as "can answer" (positive, label 1) with probability and "cannot answer" (negative, label 0) with probability 1 - s. This strategy is applied in our WikiASP experiments, enabling contrastive training based continuous performance signals rather than discrete correctness labels.
Table 17: Performance of R.GRouter and baselines on WikiASP measured by ROUGE-L.
Method
WikiASP
Qwen2.5-0. 1-5.--IIstttuee
0.1388
1een2.551..B-Insttruc
0.1411
Mamma-3.2-BB-Imm.
0.1768
Qwen2.5-3B-Inntmen
0.1528
Qwen2.5-7B-Insee
0.1897
Llama- 1.mma-3.188B -----B-Innttuctt
0.1455
KNN Router
0.1881
MF
0.1865
RAGRouter
0.1981
As shown in Table 17 RAGRouter outperforms non-RA Mon-RAG-awar routing baselines by more than 5.3%. These findings demonstrate that RAGRouter generalizes well to different RAG tasks.
29


Reviews:
[{"review_id": "GPvCYgZb8O", "reviewer": "Reviewer_rFRH", "summary": "This paper addresses the challenge of routing user queries—given specific retrieved documents—to the most suitable LLM among multiple candidates. Existing routing techniques focus on static parametric knowledge and fall short in RAG settings, where dynamically retrieved documents can significantly alter model behavior. The authors propose RAGRouter, a routing framework that incorporates both document semantics and model-specific RAG capabilities through a contrastive learning objective. Extensive experiments across five knowledge-intensive benchmarks demonstrate significant performance improvements over both individual LLMs and non-RAG-aware routing baselines.", "questions": "I am curious whether the router can maintain high routing accuracy on domain-specific queries that are underrepresented or absent in the training data. Additional evaluations—beyond the medical domain—on domain transferability or robustness to domain shift would help us better understand the generalizability of the proposed contrastive learning method.", "limitations": "None", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- This paper formalizes the RAG-aware LLM routing problem by taking into account the effect of retrieved documents and the varying RAG capabilities of different LLMs, in contrast to previous works about LLM routing.\n- The introduction of learnable RAG capability embeddings is a well-motivated and innovative design that enables RAG-aware routing decisions.\n- The proposed method, RAGRouter, significantly outperforms existing baselines across five benchmarks, demonstrating strong performance and good generalization.\n- The extended threshold-based mechanism to trade off performance and latency is a practical addition.\n\n# Weakness\n- The main contribution and novelty of this paper lies in learning the RAG Capability Embedding to enable RAG-aware routing. However, the paper lacks a dedicated ablation study to validate the effectiveness and necessity of this component. Therefore, the core contribution and motivation of the proposed method remain empirically unverified. （I will consider raising my score if the authors address this concern）\n- The paper lacks detail qualitative or quantitative analysis of cases where RAGRouter fails to select the correct LLM or route correctly, especially in comparison with traditional LLM routing methods. For example, the paper does not explicitly evaluate how RAGRouter performs when faced with noisy retrieved documents.", "was_revised": true, "final_justification": "The authors’ response has effectively addressed my concerns. The proposed method demonstrates state-of-the-art performance, strong robustness to noisy documents, and better generality under domain shifts. I recommend accepting this paper.", "rebuttal": "Thank you for your thoughtful feedback. We have addressed each of your concerns as follows.\n\n> **Q1**: Ablation study of the RAG Capability Embedding.\n\n**A1**: We have followed your suggestion to add an ablation study to validate the effectiveness and necessity of explicitly modeling the LLM’s intrinsic RAG capability.\n\nIn the ablation setting “w/o RAG Capability Embedding”, **we removed the explicit modeling of RAG capabilities** and instead assigned each LLM a single embedding that captures its overall behavior, without distinguishing between parametric knowledge and RAG capabilities.\n\n|                                         | PopQA (Local) | PopQA (Online) | MedMCQA (Local) | MedMCQA (Online) |      NQ       |     WebQ      |      TQA      |      Avg      |\n| :-------------------------------------: | :-----------: | :------------: | :-------------: | :--------------: | :-----------: | :-----------: | :-----------: | :-----------: |\n|                RAGRouter                |     48.52     |     52.59      |      71.48      |      74.44       |     56.67     |     56.67     |     90.83     |     64.46     |\n| w/o RAG Capability Embedding ($\\Delta$) | 48.52 (-0.00) | 52.22 (-0.37)  |  71.11 (-0.37)  |  74.44 (-0.00)   | 55.42 (-1.25) | 54.17 (-2.50) | 90.00 (-0.83) | 63.70 (-0.76) |\n\nThe ablation study results show that **without the RAG Capability Embedding, the performance across 5 datasets with local and online retrieval strategies drops by 0.76% in average**. Notably, on **NQ and WebQ, which involve imperfect retrieval results, the performance drops by 1.25% and 2.5%**, respectively. These findings validate the effectiveness of our **explicitly introduced and disentangled RAG capability representation**, which plays a critical role in enabling effective RAG-aware routing by accounting for the LLM’s ability to process external information in dynamic knowledge update scenarios.\n\n---\n\n> **Q2**: Failure and correct case analysis.\n\n**A2**: We have followed your suggestion to add the quantitative analysis of cases where RAGRouter and traditional LLM routing methods fail to select the correct LLM or route correctly on the datasets of PopQA (Local), MedMCQA (Local), and NQ. In particular, we categorize non-trivially unsolvable failures (i.e., excluding cases where all LLMs fail) into 4 types as follows:\n\n- F1: Failure to perceive performance degradation caused by RAG (i.e., cases where LLM's performance decreases after RAG is applied).\n\n- F2: Inherent task difficulty, where the majority of LLMs (e.g., >80%) fail, leading to routing confusion.\n\n- F3: Overconfident selection of stronger LLMs for cases where high-capacity LLMs are chosen but still fail, outside the conditions of F1 and F2.\n\n- F4: Other factors, such as outliers or ambiguous inputs.\n\nWe present the type-wise failure rates (as a percentage of all cases) and the overall failure rate (FR) in the table below.\n\n|        Method        |  F1 (%)  |  F2 (%)  |  F3 (%)  |  F4 (%)  |  FR (%)   |\n| :------------------: | :------: | :------: | :------: | :------: | :-------: |\n|      KNN Router      |   5.00   |   8.33   | **1.28** |   1.54   |   16.15   |\n|          MF          |   5.51   |   8.21   |   1.41   |   1.79   |   16.92   |\n| **RAGRouter (Ours)** | **3.85** |   **7.05**   |   1.41   | **0.64** | **12.95** |\n\nWe can observe that **compared with traditional LLM routing methods**, including KNN Router and MF, **our RAGRouter achieves** the lowest overall failure rate and **the lowest F1-type failure rate**. This suggests that **RAGRouter more correctly captures performance shifts in LLMs introduced by RAG**. We have also **provided detailed cases in Appendix F where RAGRouter selects correctly while traditional LLM routing methods fail**.\n\nWe further examine cases where **RAGRouter fails while traditional LLM routing methods succeed**. We find that **such cases are less than 2% of all the evaluated cases**. Notably, **more than 50% of these cases fall into the F2 type**, where **the performance gap between candidate LLMs is inherently small**. \n\n---\n\n> **Q3**: Performance when faced with noisy retrieved documents.\n\n**A3**: **We have reported and analyzed RAGRouter’s performance under noisy retrieved documents in Appendix C (Routing Performance under Noisy Retrieval) of the submission.** In particular, we add synthetic noise into the NQ, WebQ, and TriviaQA datasets. Take TriviaQA for example. The evaluation results under four conditions, including Golden Context, Relevant Noise, Irrelevant Noise, and Counterfactual Noise, are summarized as follows.\n\n|        Method        | Golden Context | Relevant Noise | Irrelevant Noise | Counterfactual Noise |    Avg    |\n| :------------------: | :------------: | :------------: | :--------------: | :------------------: | :-------: |\n|      KNN Router      |     96.67      |     78.33      |      83.33       |        86.67         |   86.25   |\n|          MF          |     95.00      |     76.67      |      80.00       |        80.00         |   82.92   |\n| **RAGRouter (Ours)** |   **98.33**    |   **86.67**    |    **90.00**     |      **88.33**       | **90.83** |\n\nWe can observe that **our RAGRouter consistently outperforms traditional LLM routing methods, with performance gains more than 4.58%, highlighting its strong robustness to retrieval noise**.\n\n---\n\n> **Q4**: Additional evaluations on domain transferability or robustness to domain shift.\n\n**A4**: To evaluate RAGRouter’s generalization across domains, we design two cross-domain settings:\n\n- Setting 1: **Trained on MedMCQA (Local) that falls into medical domain, but tested on a new dataset HotpotQA for Wikipedia-based multi-hop QA**.\n\n- Setting 2: **Trained on NQ that contains real-world search queries, but tested on MedMCQA (Local)**.\n\n|            | Setting 1: MedMCQA (Local) $\\to$ HotpotQA | Setting2: NQ $\\to$ MedMCQA (Local) |\n| :------------------: | :--------------------------------------: | :--------------------------------: |\n|      KNN Router      |                  18.60                   |               63.33                |\n|          MF          |                  20.20                   |               58.89                |\n| **RAGRouter (Ours)** |                **21.20**                 |             **68.89**              |\n\nEvaluation results show that **RAGRouter still outperforms traditional LLM routing methods by 1.00% and 5.56% in the two cross-domain settings, demonstrating strong domain generalization.** This performance advantage can be attributed to **RAGRouter’s contrastive learning framework, which effectively captures relative RAG capability differences among candidate LLMs, even when transferred across domains.**"}, {"review_id": "NxnMiPQL5k", "reviewer": "Reviewer_s6eC", "summary": "This paper proposes RAGRouter, a routing framework for retrieval-augmented generation (RAG) models that selects the most suitable model for each query by explicitly modeling knowledge shifts introduced by retrieval. It combines document embeddings and RAG-specific representations through contrastive learning, achieving 3.61% higher accuracy on average compared to the best single model and 3.29%–9.33% improvement over existing routing methods across five knowledge-intensive tasks.", "questions": "Will the dataset used to train RAGRouter be open-sourced? Releasing it would greatly promote progress in this area.\n\nAnd please refer to the other questions in the weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. Writing is quite clear, and I can easily follow the author's idea and logic.\n2. Code is open-sourced, it's really convenient for following research.\n3. The figures and tables are very well-designed, and I really enjoyed reading this paper.\n\nWeaknesses:\n1. In lines 134–145, the authors mention three core factors, which motivate the design of different modules in lines 146–169. However, the correspondence between them is somewhat unclear. The authors are encouraged to clearly state the design motivation at the beginning of each paragraph in lines 146–169.\n2. The experiments did not include very strong closed-source models (e.g., gpt4o) or reasoning-oriented models (e.g., deepseek-r1) as the LLMs. What is the reason for this? Is it possible that these models are powerful enough on their own and therefore do not need RAGRouter?\n3. The current capability vector is set to a dimensionality of 768. Based on experience from recommender system literature, this dimensionality can significantly impact model performance. I suggest adding related ablation or hyperparameter experiments to study this effect.", "was_revised": true, "final_justification": "All my raised issues are resolved. Meanwhile, authors supplemented my questions with experiments on reasoning-based models and powerful closed-source LLMs, so I will maintain my current positive rating.", "rebuttal": "Thank you for your careful review and valuable feedback. We address all your comments below.\n\n> **Q1**: Clarify the correspondence between the three core RAG-aware factors and the design of different modules.\n\n**A1**: We clarify the correspondence as follows:\n\n- The factor **“the non-parametric knowledge provided by the documents”** corresponds to our **Document Encoder**.\n\n- The factor **“the LLM’s ability to process external information”** is captured by the **RAG Capability Embedding Layer.**\n\n- The factor **“the query’s role in guiding knowledge retrieval”** is represented by the **Cross Encoder.**\n\nWe will incorporate these clarifications into the revised manuscript.\n\n---\n\n> **Q2**: Include very strong closed-source models.\n\n**A2**: We have followed your suggestion to **add the powerful closed-source LLMs, including GPT-4o and Qwen2.5-Max**, to the original candidate LLM pool with parameter size $\\ge$ 32B. We then evaluate our RAGRouter on NQ and WebQ, comparing its performance against non–RAG-aware baselines, including KNN Router and MF.\n\n|           Method            |    NQ     |   WebQ    |    Avg    |\n| :-------------------------: | :-------: | :-------: | :-------: |\n|    Qwen2.5-32B-Instruct     |   44.58   |   50.42   |   47.50   |\n|    Qwen2.5-72B-Instruct     |   40.00   |   48.75   |   44.38   |\n|   Llama-3.3-70B-Instruct    |   51.67   |   50.42   |   51.05   |\n|         Qwen2.5-max         |   51.25   |   52.08   |   51.67   |\n| GPT-4o |   58.33   |   57.08   |   57.71   |\n|         KNN Router          |   57.08   |   50.42   |   53.75   |\n|             MF              |   57.08   |   55.83   |   56.46   |\n|    **RAGRouter (Ours)**     | **58.75** | **57.92** | **58.34** |\n\nEvaluation results show that although **GPT-4o is the top-performing individual model**, **our RAGRouter delivers an additional +0.63% improvement**, while **invoking GPT-4o in only 52.3% of samples**. **RAGRouter further outperforms Qwen2.5-Max (+6.67%), KNN Router (+4.59%), and MF (+1.88%)**. These results highlight RAGRouter’s effectiveness even in the presence of highly powerful LLMs, **leveraging model complementarity to surpass each single LLM, with better latency and cost efficiency by selectively using GPT-4o**.\n\n---\n\n> **Q3**: Experiments on capability vector dimensionality.\n\n**A3**: We have conducted **hyperparameter studies on varying capability vector dimensionality in Appendix E (Figure 9 and Table 9) of the submission, investigating how different embedding dimensions affect performance across five different datasets.** A summary of the results is shown below:\n\n| Dimension |  128  |  256  |  384  |  512  |  768  | 1024  |\n| :-------: | :---: | :---: | :---: | :---: | :---: | :---: |\n|    Avg. over 5 Datasets    | 63.00 | 63.65 | 63.43 | 63.72 | 64.46 | 63.29 |\n\nAs 768-dimensional embeddings yielded the best overall performance, we adopted this setting in the main experiments.\n\n---\n\n> **Q4**: Will the dataset used to train RAGRouter be open-sourced?\n\n**A4**: We will release the dataset used to train RAGRouter, along with detailed descriptions of its construction and usage instructions."}, {"review_id": "B62YVhrLgk", "reviewer": "Reviewer_SuEE", "summary": "This paper introduces RAGRouter, a novel framework for routing queries to the best language model in a Retrieval-Augmented Generation (RAG) setting. RAGRouter explicitly models three factors: (1) the semantics of retrieved documents, (2) query–document interaction via a cross-encoder, and (3) each model’s learned “RAG capability” embedding. Experiments on five knowledge-intensive QA benchmarks show a large performance improvement.", "questions": "1. How does the model perform on commercial strong LLMs, such as GPT4o?\n2. How well does RAGRouter generalize to other RAG tasks?", "limitations": "See weakness above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Solid empirical gains across five diverse QA datasets, with thorough ablations on each component.\n2. The motivation is solid and easy to understand.\n3. The proposed method is elegant and modularly presented.\n\nWeaknesses:\n1. The experiments seem to only consider open-source LLMs.\n2. The evaluation is mainly conducted on QA, and it remains unclear how well RAGRouter generalizes to other RAG tasks (e.g., summarization, dialogue)\n3. It requires ground-truth labels of “can answer vs. cannot answer”.", "was_revised": true, "final_justification": "I have read the rebuttal, which has fully resolved my concerns. I decide to keep my score positive.", "rebuttal": "Thanks for your thoughtful feedback on our paper. We have provided responses to all your concerns below.\n\n> **Q1**: Performance on commercial strong LLMs, such as GPT4o.\n\n**A1**: We have followed your suggestion to **add commercial strong LLMs, including GPT-4o and Qwen2.5-Max**, to the original candidate LLM pool with parameter size $\\ge$ 32B. We then evaluate our RAGRouter on NQ and WebQ, comparing its performance against non–RAG-aware baselines, including KNN Router and MF.\n\n|           Method            |    NQ     |   WebQ    |    Avg    |\n| :-------------------------: | :-------: | :-------: | :-------: |\n|    Qwen2.5-32B-Instruct     |   44.58   |   50.42   |   47.50   |\n|    Qwen2.5-72B-Instruct     |   40.00   |   48.75   |   44.38   |\n|   Llama-3.3-70B-Instruct    |   51.67   |   50.42   |   51.05   |\n|         Qwen2.5-max         |   51.25   |   52.08   |   51.67   |\n| GPT-4o |   58.33   |   57.08   |   57.71   |\n|         KNN Router          |   57.08   |   50.42   |   53.75   |\n|             MF              |   57.08   |   55.83   |   56.46   |\n|    **RAGRouter (Ours)**     | **58.75** | **57.92** | **58.34** |\n\nEvaluation results show that although **GPT-4o is the top-performing individual model**, **our RAGRouter delivers an additional +0.63% improvement**, while **invoking GPT-4o in only 52.3% of samples**. **RAGRouter further outperforms Qwen2.5-Max (+6.67%), KNN Router (+4.59%), and MF (+1.88%)**. These results highlight RAGRouter’s effectiveness even in the presence of highly powerful LLMs, **leveraging model complementarity to surpass each single LLM, with better latency and cost efficiency by selectively using GPT-4o**.\n\n---\n\n> **Q2**: How well does RAGRouter generalize to other RAG tasks?\n\n**A2**: Following your suggestion, we have supplemented experiments on **WikiASP [1], a summarization task commonly used in RAG-related research**. In particular, WikiASP aims to generate aspect-based summaries for Wikipedia entities and naturally supports the use of retrieved evidence. We adopt ROUGE-L as the evaluation metric.\n\n|        Method         |  WikiASP   |\n| :-------------------: | :--------: |\n| Qwen2.5-0.5B-Instruct |   0.1388   |\n| Qwen2.5-1.5B-Instruct |   0.1411   |\n| Llama-3.2-3B-Instruct |   0.1768   |\n|  Qwen2.5-3B-Instruct  |   0.1528   |\n|  Qwen2.5-7B-Instruct  |   0.1897   |\n| Llama-3.1-8B-Instruct |   0.1455   |\n|      KNN Router       |   0.1881   |\n|          MF           |   0.1865   |\n| **RAGRouter (Ours)**  | **0.1981** |\n\nEvaluation results show that **RAGRouter outperforms non-RAG-aware routing baselines by more than 5.3%**. These findings demonstrate that RAGRouter generalizes well to different RAG tasks.\n\n[1] Hayashi, Hiroaki, et al. \"Wikiasp: A dataset for multi-domain aspect-based summarization.\" Transactions of the Association for Computational Linguistics 9 (2021): 211-225.\n\n---\n\n> **Q3**: It requires ground-truth labels of \"can answer vs. cannot answer\".\n\n**A3**: We clarify that **RAGRouter doesn't rely on strict \"can answer vs. cannot answer\" discrete labels.** For **continuous performance metrics like ROUGE-L**, we can use a **score-based probabilistic sampling** strategy to generate positive and negative sample pairs for contrastive learning in our RAGRouter. **Specifically, for each candidate LLM's score $s\\in[0,1]$, we label it as \"can answer\" (positive sample, label 1) with probability $s$, and \"cannot answer\" (negative sample, label 0) with probability $1-s$.** We **have applied this strategy in our newly added experiments on the WikiASP summarization task using ROUGE-L scores**, and the results validate its effectiveness. This further highlights the flexibility of RAGRouter in handling tasks with continuous performance metrics, without strictly requiring binary supervision."}, {"review_id": "r2JTeCs5tG", "reviewer": "Reviewer_WwUD", "summary": "The paper titled \"RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models\" introduces a novel routing framework, RAGRouter, designed to optimize the selection of the most suitable Large Language Model (LLM) for a given query in the context of Retrieval-Augmented Generation (RAG). The authors address the challenge of varying response quality across different LLMs when augmented with external knowledge, proposing a method that leverages document embeddings and contrastive learning to capture dynamic shifts in model capabilities. The results demonstrate that RAGRouter significantly outperforms individual LLMs and existing routing methods, achieving superior performance and efficiency trade-offs under low-latency constraints.", "questions": "1. How sensitive is RAGRouter to the quality of retrieved documents? In scenarios with poor retrieval, does the method degrade gracefully, or does it significantly impact performance?\n2. Given the computational complexity of the training process, are there any plans to explore more efficient optimization strategies or lightweight architectures to improve scalability?\n3. How does RAGRouter compare with other recent routing methods that also aim to capture dynamic shifts in LLM capabilities? Are there any specific advantages or trade-offs?", "limitations": "1. While the method is novel in its application to RAG, it builds on existing techniques in contrastive learning and routing. Future work could explore more innovative approaches to capturing dynamic shifts in LLM capabilities.\n2. The training process, particularly the contrastive learning component, requires significant computational resources. This may limit the adoption of RAGRouter in resource-constrained environments.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a approach to routing queries in RAG settings, capturing dynamic shifts in LLM capabilities through contrastive learning. This method effectively addresses the limitations of static routing strategies.\n2. The authors conduct extensive experiments across diverse knowledge-intensive tasks and retrieval settings, demonstrating that RAGRouter consistently outperforms both individual LLMs and existing routing methods.\n3. The extended score-threshold-based mechanism allows RAGRouter to achieve strong performance-efficiency trade-offs, making it suitable for low-latency applications.\n\nWeaknesses:\n1. While the method is novel in its application to RAG, it builds on existing techniques in contrastive learning and routing. The innovation may be seen as incremental rather than groundbreaking.\n2. The effectiveness of RAGRouter relies heavily on the quality and relevance of retrieved documents. In scenarios with poor retrieval, the performance gains may be diminished.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We have addressed your concerns in detail below.\n\n> **Q1**: Technical innovation.\n\n**A1**: We would like to emphasize that **our RAGRouter is not merely a combination of existing techniques in contrastive learning and routing, but a novel framework driven by a previously unaddressed problem: RAG-aware dynamic LLM routing.**\n\n**We are the first to explicitly define the new RAG-aware LLM routing problem.** As RAG becomes a mainstream paradigm, **different LLMs vary significantly in how they incorporate and utilize retrieved, non-parametric knowledge**. However, **existing LLM routing methods typically assume static model capabilities and overlook the impact of retrieval context.**\n\nTo deal with this new problem, we design a new router network architecture that introduces **document encoder and RAG capability embedding layer** to model **how LLMs absorb and respond to retrieved evidence**. Further considering **the retrieval setting can substantially alter an LLM’s answerability by shifting the knowledge representation**, we propose **intra- and cross-setting contrastive learning objectives to capture the performance variations with and without RAG settings**. Ablation studies have also validated the necessity and effectiveness of each design module.\n\n---\n\n> **Q2**: Performance gains in scenarios with poor retrieval.\n\n**A2**: **We have reported and analyzed RAGRouter’s performance under noisy retrieved documents in Appendix C (Routing Performance under Noisy Retrieval) of the submission.** In particular, we add synthetic noise into the NQ, WebQ, and TriviaQA datasets. Take TriviaQA for example. The evaluation results under four conditions, including Golden Context, Relevant Noise, Irrelevant Noise, and Counterfactual Noise, are summarized as follows.\n\n|        Method        | Golden Context | Relevant Noise | Irrelevant Noise | Counterfactual Noise |    Avg    |\n| :------------------: | :------------: | :------------: | :--------------: | :------------------: | :-------: |\n|      KNN Router      |     96.67      |     78.33      |      83.33       |        86.67         |   86.25   |\n|          MF          |     95.00      |     76.67      |      80.00       |        80.00         |   82.92   |\n| **RAGRouter (Ours)** |   **98.33**    |   **86.67**    |    **90.00**     |      **88.33**       | **90.83** |\n\nWe can observe that **our RAGRouter consistently outperforms traditional LLM routing methods, with performance gains more than 4.58%, highlighting its strong robustness to retrieval noise**.\n\n---\n\n> **Q3**: Comparison with recent routing methods that also aim to capture dynamic shifts in LLM capabilities.\n\n**A3**: We have conducted a comprehensive literature search and found some concurrent works [1,2] that also explore routing strategies under dynamically shifting LLM capabilities. **These studies** typically focus on reasoning-oriented scenarios and capture **the internal knowledge of LLMs** (e.g., different CoT strategies and internal state transitions) to guide routing decisions.\n\nIn contrast, **our RAGRouter** focuses on a totally different setting: routing for **LLM with RAG**, where **capability shifts arise from the interaction between LLMs and external knowledge.** This introduces new factors, including varying document quality, model-specific sensitivities to retrieved content, and compatibility between models and documents, which were not found in existing reasoning-centric scenarios. RAGRouter models these new factors to enable effective routing in RAG.  It captures **how different LLMs respond to varying retrieved contexts through a new network architecture and contrastive learning framework**. In contrast, **existing LLM routing methods focus on reasoning dynamics and cannot represent or utilize external knowledge**, making them **not applicable for the new RAG routing setting.**\n\nIn summary, our RAGRouter can be viewed as a complementary and parallel approach to existing LLM routing methods, offering specific advantages in RAG scenarios, where external knowledge plays a central role in shaping model behavior.\n\n[1] Pan Z, Zhang K, Zhao Y, Han Y. Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection. arXiv preprint arXiv:2505.19435. May 26, 2025.\n\n[2] Zhang W, Qiao S, Luo L, Li Y, Zheng C, Xu Q, Li M, Gui Y, He Y, Qiu J, Hong J. SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model. arXiv preprint arXiv:2507.02822. Jul 3, 2025.\n\n---\n\n> **Q4**: Training efficiency of RAGRouter.\n\n**A4**: We clarify that **RAGRouter and all other LLM routers are trained on resource-rich cloud infrastructure**. Despite this, **RAGRouter remains lightweight, with just 136M parameters**. It was trained using **a single consumer-grade NVIDIA RTX 4090D GPU**, achieving an average training time of **11.6 minutes on PopQA and MedMCQA**, and **4.5 minutes on NQ, WebQ, and TQA**. The **peak memory usage** during training was only **18.9 GB**."}]

Author rebuttal:

Meta-review: This paper introduces RAGRouter, an approach for routing queries to the most suitable LLM in a RAG setting.

The reviewers noted the following strengths of the paper:
* The proposed approach is novel. It is the first to define and address the problem of LLM routing for RAG
* The proposed architecture is technically sound and well-motivated. It features an innovative learnable RAG capability embedding for each target LLM and a unique contrastive learning objective aimed at capturing shifts in knowledge
* The experimental evaluation is extensive and clearly demonstrates the strong effectiveness of the proposed approach across a diverse range of knowledge-intensive tasks
* The framework has a practical score threshold mechanism that makes it possible to tradeoff between quality and latency

The reviewers also identified a number of weakness with the paper, including:
* The overall innovation could be viewed as incremental, as it builds on top of existing contrastive learning and model routing approaches
* The experiments reported in the initial version of the paper were limited in scope, lacking evals using SOTA models like GPT-4o and did not include non-QA tasks

The authors submitted an exceptionally thorough rebuttal. Following the rebuttal, the most engaged reviewers confirmed their favorable view of the paper, which moved the paper from a borderline case to a clear consensus. 

Overall, this is a high-quality paper that puts forth a strong solution to a novel and highly timely problem. It should therefore be accepted.
Decision: Accept (poster)