Title: Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

LLM Routing

Cost-Aware Inference

Online Algorithm

Multi-LLM Serving
Abstract: Increasing demand for Large Language Models (LLMs) services imposes substantial deployment and computation costs on providers. 
LLM routing offers a cost-efficient solution by directing queries to the optimal LLM based on model and query features.
However, existing works primarily focus on offline scenarios and struggle to adapt to online settings with high query volume and constrained token budgets.
In this work, we introduce the first training-free algorithm for online routing scenarios.
Our algorithm leverages approximate nearest neighbor search to efficiently estimate the features of queries and performs a one-time optimization over a small set of initial queries to learn a set of routing weights that guide future routing.
We provide a theoretical guarantee that the algorithm achieves a competitive ratio of $1 - o(1)$ under natural assumptions, which is further validated by extensive experiments across 3 benchmark datasets and 8 baselines, showing an average improvement of 3.55$\times$ in performance, 1.85$\times$ in cost efficiency, and nearly 4.25$\times$ in throughput.
Our code is available at https://github.com/fzwark/PORT.


Peer reviews:
comments: SUMMARY:
This paper proposes a training-free, online routing algorithm to efficiently handle high-volume, budget-constrained LLM inference requests. Unlike prior methods that rely on heavy offline training or dynamic retraining, the proposed method:
1. Uses Approximate Nearest Neighbor Search (ANNS) to estimate query-level performance and cost features from historical data.
2. Learns a single global routing strategy (γ)* based on a small subset of initial queries.
3. Applies this learned strategy for online query routing using a principled dual formulation of a relaxed MILP.

It demonstrates strong theoretical guarantees (competitive ratio of 1-o(1) and empirical results showing up to: 3.55× performance, 1.85× better cost-efficiency, 4.25× higher throughput over 8 competitive baselines on 3 benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This is the first routing method for online multi-LLM serving that is training-free, addressing the scalability bottlenecks of prior methods in realistic deployments.
2. Solid formulation of routing as a MILP and its LP relaxation.
3. Compared against 8 baselines across multiple metrics and settings.

Weaknesses:
1. While the paper claims low-latency routing, it doesn’t provide absolute latency measurements (e.g., milliseconds per query) or breakdowns.

QUESTIONS:
1. How sensitive is the routing performance to the quality and size of the historical dataset?
2. How often should γ∗ be re-learned in a real system?
3. Could reinforcement learning improve the second-stage routing?
4. Can the proposed method be applied for VLM (vision language model)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
This paper introduces a training-free routing strategy by leveraging the log of historical prompts and associated generations as well as their quality scores and costs. Specifically, it conducts semantic embedding based nearest neighbor search to retrieve similar past queries and estimate the performance by averaging recorded measures. Then, it combines estimated quality and cost of each model using learned and fixed weights to decide the final pick of the model.

STRENGTHS AND WEAKNESSES:
Strength:
- Comprehensive evaluation on three benchmarks.
- Extensive tests over query volume, arrival order, size of historical data, size of search candidates, etc.
- Code and data are released for reproducibility.

Weakness:
- The effectiveness of the proposed method depends on reliable quality labels. The experiments mainly use the pre-existing ground truth and lack of robustness evaluation against noisy quality labels.
- The baseline methods are somewhat weak. Some of the comparable works are discussed in the related work sections but are not empirically compared in the evaluation section. 
- The theoretical proof relies on strong assumptions including the existence of at least one close neighbor in the log and that generation quality and cost are smoothly tied to prompt embeddings. Similarly, evaluation is mainly conducted with an in-distribution setting, and it is unclear if the proposed method is robust enough to handle the out-of-distribution challenge and drifting query patterns in dynamic online settings.

QUESTIONS:
What is the query-level latency? How does it compare to other routing strategies including trained lightweight routers?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
Existing works for LLM router primarily focus on offline scenarios and struggle to adapt to online settings with high query volume and constrained token budgets.

This paper introduces the first training-free algorithm for online routing scenarios. The algorithm leverages approximate nearest neighbor search to efficiently estimate query features and performs a one-time optimization over a small set of initial queries to learn a routing strategy that guides future routing.

The algorithm estimates the performance scores and costs efficiently using a historical dataset, and learns a routing strategy from a small subset of observed queries (of size ϵ|Q|) to guide future query routing.

Experiments on simulated traces show the performance.

STRENGTHS AND WEAKNESSES:
Strengths

1. Improving the efficiency of LLM routing is an important topic.

2. The proposed method with approximate nearest neighbor search and online learning is helpful.

3. Theoretical analysis and experiments are provided.

Weaknesses

1. Such approximate nearest neighbor methods highly rely on the training data. E.g. Routerbench has 36,497 examples. The paper uses 26,497 queries as historical data to construct the search examples and 10,000 queries for testing.

2. Stronger model-based methods like BERT can be added. The router running time of these methods can also be compared.

3. Experiments are only on simulated traces.

QUESTIONS:
1. Move Table 1 to the next page.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a training-free algorithm for optimizing the routing of user queries to multiple LLMs in online settings with constrained token budgets. The algorithm leverages a small subset of queries to builds an offline ANNS indice for estimating performance and cost for\neach deployed LLM, and performs a one-time optimization over a small subset of queries to learn weights for routing. These weights are utilized for guiding routing decisions of online queries, ensuring high performance and cost efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nAdvantages:\n\n1. The performance of the method is good. It achieves an average improvement of 3.55× in overall performance, 1.85× in cost efficiency, and nearly 4.25× in throughput.\n\n2. The method does not require training or frequent finetuning existing models.\n\n3. The method looks not very sensitive to the distributions of user queries. Even if the query distribution might shift in online setting, it only requires an update on the HNSW index by inserting the new query data. And this process can be performed in an offline manner.\n\n4. The writing is clear.\n\n5. The evaluations are comprehensive.\n\nWhat I like most about this paper is that their method is simple but effective. It does not make strong or unrealistic assumptions and is suitable for real-world deployment. My major concern is, the method only considers cost and efficiency as features for model selection. However, in a real-world setting, the traffic of queries is also important. If a high volume of queries arrives at a same time and they are routed to a same LLM, there might be a traffic congestion. \nFor example, if a large number of users ask similar questions in a short amount of time, and the HNSW index suggests to route all the queries to the same LLM, the inference efficiency would be very low. However, at the same time, the other LLMs are idle. \nAlso, the arrival time of different queries and the sequences of different queries matter. \nI understand that solving this problem isn't straightforward, and the linear programming might not be suited. However, the problem is practical and important in a real-world setting.\n\nQUESTIONS:\n1. What parameters have you set for HNSW? I only saw the number of neighbors in the appendix. However, the number of layers of HNSW is not specified. The parameters of HNSW might impact the evaluation results. Could you add evaluations for that? \n\n2. See weakness. \n\n3. Should be HNSW not HWSN throughout the whole paper.\n\nLIMITATIONS:\nThe method only considers cost and efficiency as features for model selection but does not consider traffics of the queries. The authors mentioned to add additional constraints in the original MILP for incorporating other routing factors. However, it is not accurate as incorporating sequences of similar types of queries is not straightforward using MILP.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}