ICML 2025

BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute
Dujian Ding 1 Ankur Mallick 2 Shaokun Zhang 3 Chi Wang 4 Daniel Madrigal 2
Mirian Del Carmen Hipolito Garcia 2 Menglin Xia 2 Laks V. S. Lakshmanan 5 Qingyun Wu 3 6 Victor R¨uhle 2
Abstract
Large language models (LLMs) are powerful
tools but are often expensive to deploy at scale.
LLM query routing mitigates this by dynamically
assigning queries to models of varying cost and
quality to obtain a desired tradeoff. Prior query
routing approaches generate only one response
from the selected model and a single response
from a small (inexpensive) model was often not
good enough to beat a response from a large (ex-
pensive) model due to which they end up overus-
ing the large model and missing out on poten-
tial cost savings. However, it is well known that
for small models, generating multiple responses
and selecting the best can enhance quality while
remaining cheaper than a single large-model re-
sponse. We leverage this idea to propose BEST-
Route, a novel routing framework that chooses
a model and the number of responses to sample
from it based on query difficulty and the quality
thresholds. Experiments on real-world datasets
demonstrate that our method reduces costs by up
to 60% with less than 1% performance drop.
1. Introduction
Large language models (LLMs) have revolutionized natural
language processing (NLP) by delivering state-of-the-art
performance across a wide range of tasks, from language
understanding to creative writing, code generation, and be-
yond (Zhao et al., 2023). Their widespread deployment in
applications like ChatGPT (OpenAI, a) and other conversa-
tional agents (Zheng et al., 2023; Zhang et al., 2024a;b) has
made them a cornerstone of modern NLP systems. However,
the superior performance of these models often comes with
1University of British Columbia (work performed during in-
ternship at Microsoft) 2Microsoft 3Pennsylvania State Univer-
sity 4Google (work performed while at Microsoft...