NeurIPS 2025

Lookahead Routing for Large Language Models
Canbin Huang1, Tianyuan Shi1, Yuhua Zhu1, Ruijun Chen1, Xiaojun Quan1,2,∗
1School of Computer Science and Engineering, Sun Yat-sen University, China
2Shenzhen Loop Area Institute, China
{huangcb3, shity6, zhuyh53, chenrj8}@mail2.sysu.edu.cn
quanxj3@mail.sysu.edu.cn
Abstract
Large language model (LLM) routers improve the efficiency of multi-model sys-
tems by directing each query to the most appropriate model while leveraging the
diverse strengths of heterogeneous LLMs. Most existing approaches frame routing
as a classification problem based solely on the input query. While this reduces
overhead by avoiding inference across all models, it overlooks valuable information
that could be gleaned from potential outputs and fails to capture implicit intent
or contextual nuances that often emerge only during response generation. These
limitations can result in suboptimal routing decisions, particularly for complex
or ambiguous queries that require deeper semantic understanding. To address
this challenge, we propose Lookahead, a routing framework that “foresees” po-
tential model outputs by predicting their latent representations and uses these
predictions to guide model selection, thus enabling more informed routing without
full inference. Within this framework, we implement two approaches based on
causal and masked language models. Empirical evaluations across seven public
benchmarks—spanning instruction following, mathematical reasoning, and code
generation—show that Lookahead consistently outperforms existing routing base-
lines, achieving an average performance gain of 7.7% over the state-of-the-art. Our
code is available at https://github.com/huangcb01/lookahead-routing.
+13.4
+8.6
+20.8
+13.1
AlpacaEval-2
Arena-Hard
MT-Bench
GSM8K
MATH
HumanEval
MBPP
18
28
38
12
20
28
4
19
34
10
20
30
32
35
38
60
64
68
51
52
53
Methods
SMOOTHIE
RouterDC (SmolLM2)
RouterDC (ModernBERT)
Lookahead (SmolLM2)
Lookahead (ModernBERT)
Task categories...