ICML 2025

Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations
Evan Frick∗1 Connor Chen∗1 Joseph Tennyson∗1 Tianle Li∗1 Wei-Lin Chiang∗1
Anastasios N. Angelopoulos∗1 Ion Stoica 1
Abstract
Large language model (LLM) evaluations typi-
cally rely on aggregated metrics like accuracy
or human preference, averaging across users
and prompts.
This averaging obscures user-
and prompt-specific variations in model perfor-
mance.
To address this, we propose Prompt-
to-Leaderboard (P2L), a method that produces
leaderboards specific to a prompt or set of
prompts. The core idea is to train an LLM taking
natural language prompts as input to output a vec-
tor of Bradley-Terry coefficients which are then
used to predict the human preference vote. The
resulting prompt-dependent leaderboards allow
for unsupervised task-specific evaluation, opti-
mal routing of queries to models, personalization,
and automated evaluation of model strengths and
weaknesses. Data from Chatbot Arena suggest
that P2L better captures the nuanced landscape of
language model performance than the averaged
leaderboard. Furthermore, our findings suggest
that P2L’s ability to produce prompt-specific eval-
uations follows a power law scaling similar to that
observed in LLMs themselves. In January 2025,
the router we trained based on this methodology
achieved the #1 spot on the Chatbot Arena leader-
board. Our code is available at this GitHub link:
https://github.com/lmarena/p2l.
1. Introduction
Evaluating the real-world performance of large language
models is an unresolved challenge. A growing suite of
benchmarks, including MMLU (Hendrycks et al., 2020),
MMLU-Pro (Wang et al., 2024), and GPQA (Rein et al.,
2023), seek to address the challenge by reporting task-
specific performance metrics, such as multiple-choice
*Equal contribution 1University of California, Berkeley. Corre-
spondence to: Evan Frick <evanfrick@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267...