Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations
Authors: Evan Frick
Connor Chen
Joseph Tennyson
Tianle Li
Wei-Lin Chiang
Anastasios Nikolas Angelopoulos
Ion Stoica
Venue: ICML 2025 poster (2025)
Keywords: llm
leaderboard
evaluations
Primary area: general_machine_learning->evaluation

Abstract: Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt or set of prompts. The core idea is to train an LLM taking natural language prompts as input to output a vector of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependent leaderboards allow for unsupervised task-specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce prompt-specific evaluations follows a power law scaling similar to that observed in LLMs themselves. In January 2025, the router we trained based on this methodology achieved the \#1 spot on the Chatbot Arena leaderboard. Our code is available at this GitHub link: https://github.com/lmarena/p2l.

Prompt-to-Leaderboard:: Prompt-Adaptive LLM Evaluations
Evan Frick Connor Chen' Joseph Tennyson"
Tianle Li Ion Stoica
Wei-Lin Chiang"
Anastasios N. Angelopoulos"
Abstract
question-answeriigg ability. These highly-curated bench- marks focus on Nomain-specifie performance measures but do not capture the general and subjective nature of or- ganic human preferences. Live evaluations, such as Chathot Arena (Chiang et al., 2024), assess real-world performance by collecting millions of organic human preferences from users who visit the site and vote between pairs of model re- sponses. These pairwise comparisons are aggregated using Bradley-Terry (BT) regression (Bradley and Terry, 1952) to form leaderboard over LLMs. This leaderboard aver- ages over many users and prompts, only providing a coarse understanding of performance.
Large language model (LLM) evaluations typi- cally rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model perfor- mance. To address this, we propose Prompt- to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt or set of prompts. The core idea natural language prompts as input output vec- tor of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependen leaderboards allow for unsupervised task-specific evaluation, opti- mal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce -pemit-specific eval- uations follows power law scaling similar to that observed in LLMs themselves. January 2025, the router we trained based on this methodology achieved the #1 spot on the Chatbot Arena leader- board. Our code is available at this GitHub link: https:/ //g thub. com/ / ma rena / 1p
train an LLM taking
For example, we want to identify the best model for SQL queries, the overall Chathot Arena leaderboard may not be useful since SQL queries make up only 0.6% of organic submissions and thus have little influence in the ranking. A natural solution is to stratify the data and run a separate BT regression for SQL queries. However, collecting the 3,000-5,000 SQL votes needed for stable ranking would require around million total votes- tokes-taking months to col- lect. Finer-grained categories, for example SQL table joins, would demand even more data, making stratified regression impractical and slow. And the finest-grained analyses-for example, producing leaderboards for a specific prompt or use-case-are rendered impossible.
This manuscript proposes a solution to this problem via a method called Prompt-to- Leaderbbrbr (P2L). P2L takes a prompt as input and outputs leaderboard quantifying LLM performance on that specific prompt. Thus, P2L can be used to assess which models are best for specific use-case, as opposed to on average. Per-prompt leaderboards can also be aggregated over a group of prompts to form personalized leaderboards, showing which model is best for an individual or enterprise based on their prompt history.
1. Introduction
Evaluating the real-world performance of large language models is an unresolved challenge. A growing suite of benchmarks, including MMLU (Hendrycks et al., 2020), MMLU-Prr (Wang et al., 2024), and GPQA (Rein et al., 2023), seek to address the challenge by reporting task- specific performance metrics, such multiple-choice
The system works by training P2I. model, which is LLM trained on human preference feedback to output Bradley- (BT) coefficient for every model in question; see Section 2.1. Because P2L characterizes the prompt- conditional win rate of any two models, it enables several downstream applications. These include optimally routing prompts LLMs (Section 2.1.2), personalized evaluations based user's prompt history (Section 1.1.)), automated
'qqual contribution 'University California, Berkeley. Corre- spondence to: Evan Frick <evanfrick berkeley.cdu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Prompt-te-L.cdderbbardd Prompt-Adaptive LLM Evaluations
strength and weakness analysis of models (Section 3.4), and more. Thus, view P2I. as a general-purpose tool for highly granular evaluations extracted from large corpuses of preference data. As demonstration of P2L's utility, we tested our prompt routing strategy on Chatbot Arena and it achieved the #1 spot with a score increase of 25 points over the previous top model. momiini-xp---ee (see "P2L router performance" in Figure 1).
Each model m E [M] has a coefficient (2) m. and the higher this coefficient is, the more likely model m beats any other model on the prompt z. For different prompts, the leaderboard will be different, capturing the idea that different models are better on different prompts. Our target, 0r, is precisely the function that takes prompts and out- puts leaderboards- the name, (P2L).
between the dates
P2L is strict generalization of marginal BT regression. In marginal BT regression, we simply omit the dependence of the leaderboard on the prompt, and give the best leaderboard on average ("marginally"). That is, choosing θ to be the class of constant functions (()) = A E IR exactly recovers marginal BT regression.
More broadly, P2L is subclass of a more general methodol- ogy we call Prompt-to- -eggeressin (P2R) for training LLMs to output coefficients of parametric statistical regressions (see Section 2.2). A canonical example that we will develop throughout this paper is model taking prompts as input and outputting Bradley-Terry coefficients, as mentioned earlier. However, the method also accommodates other feedback models (ties, real values, etc.) via other parametric mod- els. We describe this method and derive the optimal routing strategy in Section 2. We show experiments and other appli- cations in Section 3.
However, P2L can be substantially more powerful than marginal BT regression due heterogeneity in the prompt- conditional performance of different language models. That is, we should leverage language models to extract infor- mation on model performance from the prompt. In par- ticular, our work takes @ to be a space of reward mod- els mapping prompts to vectors. Given training dataset Dirain = {(x,, Yi, ) we find the empirical risk min- imizer,
2. Methods
We describe the P2L method formally, beginning with no- tation. Consider M different LLMs which are presented to humans pairwise- moirwis-mmdel A on the left, and model B on the right, where A and B are randomly sampled without replacement from [M]] = [l....,,]] If the human votes for model A, we set Y 0, and if they vote for model B, set Y =1. Furthermore, we let X represent a 'two-hot' encoding of the model pair, i.e., vector of length M with zeros everywhere except +1 in the index B and 1 in the index A. We model our data- mata-gennrating process as a tuple (X, Y, Z) of two-hot encodings, votes, and prompts Z E 2 sampled from a joint distribution P, where Z denotes the space of natural-language prompts. Also, let Q denote a space of functions mapping prompts to leaderboards, i.c.,
N
1 N
i = aargmmn dEe
(()))
(1)
Then, as before, we can extract the estimated win rate be- tween any two models as
P(Y = = = == o =====(=)))
Lastly, we note that this strategy of training LLMs to output coefficients of parametric statistical models will be general- ized in Section 2.2. The resulting prompt-dependent models have both high predictive power and a useful statistical in- terpretation, which is critical to the aforementioned routing and personalization techniques.
0 E θ is function from 2 + RM,, and ((2); represents the leaderboard score of model i E [M] given prompt Finally, let l denote the binary cross-entropy loss and a denote the sigmoid function.
2. 1. AGGREGATING LEADERBOARDS
2.1. Core method
Conceptually, our method works as follows. We model the vote conditionally on the prompt and model pair as following a Bradley-Terry (BT) model (Bradley and Terry, 1952):
Many practical scenarios require a leaderboard for a dis- tribution over prompts, not just one. For example, a user may want to know which model is best for them based on their chat history, or an enterprise may want to know which model is best for their use-case. In other words, given distribution over prompts Q, we want to ensemble all 0" (=)) for E Z to form leaderboard over Q. In the case of finite chat history, we can consider Q to be the discrete uniform distribution over the observed historical prompts.
P(Y = 1| = x, 2 = 2) = olx ( (2)),
for some (unknown) 0" 2 + R The goal is to approxi- mate 8t from data.
For any prompt 2 E 2, (2) represents a leaderboard.
By the Tower property, we can decompose the win rate as
2

---

Prompt-te-L.addrbearr. Prompt-Adaptive LLM Evaluations
P2L performanos
P2L
Quatere
p.pprompt
Drsperi 13
1250
Detaledd analysaa
ChacGP--lo
113
Pilla
Figure 1: Pipeline of P2L.. P2L takes prompt or a set of prompts and outputs an M-dimensional vector that we call leaderboard. Once we have leaderboard, we can build better data products, like routers and automatic analyses (see right),
Thus, we can bypass the need for sampling to simulate Y. In other words, (2) is equivalent to
Be.ee...-BeeeoxTeTeeeeoeo X =
i(Q) argmin Ex [e (a(X o)
= =f JZEz
(z))) dQ(:).
BE
This last expression is simple to compute for discrete distri-
The win rate above no longer follows a simple logistic model, but we can fit another logistic model to match it:
butions Q, leading to an efficient algorithm.
2.1.2. OPTIMAL ROUTING
Ō(Q)= = argmin lee (x (2)))
X~ Z-Q. [e(a(X 0), Y)] (((((T)))
Next, will derive the optimal router based P2L. We will derive the exact optimal router based on 0" and approx- imate it in practice by Ó. Let us assume, for the sake of simplicity, that for each model m E [l,...,M,, there is known and fixed cost of inference, c = (CI, CM). We seek to create router that maximizes performance while remaining below constraint the average cost, C. We express the router policy, Z + A which takes prompt as input and outputs a distribution over models; we seek to estimate the optimal policy. π.. We will also consider a distribution of opponent models, E A M to act as a baseline for comparison. For instance, we can pick to be point-mass on the single best model, to be uniform over all [M] models.
(2)
The idea that, because we know P(Y =
2) = (z) for x and 2, we can simulate the data- generating process. This allows us to construct synthetic dataset and fit Bradley-Tere model to it. If 0 exists, this technique is perfect, in that it recovers the exact same BT coefficients that we would have obtained by observing infinite population of prompts from Q. In Appendix B. we explore an alternative leaderboard aggregation strategy by taking a weighted average of the leaderboards. Note also that we use 0", with the understanding that in practice we will use the plug-in estimate based on 0, and the resulting rule will be approximate.
We can make this strategy more efficient by leveraging the linearity of the binary cross-entropy loss. Namely,
One possible interpretation of "optimal" router is the one that maximizes the win rate against subject to the cost constraint; that is, for almost every x, this interpretation of " TT(2) solves the following optimization problem:
E Ex~Px. 2~ B [ (a((X 0), Y)]
=Ex~Px.2-0 [Ey Bern(o
maximize REAM
P PA-q. A.a.BBwR, Briit Y - Ben
[e(o(X 0), Y) [X, 2]]
= =Ex-rx ( 0),
(Y Z 2)
(4)
Sy=Bem(o((TT<112) 21)
subject to EB [cb] S C
Ex 2- [ (0(x T0), To (2)))]
In other words, the optimal router should maximize the
3

---

Prompt-te-L.addebbaar.. Prompt-Adaptive LLM Evaluations
average win rate against the opponent distribution 4-
Algorithm Optimal routing with BT estimate
An alternative definition of the optimal router is the one that has the highest Berry coefficient. This version of the optimal policy has π (z) equal (almost surely) to the solution to the following optimization problem:
Input: q W 8* (*)y; 1: Solve the LPP
argmax # 7
REAM #TeSO
Compute R n T
3: Solve for 0' by finding the root of the following im- plicit equation:
maximize REA
argmin IE
A~vg
PER
B'wBern( (=)) (=)a-0((s)A))
[e(o(o- - (0(0--0(2))) X²) ==
(5)
00 = R
subject to EBwalcB] SC
Output: Optimal router म", estimate of router's BT coeffi- cient 6*
That is, considering the optimal router as a separate model, it should achieve the highest possible spot in the leaderboard subject to the cost constraint.
2.2. Prompt-to-Regresee
Surprisingly, although the optimization problems in (4) and (5) look different, their optimal solution is the same under the Bradley-Terry model. The solution is given in Theorem 1. The resulting problem has a linear objective and linear constraint, and can be solved with any standard solver. If the dominant model is below the cost of C, the policy will deterministically select that model (i.e., it will place probability 1 on sampling that model). Otherwise, it will hedge its bets and randomize over multiple models.
Here, we give extensions of P2L beyond pairwise prefer- ence feedback. This is useful because, in Chatbot Arena, the voting options are not just "A is better" and "B is better"; they also include "Tie" and "Tie (both bad)". Thus, a P2L model that takes into account all this additional data may learn faster and also learn interesting signals about which prompts hard and cause models to exhibit different be- haviors or failures. Fortunately, our toolkit generalizes to the case where X is no longer two-hot encoding and Y is no longer binary. In fact, our strategy encompasses any parametric statistical model relating X and Y conditionally on Z, regardless the space in which they live. We call this models.
Theorem (Optimal prompt-dependent routing). Assume that for every prompt the Bradley- -eery model holds with coefficients 8* (z). Then, the optimization problems in (4) and (5) are both equivalent to the following problem:
more general class of models
More formally, let model the distribution of Y by saying that for all putative values y.
minimize #E
- - W'g -Wa
PY=g|Z=2,X==00 = g0 (1)
subject to
# T c 5 C,
(6)
Oa 3 म = 1M
for some (unknown) vector of parameters 6* (z). Then, we fit 0(=) by running maximum-likelihood estimation, i.c.. maximizing
म T 1M = 1,
n
II go(2.) (Y;; Xx)xx As familiar ex- 1=1
where W represents the population win matrix, with en- tries W ba = a (2)) (z)a).
ample, we can set (x) to BT model relating X and Y:
((xTT((2)) 1- oxx oxxTo(
y = 1, y y=0 0.
(x)=)
The proof is given in Appendix A. It is important to note that deviations from the Bradley-Terry model-for for example, any non-transitivity-wiii break this relationship.
Note that the formulation of (7), and X can be arbitrary, long model their conditional relationship via go(=)- Thus, the framework can admit real-valued feedback Y via ordinary least squares, count feedback via Poisson regres- sion, and so on.
Another benefit of this approach is that we able to esti- mate the value of the objective function of (5) via a standard root finder (Brent, 1973), which means we can estimate the router's ' position on the leaderboard before deploying it. We give this procedure in Algorithm 1. It is justified by (9) in the proof of Theorem 1.
As one example, we will consider incorporating ties via a Rao- Rao-Kupper (Rao and Kupper, 1967) model. Let X be a
4

---

Prompt-to-L.caderbbarrr: Prompt-Adaptive LLM Evaluations
two-hot encoding, Y E (A, B, tie], and
optimal cost-efficient routing via the algorithm developed previously in Section 2.1.2. In Section 3.4, we use P2I. to automatically identify strengths and weaknesses for dif- ferent models. In Section 3.5, we explore our aggregation technique against ground truth categories leaderboards, and observe data scaling trends. Finally, in Section 3.6, we show that the P2L has reasonable performance on out-of- distribution data.
o((z,- 0=)
B,
= A,
(0)(=)(1
((x, 01)T0(=))
tie.
In this technique, (2) is an (M + di-deeensia vec- tor, the last entry of which encodes a tie coefficient. The larger this prompt-dependent tie coefficient, the more likely the two models are to tie. Meanwhile, the first M entries, 12)) comprise the leaderboard.
3.1. Training setup
To train a P2L model, we follow this three-step procedure: (1) Begin with a pre-trained, instruction tutionnn-teedddd LLM. (2) Re- move the existing language model head and replace it with randomly initialized coefficient head. In the BT case, the co- efficient head is a linear layer producing M outputs, one per model. (3) Train the model by running stochastic gradient descent on all parameters to minimize the negative log-
Finally, we consider how to handle the "Tie (both bad)" category. For this, we developed a non-standard statistical model which we call the grounded Kao-Kupper model. In this model, if both model coefficients are small, it increases the probability of "Tie (both bad)". Inspired by the Plackett- Luce model (Plackett, 1975; Luce, 1959), we imagine the existence of a fictitious "bad"" model with a coefficient of zero, and use this as a grounding point for the model coeff- cients.
likelihood: L()) > log (g0(2.) (Y:) The result
of this procedure is the trained model Ô = argmingge (()), which a direct generalization of (1).
We train up to man preference pairs from Chatbot Arena, containing M = 130 unique models. Note that we find mini- mal left/right positional bias from voters. We always train for epoch. In order to study the scaling laws of P2L as a function of model size, we used the fol- lowing models as the initializations: 360 -M0MMIInstrus and Qwen2. 1een2.5-100 50..5 1.5, 3, 7 BJB-Instruct (Allal et al., 2024; Team, 2024). We refer to our post-trained versions of these models as P2L- 360 JM and 22L-10.5,12 7 re- spectively.
1.5 million crowdsourced hu-
Let Y E B, tie, bad], and for the sake of notational convenience, let (z)
(z), where (z) E RM and (2) E R21}. }. For notational convenience, we define 40" (z); == exp( (8 (z):). The grounded Sup-Kr
model is defined as:
pe (2)
A
(3) + ())xt())+t
(2)e
= B
(3)B (2")
bad
(=)(())
Ite (s)x
p (=)A
x (())
y = tie.
3.2. Feedback prediction
(B)
(8) This model allows us to make efficient use of all data col- lected on Chatbot Arena by incorporating all votes. It also has the additional advantage that models with higher coeffi- cients have lower probability of being labeled "Tie (both bad)". Thus, the raw coefficient value of a model speaks to its absolute quality, as opposed to its comparative quality against other LLMs as in the BT model.
We begin by evaluating P2L on its ability to predict human feedback on a prompt-by-promp basis. In other words, given two models and a prompt, we ask how effectively P2L can predict which model will win on that prompt. These experiments measure the ability of P2L to accurately assess relative model quality prompt-by -ooooppppt basis.
In this section, we evaluate the ability of P2I. to predict hu- man preferences on Chatbot Arena. We construct a holdout validation set containing 41,507 annotated pairwise com- parisons across 34 well-used models. We then measure the negative log- likeeiidldood (validation loss) on this dataset; a lower validation loss indicates better preference prediction performance.
3. Experiments
This section contains a suite of experiments that validate the P2L method and demonstrate its utility. In Section 3.2, we show that P2I. leads to gains in human preference pre- diction that scale with model size and data. In Section 3.2, we show direct predictive performance on pairwise human preferences, as well as scaling behavior with data size and parameter count, In Section 3.3, we show P2L. allows for
Figure 2 shows the results of our procedure against two baselines. First, we include the constant predictor that gives an equal probability of all preference outcomes; this is an
5

---

Prompt-te-L.addrbearr.. Prompt-Adaptive LLM Evaluations
1.35
1000
Velidation loss
OvamT arna unve
1880
1.30-
1980
1.25
1310
1.20
1060
1.15
1850
250k
J00K
IM
L.5M
Training points seen
1910
P2L-7B
P31-0
Pa SB
PaLasB SB
Gomlnkeap- E316
Meaa squared error
0.55
Validation loss
Wars: Want-ola -2----00 areni acoe in cidggoyy
L3-
1980
0.50
1960
1.24
0.45
1340
Coustant
P2L (0.5B)
1000
Marginal
P2L (1.5B)
PML-70
Pa 20
Pat- 1B
Pala:50
Gunlsing-/22 136
P2L (135M)
P2L (3B)
P2L (360M)
P2L (7B)
Figure 2: Loss metrics. The line plot shows the valida- tion loss as function of the number of data points seen during training. The P2I. models all substantially outper- form the baselines, and performance scales with dataset and model size. The bar plots show the validation loss and mean squared error of the models trained on all 1.5M training points. A table for accuracy can be found in Appendix E.2.
Figure 3: P2L router performance on Chatbot Arena. The left barplot shows the overall score of router after was deployed prospectively on Chatbot Arena. The right barplot shows the worst-case category score Chatbot Arena. Overall, larger models lead to higher Arena scores, i.e., better routers. The exception is P2I-1 101,,15B,, 5B, which has a large bump in overall performance. However, the confidence intervals indicate that this bump explainable by statistical variations its BT coefficient estimate.
extremely weak baseline akin to flipping a coin to decide the winner. Second, we include the average ("marginall) leaderboard. For P2I., we show a ladder increasing model and dataset sizes. The more data used to train P2L, the better the preference predictions become. Notably, the gap between the best P2L leaderboard and the marginal model is several times the gap between the marginal leaderboard and the constant predictor. This indicates that by capturing the prompt-depender differences in model performance, P2L is able to produce much better predictions of human preference.
Chatbot Arena, crowdsourcing a total of 8,616 pairwise comparisons between P2L models and public models hosted Chatbot Arena. The P2L models routed between 34 mod- els, including top models such as Gemini cemmini-xxp-2200 o 01-2024-12-20 well as other models. (See Appendix E.4 for a full model list.)
and ChatGPT-40-202411
as
Because there no cost-constraint, the P2L router al- ways picks the highest-ranked model conditionally on the prompt, i.e., the highest entry in 0(2). Marginally, the strongest singular candidate model in the P2L router was mamini-exp-1225 with a score of 1364.
3.3. Optimal routing
Next, we evaluate the performance of the optimal router based on P2L as derived Section 2. 1.2.2. Our evaluations are based on prospective deployments of our router Chat- bot Arena. We treat the router separate model. For all deployments of routers, collect blind pairwise com- parisons against all active public models hosted Chatbot Arena in process identical to how standard models are added to the Chatbot Arena Leaderboard.
As shown in the top plot in Figure 3, all P2L routers, regardless of parameter count, outperformed Gemini-exp-122
The best model, P2L-1, 5B, reached #1 on Chatbot Arena during our testing period with score of 1389. This shows the utility of P2L: differences in model performance on prompt-by-pret by-prompt basis allow P2L to outperform all individual LLMs.
We deployed the grounded Rao-Kupper versions of P21-0. 5B, P2L-1. 5B, PB,-3B, and P2L-7B onto
Next, we discuss scaling performance with respect to the Arena score of the router. We see a general trend in Fig-
6

---

Prompt-te-L.aderrbaard Prompt-Adaptive LLM Evaluations
3 that bigger models do better overall. The exception is
Single moddlls
P2L-1.55 55, whose performance unexplainably strong; otherwise, the trend holds. We also tested other metrics, such as worst-case performance (bottom of Figure 3). The worst-case performance of P21. scales with parameter count as expected, and is uniformly much better than that of the marginal leaderboard.
1024
1000
07207
100
1330
1003
We also observe that the gap between the P2L routers and static models is large. The P2L routers are able avoid per-prompt model weaknesses and route elsewhere. In fact, the gap between the best P2L router and the best non-routed static model in the overall comparison was 25 points, while this gap grew to 51 points in the minimum category perfor- mance case. Appendix Figure shows P2I-7B's routing distribution conditioned on each Chatbot Arena category. Notably, we see relatively diverse routing patterns, even within a single category. We also observe intuitive behavior patterns, such that heavily routing to o1 01-2024-12-11 for math prompts and prompts.
Expacted query cost (8)
Arme ereativo
1200
2313
0100
1007
U.K.
1013
107-
F10
Expected query cost ($)
Figure 4: Arena score versus cost. Both plots show routing performance as a function of average cost. The left plot shows the averaged performance across all categories, and the right plot shows the performance in the creative writing category. The black open circles give the raw performance and cost of the models used by the router. Each gold dot represents the Arena score of the P P2L-7B router as func- tion of the cost constraint in (6). The plots show that the P2L router dominates and substantially improves the cost- performance Pareto frontier. All confidence intervals are 95%.
ex0 for creative
3.3.1. COST-OPTIMAL ROUTING
We show results of the optimal routing procedure detailed in Theorem with P2L-7B model on Chatbot Arena. Here, we use P2L to route between ol-mini, -laude-3-5-ssome--2224400 spt-40-2025-05-13, gemini-1. 100i--155-pr--000 mistral-1arge-2400 and and with budgets of (0.00218, 0044, 0. 00675, 0.00945,
2laude--3-S-haaiku-2221111
yemini-1.5-flatth.
sumably more advanced) model. In such a business, testing for regression of the model worse performance may be important. For example, they might ask the question: if I switch from GPT-4e 4o to GPT-40 ----miinnnniiii minnn, can I do so safely, and will my performance get worse on my customers?
0.0123, 00). To get reasonable cost estimates, calculate the expected cost per query with q; O; B[T,] for all models i E [M]. where O, is the output cost per token of model i, and T; is a random variable representing the number of tokens in response from model We estimate E[T;] as the response token length mean overall responses from model in Chatbot Arena. Additionally, estimate 9 in Theorem according to the Chatbot Arena model sampling distribution. We find the P2L router performs well, with Pareto frontier Arena score versus cost. Furthermore, on the right plot in Figure 4 we find the P2I. router continues to show dominant performance in Chatbot Arena's 's creative category despite large shifts in individual model performances.
This is a challenging question to answer because it requires knowledge of the enterprise's customer distribution which may require lengthy instrumentation and data collection procedures. However, P2L provides a partial solution to this problem. Given large unlabeled dataset of prompts (e.g., customer use-cases), we seek to: (I) Categorize these prompts automatically using an LLM. (2) Produce a pref- erence leaderboard within each category, and (3) On a per- model basis, analyze for which categories it is weak and strong (relative to itself or its competition).
3.4. Testing for regression and weraness/weakness analysis
For this, we can use a standard hierarchical clustering ap- proach. Assume access to a multilevel hierarchical catego- rization of prompts (this can be obtained from LLM). That is, we have a function categorize that takes in a prompt z and an integer level and outputs a category in (1....,ki) kr), for some integer ky. Given a set of prompts, grategory, we can compute a per-category leaderboard using (unif(2))
An important question when developing models is under- stand their category-leve performance. along with strengths and weaknesses. Imagine, for example, a business seeking to upgrade their workflow to cheaper or newer (and pre-

---

Prompt-te-L.addrbearr. Prompt-Adaptive LLM Evaluations
Chiness
Creative Weiting
0.10
ort-4omnini 69.06
Janncionn distance
CharaPo-to 60.69
0.08
olomini 46.55
53.26
0.00
0.00
suspnsefull Story
and Scenarios
0.02
100
16
16'
53.21
Math
62.45
Chetter-4e
60.26
Fuzetien distacee
0.125
ol-xini
ol-mini
0.100-
0,075
Math
0,050-
urp-deraini
49.83
ChardPn-to 51. 95
0.020
ol-nini
80.08
10P
Ii
16P
10'
57.95
Aggregated prempts
Optimal
PaL (32(C)
PLL (1)MM)
FIL (20083)
(780k)
Geowstry Froblere withh Circles
Arithvetio Oparations and Calcolations
Figure 6: Aggregation scaling, The L1 distance between the aggregated leaderboard and the marginal BT regression as a function of the number randomly sampled and aggre- gated datapoints in two categories: Chinese (left) and Math (right). The L1 distance plateaus the optimal performance, which is around 0.025. A nonzero optimal distance is ex- pected as the empirical BT coefficients are derived from finite validation sample, and so these coefficients have their own irreducible statistical error. Thus, the P2L. estimate converges to near-optimal solution with increased data.
G77-fo-aint 50,
6P7-to-sint 47.51
Chatari-40 30.35
Chet527-4e
ol-aiai
ol-mini 68.81
60.54 54
60.1 60:74
Figure 5: Regression test. We show the strengths of dif- ferent OpenAl models on various topic clusters based on their win rate against -PT-40-202--000000---- 00-13 as predicted by P2L-7B. For each category, we show the probability a given model wins against GP7-40-2024-05-11 under the BT model. The results show strong category-specific variability in performance; for example, o1 mimmini is sub- in "Arith-
stantially better than 202-4--2024-1 metic Operations and Calculations" but substantially worse when asked to write a "Suspenseful Horror Story".
3.5. Aggregation scaling
Given a distribution of prompts, we aim to evaluate how P2L behaves using the aggregation technique described in 2.1.1. Specifically, we analyze how P2L's aggregated leader- boards compare to ground truth category leaderboards as well as how this relationship scales with data. First, calculate ground truth leaderboards over a large category from the validation set with marginal regression. We then aggregate P2I. over increasing subsets of this category's prompts. Lastly, we plot the LI function distance between the aggregated leaderboard's predicted probabilities and the ground truth leaderboard's predicted probabilities subset size increases. Since both the train and validation set drawn from the same distribution, we denote the optimal value to be the L1 function distance between the ground truth category leaderboard and the category leaderboard derived from marginal regression on the train set.
as in (3). Note that the finest-grained categories may have very little data, motivating the need for P2L.
Figure 5 shows an example analysis of five different OpenAl models. The clustering method used is detailed Appendix Section E. Here, the percentages are calculated the win rate against -PT--4-2024--05555-
under the BT model. According P2L- 7B, OpenAl models' performance varies across different categories and topic clusters. While ol might be a better model on average, it is essentially the same compared to GPT-4o-mini on certain creativity tasks, notably the former is 100x more expensive than the latter. In
tasks, the gap widens significantly. These intuitive results demonstrate the reliability and effectiveness of P2L. See Figures& 8 and 9 for similar and more detailed plots on Llama 3 fine-tunes. We also include a variant of our regression analysis under the grounded RK model from (8); this provides guidance as to the absolute reliability of the model, not just preference over alternative models; see Figure 10.
In contrast to marginal regression, which requires thousands of prompts for stable leaderboard. P2L converges near this optimal value within 100-250 prompts (Figure 6). Here, we see P2L's potential to create accurate aggregated leader- boards efficiently, while also reinforcing the validity of its per prompt outputs. Furthermore, as we scale the amount of training data seen, P2L's predictions over singular prompts differ more drastically from category leaderboards while
8

---

Prompt-to-L.caderbbarrr: Prompt-Adaptive LLM Evaluations
still converging with more prompts (Figure 6). A clear seal- ing law ensues, as increased data allows P2L to make more distinguished individual leaderboards while still maintain- ing its aggregation ability at the category level.
is often trained a proxy human preference. Similar to P2I., reward model training may use contrastive pairwise or K-wise loss, for example using the BT model (Chris- tiano et al., 2023; Bai et al., 2022; Ouyang et al., 2022; Zhu et al., 2023). However, reward models are agnostic to model identity, requiring a prompt and response to return a single score for the response. P2L. which is aware of model identities, instead seeks to output expected model response quality, conditioned input prompt, instantly generating full leaderboard over all models without requiring model responses to be generated. This yields efficient leaderboard creation over arbitrary prompt sets.
3.6. Performance on out-of-distribution prompts
To assess how P2L generalizes to unseen prompts, we eval- uate it on LiveBench (White et al., 2024), a verifiable, contamination-frel benchmark with 1,000 questions cover- ing diverse categories (c.g.. math, coding, reasoning). Un- like Chatbot Arena, it utilizes objective success metrics. We restrict our evaluation to smaller pool of models. Among these models, P2I. selects its candidate models for each ques- tion based on the predicted prompt-specific performance and then uses the output of the chosen model as the final answer. Table 1 shows that P2L-7B surpasses the static oracle baseline among the model subset, achieving an over- all LiveBench score of 59.275. Even far smaller versions (e.g., 1.5B) match or exceed top static models. This means P2L. having never seen ground truth labels or model re- sponses, performs as well better than rumning all models on LiveBench, scoring them using the benchmark's ground truth labels, and selecting the best model after the fact. This demonstrates that preference-triie trainned routing generalizes well to an out-of-distribution, ground-truth benchmark.
Meta-learning, P2L is related to meta learning (Schmidhu- her, 1987; Santoro et al., 2016; Finn et al., 2017) insofar as we are training model to output models. For example, we have discussed training an LLM (the meta-learner) output coefficients of a BT regression (the learner). However, the meta-learning literature primarily focuses on learners that are deep neural networks. Instead, we let the learner be an extremely simple statistical model that is used for inference.
Routing. Prior work on routing LLM queries optimizes trade-offss between cost and performance, typically through classifiers or gating mechanisms. RouteLLM (Ong et al., 2024) and AutoMix (Madaan et al., 2023), Hybrid LLM (Ding et al., 2024), train classifiers to decide between a strong and weak model, while LLMM-Blender (Jiang et al., 2023) ranks candidate responses and blends them. Rou- terDC (Chen et al., 2024) uses contrastive losses to train a query- router. Unlike these approaches, which operate over small fixed set of models, P2I. learns a parametric function mapping prompts to full model leaderboards. Its statistical structure supports efficient cost-aware routing, outperforming static models in live crowdsourced settings while scaling to personalized and task-specific selections.
Moreover, we consider the cost-constrained routing case. To examine this trade-off, we apply LeadbrrLedd to LiveBench at various cost thresholds (c.g.. $2, $5, $10, $15 per million tokens) using the cost-optimal routing method discussed in Section 3. Figure 11 (in the appendix) shows that, in all budgets tested, the P2L cost-aware router consistently scores higher or comparable LiveBench scores to the perf-perfo model within that specific cost thresh- old. These gains are most pronounced when the budget permits occasional routing to a more expensive (and often stronger) model for prompts that particularly benefit from it.
Benchmark Compression. P2L can be related to bench- mark compression (Polo et al., 2024). Rather than reducing benchmark size via pruning data examples, P2L captures and compresses benchmark information parametricully.
4. Discussion and Related Work
Parametric statistical models. Our work builds on clas- sic log-linear models and GLMs, like those of Bradley and Terry (1952); Rao and Kupper (1967); see (McCullagh, 2019) for a review, and (Ameli et al., 2024) for further ex- tensions that enrich this model class for better LLM ranking. The closest piece of work to ours is Hastie and Tibshirani (1993), which proposes varying- can be seen as a subclass of cooffig-oeeffccient models. P2L. our knowledge, ours is the first work to parameterize such a model via a foundation model and backpropagate it end-to- end, while the techniques in Hastie and Tibshirani (1993) use bespoke fitting procedures and simpler statistical models than LLMs.
This work develops fundamental tools for granular and query-specific evaluations in all evaluation tasks. Although our experiments are largely based on Chatbot Arena, this is not the only evaluation that could benefit from P2L. As discussed in Section 2, any feedback signal can be accom- modated. Thus, our techniques would equally work well for other evaluations (Hendrycks et al., 2020; Zellers et al., 2019; Cobbe et al., 2021: Srivastava al.. 2023; Zhong et al., 2023; Chen et al., 2021; Lin et al., 2023; Liang al., 2022) as well as cost and lateney prediction.
models. To
Modeling human preference. During Reinforcement Learning from Human Feedback (RLHF), a reward model
9

---

Prompt-te-L.caderbbarrr: Prompt-Adaptive LLM Evaluations
Impact Statement
Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, and Yu Zhang. Routerde: Query-based router by dual contrastive learning for assembling large language mod- els, 2024. URL https: /aaxivivvvv..////a/ org/ abs 2409. 1 19886.
This paper presents work whose goal is to advance the field of Large Language Model evaluation. There are many po- tential societal consequences of our work, none which we feel must be specifically highlighted here.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anasta- sios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gon- zalez, et al. Chatbot Arena: An open platform for evaluating LLMs by human preference. arXiv preprint 243:01240..4132,, 2024.
References
AI@ Meta. Llama 3 model card. 2024. URL
https://githann wtw.stttub..oommmooeemmemee comm/meta-lllamm /11ama3/ blob/main/MOnl BWWDMMOOEEDDD....LLM
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Lewis Tunstall, Agustín Pi- queres, Andres Marafioti, Cyril Zakka, Leandro von Werra, and Thomas Wolf. SmolLM2 with great data, comes great performance, 2024.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap- pert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training ver- ifiers solve math word problems. arXiv preprint 1.Xi::2110.14168. 2021.
Siavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W Mahoney. A statistical framework for ranking Ilm-based chatbots. arXiv preprint arXiv:2412.18407, 2024.
Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, 2024. (Accessed on 06/05/2024).
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Sub- habrata Mukherjee, Vietor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost- efficient and quality-aware query routing. arXiv preprint arXiv:2404. 14618, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint ar arXiv:2204.080 058622. 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model- agnostic meta-learning for fast adaptation of deep net- works. In International conference on machine learning, pages 1 1126-1135. 115.. PMLR, 2017.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, (9(3/4):324-445 1952.
Richard P. Brent. An algorithm with guaranteed conver- gence for finding zero of a function. In Algorithms for Minimization without Derivatives, chapter 4. Prentice- Hall, Englewood Cliffs, NJ, 1973. ISBN 0- 0-13-022335-2.
Evan Frick, Peter Jin, Tianle Li, Karthik Ganesan, Jian Zhang, Jiantao Jiao, and Banghua Zhu. Athene-70b: Redefining the boundaries of post-training for open mod- els. https: / pagcennnncc......o co/ co/Nexus flow/ Athene- 703, 2024. Accessed: 2025-02-12.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Ponde Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew Carr, Jan Leike, Josh Achiam, Vedant Mishra, Evan Morikawa, Catherine Olsson, Jakub Pachocki, Jack Hewitt, Bowen DasSarma, Sam McCandlish, Dario Amodei, and Tom Brown. Evaluating large language models trained on code. arXiv preprint arXiv: 107.03374 2021.
Trevor Hastie and Robert Tibshirani. Varying-coefficient coefficient models. Journal of the Royal Statistical Society: Series B, 55(4), 1993.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Michael Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033:1 2020.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ki-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. OpenAl ol system card. arXiv preprint arXiv:2412.16720, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna,
10

---

Prompt-to-Lcaderbbard. Prompt-Adaptive LLM Evaluations
Florian Bressand, Gianna Lengyel, Guillaume Bour, Guil- laume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subrama- nian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavrit, Thomas Wang. Timo- thée Lacroix, and William El Sayed. Mixtral of experts, 2024.
feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Robin I. Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193-222, 1975.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybench- marks: evaluating Ilms with fewer examples, 2024. URL https: http::///axvvvvvo wwtp://////...././/////// 14992.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm- blender: Ensembling large language models with pair- wise ranking and generative fusion. arXiv preprint arXiv:2306. 025611, 2023.
PV Rao and Lawrence L Kupper.
Ties in paired-companee comparison experiments: A generalization of the bradley-terry model. Journal of the American Statistical Association, 62(317): 194-204, 1967.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tian- hao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data high-quality bench- marks: Arena-hard and benchbuilder pipeline, 2024. URL https: /tta:////xxivi org/ abs 2406.10 11006.1199 11939.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate- level Google-proof Q&A benchmark. arXiv preprint MIXi::2311.12022 2023.
Percy Liang al. Holistic evaluation of language models. arXiv preprint arXiv:2211.091110 2022.
Zhixing Lin al. Toxicchat: Analyzing the patterns of toxic behaviors in open-source LLM chat logs. arXiv preprint arXiv:2308.07 01966, 2023.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3s technical report. arXiv preprint arXiv:2412. 19437, 2024.
Jürgen Schmidhuber. referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Evolutionary principles in self-
R Duncan Luce. Individual choice behavior, volume 4. Wiley New York, 1959.
Aman Madaan, Pranjal Aggarwal, Ankit Anand, Sriv- idya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, et al. Automix: Automatically mixing lan- guage models. arXiv preprint arXiv:2310. 12963, 2023.
Aarohi Srivastava et al. Beyond the imitation game: Quanti- fying and extrapolating the capabilities of language mod- els. arXiv preprint arXiv:2206.0460 04661, 2023.
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024a.
Peter McCullagh. Generalized linear models. Routledge, 2019.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs with preference data. arXiv preprint arXiv:2406.18666 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arX.v:2403.08295, 2024b.
OpenAl. New models and developer products announced at DevDay, 2023. (Accessed on 06/05/2024).
OpenAL Hello GPT-4o.
https: / /opp/ai..c.oo
ftaex/hheloo 4otpp////11.. 06/05/2024).
40/,
2024. (Accessed on
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: /ttp:://wweemmmmmmm github. i io/blog/ .welo.//wen2.5.. 5/.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro:

---

Prompt-te-L.addrbbarr.. Prompt-Adaptive LLM Evaluations
more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574, 2024.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, al. Livebench: A challenging, contamination-free IIm benchmark. arXiv preprint arXiv:2406. 1931144466.114.... 2024.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang. Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yux- uan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open Foundation Models by 01.AI, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07833 2019.
Wanjun Zhong et al. Agieval: A human-centric bench- mark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
Banghua Zhu, Jiantao Jiao, and Michael Jordan. Prin- cipled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.112700 2023.
12

---

Prompt-te-Lcadderbbarii Prompt-Adaptive LLM Evaluations
A. Proofs
Proof of Theorem I. The equivalence of (4) and (6) immediate. Proving the equivalence of (5) and (6) more challenging. and we focus there.
We begin by simplifying the expressions in (5). The cost constraint be succinetly written as T S C. Regarding the objective, because the binary cross-entropy loss is linear in the response.
E BB Y (00 (=)))=))))))))))))))))))))=)) [((a (2)a), (0)) A 0 (3)))-0( (=)aa)) 2 = -EAng e (00(0-0(()))) (z)a), ( "=" a) 2 =
where again W* represents the population win matrix, with entries W = o - 0² (z)a). Thus, the optimization problem in (5) can be equivalently rewritten as
maximize #EA Ar
(T)
subject to #ecS.
where
(( = argmin Eang (((0(0-0 (# Ww))
OEB
Examining the first-order conditions of the inner optimization problel for (#) shows that the solution satisfies
-
W
(9)
4
Define
R( = T W'g. q, G(0) Σ qA a 0 (z)a).
Then Since G-1 is strictly increasing,
maximize 0'
maximize R(T)).
Thus, the problem reduces to:
maximize क W' 4 REAM, # FTeSC
which is exactly the problem in (6).
B. Additional theory
B.1. Aggregating leaderboards via averaging
The BT model tells us that for all = E Z,
xx, - -
=, x P(Y =
log
1 1- -
=
Thus,
T- X 2) 2
P(Y - X
) =x= J (=))d(((=)
Ez~Q
log
i(Q)
That is, taking (weighted) average of the values of 0" (z) leads to predictor of the expected log-odds.
This method has two downsides: firstly, increasing the mth coordinate of é(Q) does not mean that model m is more likely to win against other models on average. Secondly, the function ((Q) does not have a simple relationship with the win rate. This motivates the need for the aggregation metric from Section 1.
13

---

Prompt-te-Lcaderbbard.. Prompt-Adaptive LLM Evaluations
C. Additional Routing Figures
31-2024 12-17
14.199%
2.206
2,841
10 90015
80%
6.549
21 777
22.1 1.5%
25 F5
1015
10 16.
1066
1: 1000
chstgpe-co-laee 20241120
1:205
4.006%
2.31%
6.325%
15.0056
1.74
6.6755
6.32%
deepoook xd
0,11%
0.045
0.00K
2.006
3.505
4.1500
cleedde-- www.eeeeeeet--oeeee 10000
9.500
Math
Hard
Ciueo
leat. Folior
Croalive
Nont
000075
Chatbot Arena calegoey
Figure 7: Router model choice distribution in each prompt category. The rows are different models, and the columns are different categories. Each cell represents the probability that the model was selected within that category (i.e., columns sum 1). Models with an average selection rate below 1 1% are not shown.
D. Additional regression tests
Creative Mriting
Math
Llama3-70b Athene
Llana3-700 20b-Athene
11am3.11-73b
Llama3. 1-70b
65.66
1lamm3.1-700--eeee Wenotooon
65.5t
11amm3.2-73b-ruul.
57.94
1lamm3.1-70t-uult
56.5t
11amm3.3-73b
L1ama3.3-70h
1lama3.1-435b-
59.34
L.ama3. 1-405bbb fp8
59.76
1.1 3.1m63.1-43.00
62.04
L.aana3 1-4056-bb16
61.89
Suspenseful Borroz Story
Roleplay Dynamica and Scenarioa
Geone Problena with Circles
Axi thme tic Operations and Calculationa
M1 3-003-300---00 Athene
54.1%
Athene 1.1 11am33-70b-- -oheee
53.8%
L. L2ama3-700 70b Athene
Atheee
58.6%
Llama33700 Athene
Atheea
51.59
i1ama3.1-700
49.9%
11aam.31111
53.0%
Llama3.1-700
56.2%
Llama3.
56.3%
1lama2.1-70b-Weae toon
1-700----ee Nemotron
--70h-Memottro
1lama3.1-70b-1 Hemotron
1aama3.1-70b-uul..
49.4%
11ama3. 1aamm3.1-70bTuu... -bbuuu
66.5%
57.4%
tlama3.1-702-1
53.64
1lama3.3-128
52.34
11an=3. 3-70b 3-70b
52.6%
Llama3.3-900
58.14
11ama3.3-703
51.0%
11mm3.11-05bb #p8
51.9%
3--03..1405b---5 Ep8
52.24
1-405.1-405b---0 fp8
59.14
T.aama3 1-C05b fp8
53.76
1.
54.4%
11 1-10-------1
53.4%
17ama5.1-44
1.1em33. 140.1-1-0bbbbbb
Figure 8: Regression test on Llama models with creative writing and math prompts. The percentages shown signify win against Llama -3 3-700 under the BT coefficients predicted from P2L-7B.
14

---

Prompt-oe-Lcaderbeari. Prompt-Adaptive LLM Evaluations
Instruction Following
Coding
Llame A-hen-A
60.94
1. 1- 30ma3-70b-BBl Ath-et
63.36
57.24
1.aaa3. 1-706
59.06
-.aama.2-77b---- trmo
1-70002-11----0001 Memotron
68.36
11ama3.2-73b-- 1-73b-Taau
58.05
Llama3. 11-00------- Pulu
58.56
11ama3.3-722
59.05
Liama3. 3-700
59.66
1-40 5b-
Llame 3-1040bb-bbbb
64.66
L. 10mma3.1-405 -b-b66
63.96
Instruction Repetition and Logging
Text Reversel Spelling
Pythan Progeaming
DataPrane Operations with Pandaa
Concapts and Examplea
56.54
315ma3-77b-Athene
49.64
30ana3-70b-Athee
L1ena3-73b-Ate
53.84
1. 3.
49.84
L2 3-700
50.14
11ama3.1-70
59.24
61.15
1lama3. 1ew...1-0-beaeee Nemo
57.25
-
LLama3. 1-70b-Nemotroo
67.35
11ama3. 1attaa3.-722-eenaee
67.11
ilama3.1-79b-Tun
52.61
11ama3 -uluu
51.51
Liama3. 1-lu
55.91
Llama3. 1-T
57.9%
11ama3.3-70B
55.54
11ama3.3-700
Liama3. 3-705
1lama3.
Llama3. 1.ama3.1-405b-B5 8pp
56.44
L1ama3, 1.mma3.1-405b-ppe
$5.04
1.ama3.1-405b--
60.14
62.24
Alama3.1- 1-105bbbbb
57.04
31ama3.1-4056--33
60.84
L-ama3.1-003b-F1e
62.24
1lama3.1-405b-
64.0t
Figure 9: Regression test on Llama models with instruction following and coding prompts. The percentages shown signify win rates against alama-3 1------111 3-7 70 under the BT coefficients predicted from P P2L-7B.
Creative Writimg
Math
GPT-4o-mini
77.98
GPT-4o-mini
60.78
GPT-4o 78.28
GPT-4o 62.98
ChatGPT-4o
83.98
ChatGPT-4o 62.88
ol ol-mini 73.08
ol ol-mini
73.8%
ol
80.18
ol
76.98
Suspenseful Horror Story
Roleplay Dynamics and Scenarios
Geometry Problems with Circles
Ari thmetic Operations and Calculations
GPT-4o-mini
81.0%
GPT-4o-mini 75.28
GPT-4o-mini 42.38
GPT-4o-mini
23.54
GPT-40
80.5%
GPT-4o 71.18
GPT-40 44.28
GPT-40
24.94
ChatGPT-4o
8%.0%
Cha-GPT-4o
75.6%
ChatGPT-4c
42.28
ChatGPT-4o
25.6%
ol-mini 77.14
ol-mini 69.66
cl-mini 71.84
cl-mini
90.49
81.78
o1
70.58
c1
75.18
o1
81.48
Figure 10: Regression test using grounded Rao-Kupper. We show the strengths of different OpenAl models various topic clusters based on P2L-7B with grounded RK regression head (see Section 2.2) and dataset of unlabeled prompts. The percentage represents the sigmoid of the model coefficient. Because the RK model is grounded, this corresponds roughly to a signal of the model's reliability, i.c., its tendency to produce an answer that exceeds the voter's minimum bar of quality. The results show strong category-specific variability performance; for example, GPT-4o-ming and o1 have roughly the same reliability in the category "Suspenseful Horror Story", but not "Arithmetic Operations and Calculations". We can also see that some categories are more difficult in general L.LMs to answer reliably, and thus we see larger performance improvements from test-time compute models like ol and -mini.
15

---

Prompt-te-Lcaderbeart.. Prompt-Adaptive LLM Evaluations
Cosl $2.0/M
p2l/Qpeen2.5 70
541
241
atbeme--2 +2
Laeelemh
51.2
510
Bamn-3, 1-700-inelzut
Bamm- 3. 1-8b-ststruee
mixdadl Sa7h instodd--
GPP--4-10 4-Trriko
Best fized morde)
pat 1XRM
pal-t 3D
Math
Cating
Roasning
lusticciin Falkssing
Language
Data Analala
you
$5.0/M
pal/Qweu22577
312
35.0
34.
Homos 1-7
LicelBomhh
R12
32.
mistral laree-2107
sibemnee2ddoot
Bama-S. 1-770-iinttttt
9
mixtrol Sec22h instrnct v0.1
GPT Tasho
Best Rood madel
p21-1XMM
pal-0 pal All--001
Bht
par-10
Math
Codlug
Roaaning
lastrne Institttonn Falket ung
Langtaage
Data Anabyia
Cast < $100/MM
332
58.0
p2t p21/Qqeen22 7B
65.4
cloude-3 Soande---S-oomae -020m--02220022
geminz
Lomileechh
qpentl. 7-7.----on----
mistral lange 2107
athene-v2
--r-44-hutho
Best fise seedee
p26 I31M
patro 13
Math
Codiug
Rownning
Iudriccion tiat Folbering
Language
Data Anabyia
Cust $15.0/M
dhudls Somemer 0004000000000000
3833
p21y claude (85 2014422222 and 4ox 2024 0S qpeend 8-721-ino miskral 2107 22-rnt-22-dist
55.7
M6.
54.8
Loveloemhh
GPT 117-4-100 T-Too
Best Eisel monkel
pa akoys
122-0
pol-1 .16
pot- uu
pat
Math
Coding
Howwning
Indtraciinn Falbarin
Langung
Data Anabala
Cast Constraint
ahoudo 3.5 300mmn20020000 20243300
97.0
58.4
57.8
59.3
p21//qumn2.5-0 5-78
X1.9
Slaudo
202
4pt-L-tuchu 1it----tuhh2222-00000
2012-01-09
LiveBoehh
61.2
And- 2ol--lo22002 (405-1
opeen2 5-72h-inetrrut
allameee 2-dist
GPT ToTho
Best lized mandel
AG
PDF-30
101-1B
Math
Codiig
Roamnning
Indmiciinn Fallssins
Langtagg
Data Analyaa
Figure 11: LiveBench cost routing. Comparison of the P2L cost-aware router and static models on LiveBench under various inference-coss constraints. The left plots show each model's overall LiveBench performance at different maximum cost thresholds, while the right plots display models' relative rankings across multiple categories at the specific cost limit. By adaptively allocating prompts to cheaper or more expensive models when advantageous, the P2L. router consistently matches or surpasses the best single model within each budget.
16

---

Prompt-te-Lcadeebbaatt Prompt-Adaptive LLM Evaluations
Model
LiveBench Score
Math
Coding
Reasoning
Language
Instruction Following
Data Analysis
P21-7B
59.3
$1.9
63.2
50.0
56.5
75.8
56.3
59.2
51.3
63.5
54.7
56.8
72.3
56.7
claude=335--cmme-22200
202441022
59.0
$1.3
66.8
50.0
57.0
74.1
54.9
P2I-1.5B
58.4
55.3
67.5
48.0
51.4
71.9
56.7
P21-3B
57.8
49.6
66.8
50.7
53.3
70.4
56.2
P21-0.55
57.0
51.9
59.6
50,7
51.7
73,4
54.8
P21-135M
56.2
48.9
63.5
50.0
47.1
74.1
54.0
P2I-360M
54.9
52.4
58.1
44.0
44.1
74.4
56.7
athene-22-chat
53.4
53.4
48.0
37.5
71.6
50.2
gpt-4o-2024-05-13
52.8
42.7
50.4
47.3
49.3
72.4
54.4
gwen2.5-72biinstruet
52.6
52.3
55.6
47.3
36.0
73.3
51.1
9ot-3-turbb--2224040099
51.2
40.3
45.8
52.7
45.3
68.4
54.5
mistral-large-2407
50.4
48.4
45.8
44.0
40.5
73.1
50.4
chatget-4c-lltttes-222122
49.4
37.7
44.4
44.7
43.7
74.1
51.7
gemini-1.5-proo001
44.2
36.2
33.7
34.0
37.6
68.9
54.8
1lama-3.1-77b-insttre
42.4
34.4
32.9
34.7
36.4
68.9
47.3
-lama-t-770binnstrucc
41.7
26.3
28.7
40.0
36.3
68.5
50.7
mixtral--a222-iiistuttttv...
37.5
36.0
27.9
35.5
26.3
19.5
14.5
18.7
17.8
53.9
33.3
mmxtzal-9x76--nstrreee
22.1
12.4
10.6
23.3
12.8
46.1
27.4
Table 1: LiveBench performance comparison. Comprehensive evaluation of language models across seven capability categories: overall LiveBench score, mathematics, coding, reasoning, language understanding, instruction following, and data analysis. Results show performance comparison between p21 models at different parameter scales (135M to 7B), Claude-3.5 Sonnet versions, and other leading language models including GPT-4, Gemini, and LLaMA variants. All models were evaluated using identical inference settings those employed Chatbot Arena ensure fair comparison. Scores are presented percentages, with the highest score each category shown bold and second-highest underlined. P2L-72 72 achieves top performance in LiveBench Score (59.3) and Instruction Following (75.8), while maintaining competitive performance across other categories.
E. Additional information
E.1. Clustering Method
We leverage topic modeling approach using BERTopic. We first encode each prompt using OpenAI's embedding model, text-embeddinng-
reduce dimensions with UMAP, and apply hierarchical-based clustering algorithm (HDBSCAN) with min size cluster 8. This process generates distinct topic clusters. Each topic is then summarized and named using an LLM (GPT-4o-mini). This process is replicated from ArenaHard's clustering pipeline (Li et al., 2024).
E.2. Accuracy
Model
Accuracy (%)
Random
25.00
Marginal
37.40
0. 0.135B
40.42
0.36B
42.23
0.5B
46.06
1.5B
47.06
3B
47.41
7B
47.88
Table Grounded Rao-Kupper model accuracy by parameter size. Accuracy calculated according correctly predicting the classes;
(win, loss, tie, tie (both bad)
17

---

Prompt-te-Lccdderbbarri: Prompt-Adaptive LLM Evaluations
E.3. Training Costs
P2L models fairly inexpensive to train: P2L-7B on 1.5 million data points costs less than S250 train end-to-end using relatively unoptimized Deepspeed and Huggingface Trainer infrastructure ($23.92 per hour for 8xH100 on Runpod). The well-performing 3B and 1.5B variants train with negligible cost.
E.4. Model list
The full list of models is: athene-v2-chat (Frick et al., 2024), chatggt-4b-latess-2241111 Mlaude-3-S-haixx-2222100
clauee-3---soont-2222430
claude-3-5---0nnet-22221222
(Anthropic, 2024), deepseek-v3 (Liu et al., 2024), 1ammiii-1.--llsh-aee
gemini-1..5-flash-000
1omini-1.5-pros 001, gemoo-21.5-pro-02 (Team et al., 2024a), gemini-2.0-flashh-e
gemini-2.0 - gemini-exp-1266, 0et
(Team al., 2024b), glm-4-plus, gpt-4--106-prevveew, nAI, 2023), cpt-44-2024-05-13, 2024), 3 1e 11ama=3.1-7700-nnsttrru
opt-44-burro--20244400-999 (Ope-
gpt-40-2024408--06 gpt-44-miii-22024-----88 (OpenAl,
1iama-3.1-8b-insee -.nt-tt---
(AI@Meta, 2024), mistral-large-2077, (Jiang et al., 00-2 (Team, 2024), and yi-lightning (Young
ol ol-mini, ol-oo--pppp ppreview (Jacch et al., 2024),
et al., 2024).
18


Reviews:
[{"review_id": "qrMukzp3Rh", "reviewer": "Reviewer_Lp4H", "summary": "This paper proposes a prompt-to-leaderboard (P2L) method to predict prompt-specific leaderboards via large language models (LLMs) trained on human preference data. The authors make LLMs to output the coefficients of parametric regressionsthat represent per-prompt leaderboards. Thus, this leaderboard supports optimal routing, personalized leaderboards, task-specific performance analysis, and automated evaluation of strengths and weaknesses. Empirical results show that P2L's router achieved the #1 spot in the Chatbot Arena leaderboard in January 2025.\n\n### update after rebuttal\n\nThe authors' rebuttal solves my concerns. I hightly suggest the authors to incorporate the content in the rebuttal into the revision.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We thank the reviewer for considering the novelty and soundness of our work. We agree with the reviewer on generalization to new LLMs; this is of interest, and we are excited to continue exploring methods, such as online learning, to optimize this in future work. Roughly, we have seen it takes roughly 6k votes to get stable rankings for a given model— similar to what is needed for the marginal case. Fast adaptation to new models is a promising future direction. We also note our P2L models are fairly inexpensive to train: P2L-7B on 1.5 million data points costs less than \\\\$250 to train end-to-end using a relatively unoptimized Deepspeed + Huggingface Trainer infrastructure (\\\\$23.92 per hour for 8xH100 on Runpod). The well-performing 3B and 1.5B variants train with negligible cost. We will include these exact training hardware, time, and cost numbers in the appendix for future revisions.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing theoretical / empirical evidence.", "methods_and_evaluation_criteria": "The proposed methods and corresponding evaluation criteria make sense for the research problem.", "theoretical_claims": "I have checked the correctness of most of the theoretical claims in this paper, especially Theorem 1.", "experimental_designs_or_analyses": "The experimental designs or analyses in Section 3 are mostly sound and valid.", "relation_to_broader_scientific_literature": "Compared with the broader scientific literature, this paper first (to the best of my knowledge) proposes a prompt-specific leaderboard generation method which can support optimal routing and task-specific performance analysis.", "essential_references_not_discussed": "Most of the essential related works are discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The proposed prompt-to-leaderboard method is interesting and novel, which provides meaningful insights into the nature of LLM leaderboards which are shaped by human preferences.\n2. Well-designed experiments show the effectiveness of the proposed method from different perspectives, which are inspiring.\n3. This paper is easy to follow.\n\nWeaknesses:\n\n1. I am curious about the generalization ability of P2L when new LLMs are incorporated into the leaderboard. This may be important because new LLMs are still emerging rapidly. How much preference data about new LLMs are enough for accurate leaderboard re-estimation when they added to the leaderboard? Can original P2L model parameters be utilized to accelerate the training process of P2L when new LLMs are incorporated? The authors are suggested to add more discussions about the generalization ability of P2L to new LLMs.", "other_comments_or_suggestions": "None.", "questions_for_authors": "I have included my questions in other parts of the review.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jpp1NI0Po3", "reviewer": "Reviewer_Hdbw", "summary": "This paper is motivated by the fact that LLMs are sensitive to prompts, and current benchmarks such as Chatbot Arena leverage pair-wise comparisons from users to rank models without explicitly controlling the prompt distribution. The paper proposes a method that essentially trains a model to predict the \"model advantages\" vector from a collection of tuples (prompt, two-hot vectors). The authors further consider ties in human preferences to gain better and more precise signals for interpreting model abilities with different prompts. The trained models vary in different sizes and have shown good performance on the leaderboard as well as a balance between performance and cost.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:\n\n**W1 Intro**: We will revise our introduction to explicitly define 'leaderboard of LLMs' as a prompt-specific ranking of multiple LLMs to improve clarity.\n\n**W2 LLMSelector**: We believe LLMSelector is an excellent work which shares some philosophies with P2L. Importantly, it is concurrent work, and we should not be responsible for establishing novelty over LLMSelector.\nAdditionally,  the goals of the two works are different. LLMSelector aims to select models in compound AI systems using an LLM judge. Instead, P2L’s methodology instead gives full rankings of chat models for a single input. \n\n**W3 Regarding MOE**: Good point, this does bear some resemblance to MOE. In fact, it would be interesting future work to see if our strategy can be incorporated in model training as a principled approach to MOE. However, at the moment the two are different. MOE typically trains model parameters end-to-end for inference optimization, whereas P2L's approach predicts leaderboard coefficients over fixed sets of external, possibly black-box APIs. The P2L framework provides granular evaluation of external models without retraining these models or modifying their parameters.\n\n**W4 Method Clarity**: We thank the reviewer for pointing out clarity issues regarding terminology. In our revision, we will simplify and clearly define key terminology (e.g., BT regression, leaderboard vectors, routing policy notation) to improve readability, especially in Section 2.\n\n**W5 Comparison w/ Previous work**: We will make sure to provide more detail on our training method in future revisions. Our architecture is similar to reward modeling, with a pretrained transformer initialized with a new output linear layer. However, we output a dimension per model in the leaderboard, and use a loss that supervises 2 dimensions of the output per datapoint– this, to our knowledge, is novel.\n\n**W6 Baselines**: We understand the reviewers' concern around solid method baselines. We previously tested retrieval-based methods nearly exactly as described by the reviewer (e.g., retrieving k nearest prompts, weighting similar prompts). Empirically, we found the log-loss did not improve over marginal BT regression, with average loss differences smaller than 0.01 across various similarity metrics. Primarily, k must be quite large to obtain stable BT regression, which nullifies any per-prompt estimation advantage. Moreover, existing embeddings may encode “similar” in “semantic” subspace, and not “model performance” subspace. For example, “Explain BT regression” vs “Implement BT regression” are semantically similar, but fundamentally different tasks (explaining vs coding). Ultimately, we chose the marginal BT as baseline, since this was the most effective and stable baseline.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.", "claims_and_evidence": "Evidence is well-supported. However, the motivation in the introduction is rather confusing at first. After \"In other words, P2L models take a prompt as input and output a leaderboard of LLMs for that specific prompt,\" the authors don't explain further about the meaning of \"leaderboard of LLMs\" but rather continue by stating \"The P2L model can be trained based on any feedback signal, for example, binary human preference data...\" I was able to understand the meaning of \"prompt to leaderboard\" when I got to the method section, but it's not clear before that.", "methods_and_evaluation_criteria": "The evaluation is done with a comparison between P2L and multiple prevailing models, especially the closed-source ones.", "theoretical_claims": "I've read through the theories in the main paper, I believe they are correct based on my knowledge.", "experimental_designs_or_analyses": "The experiments include quantitative results from P2L router performance and cost study.", "relation_to_broader_scientific_literature": "The key idea of modeling the probabilistic distribution based on model preference seems to align with the key idea in mixture-of-expert models, i.e., the training of the gating model. Particularly, one can have dense aggregation or sparse aggregation (max pool) given the gating model's output. The main difference seems to be in the problem setting where human preference, i.e., two-hot encoding, is given, which requires the fitting schema introduced in the core method.", "essential_references_not_discussed": "This paper comes after the ICML submission deadline but is highly relevant: https://arxiv.org/pdf/2502.14815\n\nWhile experimemtal comparison is hard and there's difference in terms of the settings for sure, I am wondering the advantage of P2L over https://arxiv.org/pdf/2502.14815 in the methodology level. \n\nAnother line of relevant work may be mixture-of-expert models and particularily similar training paradigms as the method introduced in the paper.", "other_strengths_and_weaknesses": "The method is clear, but the writing is a bit hard to follow with ~10 variables defined and mixed with the main text. Some terminologies could be made more official and easier for people to follow.\nThe experiments seem to be adequate and come along with a nice clustering study in Figure 6, and the cost-performance trade-off study in Figure 5.\n\nThe problem setting itself is novel to me. However, I am not so sure about novelty at the method level, and it would help if the author could provide some clear explanations and comparison with existing works, not limited to the field of model routing.\n\nMoreover, the main thing that is not clear for me is that some more intuitive baselines should be compared with to justify the model training. For example, given a dataset {(p_i, z_i) | i=1,...}, where p_i is the prompt and z_i is the two-hot vector from human preference. Given a new user prompt p, why not retrieve the top k (e.g., k=100) similar prompts, say {p1, ..., p100} as well as their zs {z1, ..., z100} from the dataset (let the similarity between p and p_i be s_i) and use the weighted score \\sum z_i * s_i as the predicted final performance. How does it compare with P2L?\n\nThese being said, the problem itself is interesting and the experiments seem to be solid. I will consider adjusting my scores if the concerns above are addressed in some way.", "other_comments_or_suggestions": "line 80: from Z->R^M => \\theta: Z->R^M", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2gcBVjtHH", "reviewer": "Reviewer_d5DE", "summary": "This paper proposes a method that routes a prompt to a specific LLM from a given LLM list. Given a dataset of various prompts, responses from different models and the pairwise preference result, the method train a mapping from prompt to feature that fit the reward gain by the model when fed with the prompt. Then when given a new prompt, the method can be used to predict the most suitable LLM for this prompt. Experiment shows that the proposed method outperforms the best single model on Chatbot Arena.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:\n\n**W1 Scalability**: We agree with the reviewer that reducing the cost of adding new models is of interest, and are excited to continue exploring methods, such as online learning, to optimize this in future work. However, we also note our P2L models are fairly inexpensive to train: P2L-7B on 1.5 million data points costs less than \\\\$250 to train end-to-end using a relatively unoptimized Deepspeed + Huggingface Trainer infrastructure (\\\\$23.92 per hour for 8xH100 on Runpod). The well-performing 3B and 1.5B variants train with negligible cost. We will include these exact training hardware, time, and cost numbers in the appendix for future revisions, which we believe will clarify cost concerns.\n\n**W2 Cost**: The cost of the P2L model is negligible, both in terms of compute and time. P2L is, at its largest, a 7B model inferencing a single forward pass on the prompt only, which is very fast: P2L 7B add around 5% overhead on first token latency on an A40, $0.40 per hour on Rundpod, and less than 1% of the average routed LLM’s cost. The performance gain, on the other hand, is substantial. Therefore we do not see this point as a concern in practice. We will update the paper to communicate this fact.\nW3 Validation Metrics: We will include clear full definitions detailing the loss calculation for additional clarity in future revisions. Moreover, we will include straight forward accuracy metrics in the next revision. For context, we have computed accuracies for the grounded RK models which classify {A, B, Tie, Tie Both Bad}:\n\n\n| Model     | Accuracy (%) |\n|-----------|--------------|\n| Random    | 25.00        |\n| Marginal  | 37.40        |\n| 0.135B    | 40.42        |\n| 0.36B     | 42.23        |\n| 0.5B      | 46.06        |\n| 1.5B      | 47.06        |\n| 3B        | 47.41        |\n| 7B        | 47.88        |\n\n**W4 Deployment Clarity**: We will update our revised paper to include greater detail on deployment to Chatbot Arena. Specifically, we currently detail the P2L router’s model list in Appendix D1. We will additionally specify that we collect blind pairwise comparisons against all active public models hosted on Chatbot Arena in a process identical to how standard models are added to the Chatbot Arena Leaderboard.\n\n**W5 Training Params**: When training P2L we do full parameter training. This means we update the weights of both the pretrained transformer and the newly initialized linear head. We will ensure this section is fully specified in section 3.1 in future revisions.\n\n**W6 Missing from Leaderboard**: It was deployed in battle mode on Chatbot Arena for some time, and with that data, we are able to calculate its leaderboard position even though it is not displayed. The model does not appear on the Chatbot Arena leaderboard because routers are not allowed on the leaderboard– however, results, including comparison vote data, will be publicly available after double blind is lifted. \n\n**W7 Clustering Method**: We leverage a topic modeling approach using BERTopic. We first encode each prompt using OpenAI’s embedding model, text-embedding-3-small reduce dimensions with UMAP, and apply a hierarchical-based clustering algorithm (HDBSCAN) with min size cluster 8. This process generates distinct topic clusters. Each topic is then summarized and named using an LLM (GPT-4o-mini).\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.", "claims_and_evidence": "The P2L model, when doing optimal routing, outperform single models. This claim is properly consolidated by the experiments in presented in the paper.", "methods_and_evaluation_criteria": "My concerns regarding the method are listed as follows\n\n1. It looks like that the method is not scalable enoough. For example, initially we have 30 models. According to Section 3.1, we are initializing the coefficient head to map to $\\mathbb{R}^{30}$. However, if now we need to add one more model to our model list, then we need the coefficient head to be a mapping with co-domain $\\mathbb{R}^{31}$, meaning that we need to re-train all the parameters from scratch. This issue is especially severe given P2L requires 1.5M training data.\n\n2. Given a prompt, the P2L framework requires first using the P2L model (backboned by an LLM) to compute the leaderboard. This introduces additional computational overhead and might be time consuming. It is doubtable whether this sacrifice is worthy the performance gain from selecting a (potentially) more suitable LLM given that the overall improvement brought by P2L is not significant.\n\nAlso, the paper employs two evaluation criteria for optimal routing. Some of my concerns are listed below\n\n1. First, the paper study the generalization of feedback prediction on a hold-out validation set from Chat-bot Arena and report the validation loss and squared error. This is not straightfoward enough because the calculation of the loss is not clearly stated in the paper. A more straightforward way is to report the validation accuracy, which is not reported in the paper\n\n2. Secondly, the paper considered Chatbot Arena as the benchmark for the optimal routing. While Chatbot Arena is a widely-recognized benchmark for alignment, it is unclear how the author deploy and test their model with Chatbot Arena", "theoretical_claims": "The proof of the theorems looks correct although I didn't check very detailedly.", "experimental_designs_or_analyses": "This paper seems lack some details in experiment setup, which make paper hard to follow\n\n1. What is the training parameters when training the P2L models?\n\n2. How is the evaluation on Chatbot Arena conducted? Also the author claims that they deployed P2L on Chatbot Arena but I didn't find a model called \"P2L\" on the [leaderboard](https://lmarena.ai/?leaderboard). Could the authors provide more detailed information?\n\n3. In section 3.4, how did the authors map a prompt to categories? Also, to conduct hierarchical clustering, how did the authors define the distance between different categories?", "relation_to_broader_scientific_literature": "This paper is related to the broad related works focusing on LLM routing, to which I am not very familiar.", "essential_references_not_discussed": "No significant missing of references", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bMu3TcGYs", "reviewer": "Reviewer_EY29", "summary": "This paper introduces Prompt-to-Leaderboard (P2L), a method for generating prompt-specific leaderboards of large language models (LLMs) by training LLMs on human preference data. The core idea is to model prompt-dependent Bradley-Terry coefficients, enabling per-prompt performance comparisons. Key applications include optimal query routing, personalized evaluations, and automated model strength/weakness analysis. The authors validate P2L on Chatbot Arena and LiveBench, demonstrating that P2L routers outperform static models in live evaluations and generalize well to out-of-distribution tasks. Scaling experiments suggest P2L follows power-law improvements with model size and data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:\n\n**W1 Arena Score Computation:** We deploy the P2L router onto Chatbot Arena, routing between the models detailed in Appendix D1. We collect blind human preference votes between responses from the P2L router against active public models on Chatbot Arena. We use these preference votes to calculate the Bradley-Terry regression over all Chatbot Arena battles to produce an Arena Score. This is the standard method to add a model to the Chatbot Arena leaderboard. We will ensure Arena Score calculations are well documented in future revisions. [This Google Colab Notebook](https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WByFNFiqxWQquwH) details exactly how Arena Scores are computed on the Chatbot Arena leaderboard. Our results, including comparison vote data, will be publicly available after double blind is lifted. \n\n**W2 Categorization:** We acknowledge that the automated clustering strategy has risk for noise. Note P2L can create an aggregate leaderboard over any subset of input prompts; this is detailed in section 2.1.1. Thus, any clustering strategy can be employed with P2L– our paper only details one such strategy as an example. We believe that co-developing clustering methods with P2L based rankings is an interesting future research direction.\n\n**W3 Theoretical Assumptions:** The reviewer is correct, and we try to be clear about this in the paper. The proof says that under the Bradley-Terry model, these routers are equivalent. We explicitly say, after the theorem statement, that “It is important to note that deviations from the BT model—for example, any non-transitivity—will break this relationship\" (line 162). \nThus, we believe this limitation has already been sufficiently communicated.\n\n**W4 Cost Estimation:** This is correct. We will mention this limitation explicitly in the paper.\n\n**W5 Paper Density:** We thank the reviewer for feedback. We will aim to increase accessibility in future revisions with intuitive visualization illustrating P2L’s functionality.\n\n**Q1:**\n> What explains the outlier performance of P2L-1.5B in Figure 3?\n\nWe attribute this to noise. The bootstrapped 95% confidence intervals overlap, indicating this variation is well-within sampling variability. We view the trend in the plots as more informative than the specific values.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.", "claims_and_evidence": "The assertion that P2L \"achieves #1 on Chatbot Arena\" (Section 3.3.1) lacks transparency: How was the Arena score computed?", "methods_and_evaluation_criteria": "Methods: The use of parametric regression coefficients (e.g., BT) is appropriate for modeling pairwise preferences. Extensions to ties and \"both bad\" scenarios are innovative.\n\nEvaluation: LiveBench and Chatbot Arena are standard benchmarks. However, category-specific leaderboards (Section 3.4) rely on automated clustering without human validation. It may result in noisy or subjective categories, and lead to ambiguous categorization criteria.", "theoretical_claims": "Theorem 1’s proof (Appendix A) assumes ideal BT conditions and ignores non-transitivity. Practical deviations (e.g., model ties) could invalidate equivalence.", "experimental_designs_or_analyses": "The power-law trend (Section 3.2) is convincing.\n\nHowever, simulated costs (Section 3.3.2) rely on token-length averages, ignoring variance in real deployments.", "relation_to_broader_scientific_literature": "- Builds on RLHF and BT models but innovates by integrating prompt conditioning.\n\n- Contrasts with routing methods (e.g., RouteLLM, AutoMix) by enabling large-scale, cost-aware routing.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "- The paper combines parametric statistics with LLMs for prompt-adaptive evaluation, which is novel and impactful.\n\n- The paper is well-structured but dense; more visualizations would improve accessibility.", "other_comments_or_suggestions": "NA", "questions_for_authors": "What explains the outlier performance of P2L-1.5B in Figure 3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HdHci2fAAB", "reviewer": "Reviewer_Xq7j", "summary": "Language model evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. The authors propose Prompt-to-Leaderboard (P2L), a method that predicts prompt-specific leaderboards by training large language models on ChatBot Arena preference data and outputting predicted scores for each model in ChatBot Arena, for each input prompt. The authors train routers based on this methodology and achieve the #1 spot in the Chatbot Arena leaderboard.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:\n\n\n**W1 OOD Results:** We understand that using the top static model might appear an intuitive baseline. However, practically, this static model is unknown ahead of time, and P2L’s value is precisely in selecting the best-performing model dynamically. \nThis means P2L, which never sees ground truth labels or model responses, performs as good as running all models on LiveBench, scoring them using the benchmark’s ground truth labels, and selecting the best model *after the fact*.\nWe will communicate this more clearly in future revisions.\n\n**W2 Baselines:** During development, we attempted non-deep learning approaches such as KNN and embedding based methods. We found these did not perform better than, and sometimes worse than, the marginal baseline– with a log-loss delta of less than 0.01. Ultimately, we chose the marginal BT as baseline, since this was the most effective and stable baseline– we will ensure to communicate this in the revision. In the end, the goal of this work is to create a scalable (along data and parameter count) method to provide granular model evaluations, which our deep learning approach provides.\n\n**W3 Clustering Novelty:** We are not claiming any novelty in the clustering algorithm. We are claiming that ours is the first approach that allows us to create a leaderboard for small, possibly singular, clusters, moreover doing so label-free. We will clarify in the final revision that our clustering mechanism is for demonstrative purposes and will relegate extensive discussions to supplementary materials.\n\n**W4 Fig 7:** We recognize our current presentation was insufficiently clear. We will enhance the caption of Fig. 7 to explicitly state what the function distance represents, how it relates to prompt-specific leaderboards, and how this concretely supports our claim. Specifically, the large function distance between P2L’s leaderboard and the marginal leaderboards for a smaller number of prompts (<= 10) suggest P2L's predictions differ from the marginal.\n\n**W5 Uninteresting Results:** We appreciate the reviewer’s feedback on Figures 6, 8, and 9, and agree that some of these results align with existing expectations—such as o1 performing well on mathematical tasks and ChatGPT-4o excelling in creative tasks. However, we argue that these confirmatory results are precisely what demonstrate the reliability and effectiveness of P2L. Moreover, P2L systematically captures these known performance trends without requiring expensive per-prompt annotations, thus validating our approach as robust and practically useful for model selection and routing. Additionally, our finding that GPT-4o-mini matches or surpasses substantially more expensive (up to 100x) models (e.g., o1 or GPT-4o) in certain prompt clusters highlights an important, practically valuable insight into cost-effective model utilization that is only clearly revealed by our method. We will revise the manuscript to more explicitly emphasize these practical insights, clarify how they support the validity of our approach, and ensure the results' practical implications are clearly communicated.\n\n**W6 Public Release:** We will release all P2L models, routing code, training code, evaluation code publicly. We will host a P2L endpoint to provide a way for readers to calculate raw P2L outputs for any prompt of interest.\n\n**W7 Related Work:** We will elaborate our connection to benchmark compression in related work. P2L uses deep learning to compress evaluation signals parametrically, uniquely allowing it to be label-free at test time.\n\n**W8 Figure Clarity:** We will comprehensively revise all figure captions in the revision, ensuring they succinctly summarize methods, key findings, and implications. Where necessary, we will include hyperlinks to relevant text.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.", "claims_and_evidence": "I think this paper's contributions are sound and would be of general interest, but I have some concerns about the clarity of the presentation and the rigor of the conclusions which I would like to see addressed. The authors overclaim in several places, which is unnecessary, as the work is sufficiently interesting without hyperbole.\n\nIn Sec. 3.6, I appreciate the authors' inclusion of a second, so-called \"out of distribution\" benchmark to evaluate their method; LiveBench is a reasonable choice here. However, the analysis of the authors' results considerably oversells the reality. The obvious naive baseline for their comparatively computationally intensive, complex and expensive method is to simply use the best performing model on the leaderboard for every prompt. In the case of LiveBench, this trivial baseline is either better than, or statistically no different from, P2L, when not controlling for cost. This is a very important limitation, and I would like to see the authors acknowledge it as such, and address why this may be the case. Their point about the cost-effectiveness of P2L in this setting is more compelling, and should be retained.", "methods_and_evaluation_criteria": "In Related Work, the authors claim that P2L is unlike prior routing approaches; is this intended to justify the fact that no baseline methods are included in several key experimental figures, such as 3 and 7, and the baselines in 2 are weak, by the authors' own acknowledgement? A few more cheap-to-evaluate baselines, like decision trees over embeddings of the prompts, or a bag-of-words approach, would increase my confidence in the authors' claims that their method is valuable.", "theoretical_claims": "I reviewed the theoretical claims in the main paper and I have no concerns.", "experimental_designs_or_analyses": "The categorization mechanism described in 3.4 is superfluous; LiveBench already includes these categorizations, ChatBot Arena allows reranking according to a wide range of types (https://lmarena.ai/), and many prior works anticipate the strategy, so it is not novel. This could be relegated to the appendix.\n\nIt's not clear to me how the claim \"P2L’s predictions over singular prompts differ more drastically from category leaderboards\" is supported by Fig. 7.\n\nThe authors make frequent use of a particular type of figure, which reports distinct model rankings for topic clusters. These figures take up a lot of space in the paper, and I find the results themselves puzzling. In Fig. 6, o1 is the best for all math-related tasks, and ChatGPT-4o is the best model for all other kinds of tasks. This distinction was discussed by OpenAI and many others, and it doesn't require an LLM to guess that this would be the outcome. In Fig. 8 and Fig. 9, Nemotron is the best on every subset, an even less interesting result. In short, I don't understand how these summary figures are useful; I would have much preferred to have had a link to raw model outputs for each benchmark, so that I could have evaluated the results myself.", "relation_to_broader_scientific_literature": "The idea of a reward model going from prompts directly to leaderboards is, as far as I know, novel.", "essential_references_not_discussed": "The authors should document in their related works the considerable research on benchmark compression which has emerged lately. It is at this point well understood that even for carefully curated benchmarks, only a small subset of the entire benchmark is necessary to establish stable model rankings (https://arxiv.org/abs/2402.14992). Aggregated compressed benchmarks such as (https://mixeval.github.io/) are relatively inexpensive to curate and run, compared to P2L, and correlate well with LMSYS Chatbot Arena.", "other_strengths_and_weaknesses": "I have no other strengths or weaknesses to note.", "other_comments_or_suggestions": "In general, the figure captions in the paper are inadequate. Figure captions should be expanded to give all the details necessary to understand the figure, and where those details are too extensive, should include hyperlinks to the relevant sections of the paper.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Prompt-to-Leaderboard (P2L), a method for generating prompt-specific leaderboards of large language models (LLMs) by training LLMs on human preference data. The core idea is to model prompt-dependent Bradley-Terry coefficients, enabling per-prompt performance comparisons. Key applications include optimal query routing, personalized evaluations, and automated model strength/weakness analysis. The authors validate P2L on Chatbot Arena and LiveBench, demonstrating that P2L routers outperform static models in live evaluations and generalize well to out-of-distribution tasks. Scaling experiments suggest P2L follows power-law improvements with model size and data.

The scores, 4, 2, 3, 3, tend towards accept, and some scores were raised after rebuttal, but based on promised improvements and materials release by the authors.  

Overall I am recommending weak accept.

Strengths: 

In general the reviewers assessed that the paper's contributions are sound and would be of general interest.  The paper combines parametric statistics with LLMs for prompt-adaptive evaluation, which is novel and impactful. The power-law trend (Section 3.2) is convincing.
The method is interesting and novel, which provides meaningful insights into the nature of LLM leaderboards which are shaped by human preferences.  The experiments are well-designed and show the effectiveness of the proposed method from different perspectives, which are inspiring.

Weaknesses:

Several author commented on a lack of clarity in the writing of the paper: 

“I have some concerns about the clarity of the presentation and the rigor of the conclusions which I would like to see addressed. The authors overclaim in several places, which is unnecessary, as the work is sufficiently interesting without hyperbole.” And “The method is clear, but the writing is a bit hard to follow”

Other weaknesses noted are: the method is not scalable enough, the generalization ability of P2L when new LLMs are incorporated into the leaderboard is questioned.  The method seems to incur additional computational overhear that may not justify the performance gains.

The reviewers also commented that although LiveBench and Chatbot Arena are standard benchmarks the derived category-specific leaderboards (Section 3.4) rely on automated clustering without human validation that may result in noisy or subjective categories, and lead to ambiguous categorization criteria.

The  simulated costs (Section 3.3.2) rely on token-length averages, ignoring variance in real deployments.
Decision: Accept (poster)