EIP: Weighted Ranking of LLMs by Quantifying Question Difficulty
Authors: Xingjian Hu
Ziqian Zhang
Yue Huang
Kai Zhang
Ruoxi Chen
Yixin Liu
Qingsong Wen
Kaidi Xu
Xiangliang Zhang
Neil Zhenqiang Gong
Lichao Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Large Language Model
Evaluation
PageRank
Primary area: datasets and benchmarks

Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose Empirical Interaction Propagation (EIP), a novel framework designed to quantify both question difficulty and model competency. EIP introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. EIP's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of EIP is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. EIP achieves 90\% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation. Code is available at https://github.com/Leozz04/EIP.

Published as a conference paper at ICLR 2026
EIP: WEIGHTED RANKING OF LLMS BY QUANTIFYING QUESTION DIFFICULTY
Xingjian Hu', Ziqian Zhang'; Yue Huang", Kai Zhang', Ruoxi Chen Yixin Liu', Qingsong Wen", Kaidi Xu Xiangliang Zhang", Neil Zhenqiang Gong", Lichao Sun 17
Lehigg University, University of Notre Dame, 3Zejjang Wanli University, "Squirrel Ai Learning, CCiyy University of Hong Kong, Duke University
ABSTRACT
Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective compar- isons and driving advancements in the field. However, existing benchmarks fail differentiate question difficulty, limiting their ability effectively distinguish models' capabilities. To address this limitation, we propose Empirical Interaction Propagation (EIP), novel framework designed to quantify both question diffi- culty and model competency. EIP introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. EIP's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of EIP that model earns a competency score when it correctly answers question, while question's 's difficulty score increases when it challenges model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. EIP achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-awart LLM evaluation. Code is available at https://github..cm http:/giihhu.oooo/oo////////////
INTRODUCTION
Large Language Models (LLMs) have shown impressive breadth across tasks from natural language understanding to multi-step problem solving (Young et al., 2018; Thirunavukarasu et al., 2023; Hadi al., 2023; Kaddour et al., 2023; Zhou al., 2023). As capability grows, reliable evaluation becomes the community's 's dashboard. However, popular benchmarks-e. c.g.. MMLU-Pro (Wang et al., 2024) and MATH (Hendrycks al., ttycallyy collapse performance accuracy within topical categories, implicitly treating all items as equally informative. This masks differences that hinge on difficulty and can flip model rankings when the mixture of easy vs. hard items shifts. Counting single arithmetic fact and a multi-step calculus derivation as equally "correct" illustrates the problem: the evaluation fails to distinguish routine pattern matching from advanced reasoning.
We posit that difficulty must be modeled explicitly at the item level. We introduce Empirical Interaction Propagation (EIP), simple, robust, and scalable framework that jointly estimates question difficulty and model competency from observed successes/failures. Inspired by PageR- (Page et al., 1999), EIP constructs a directed bipartite interaction graph between models and questions and performs damped bidirectional score propagation: solving hard question carries more evidential weight for competency; failing an easy question contributes more weight to difficulty. The process converges unique stationary solution, yielding difficulty and competency scores that support difficulty-aware comparison at scale. Compared to Item Response Theory (IRT), HIP is non-parametric, avoids per-item logistic fits, and runs in linear time in the number of model-quession interactions, making it practical when per- per-model sample sizes are small and datasets are large.
"Equal contribution, random order
1 "orresponniig author.

---

Published as a conference paper at ICLR 2026
Quastions
Model
Rinary Performene Marrie
EIP Store
Re: trytngg
behwern two fora
eurpree party
O
Queriiso DiTicully
Model Model Competency
manyy proede
be equall
pcosasen
veruce to
DeepSeek
Liana
Mu
Qu:75.93
43-94
Lama- 3-2 2g
Gemini 92
where
We baw sianale ABC
Chorser
Owen
M.
Qui 10.94
DespSack V3:
&. lat 36 the milppinn
longtt
Prepauation
Figure 1: Schematic of EIP's weighted ranking pipeline.
What EIP brings (methodologiial advantages). We highlight five core advantages that make EIP practical and reliable for today's 's evaluation regimes: 1) Human alignment: difficulty estimates reach 90% consensus with human judgments, outperforming multiple IRT baselines across datasets; 2) Efficiency and scalability: propagation converges in 0.006 consumer hardware and scales linearly with the number of model--uestionn interactions, supporting tens of thousands of items and dozens of models; 3) Robustness: question-side difficulty rankings remain highly stable under removal of individual models (p- = 0.998), while model-side competency rankings remain stable under dataset perturbations (p > 0.99); 4) Sensitivity beyond accuracy: in controlled simulations, EIP recovers subtle performance gaps missed by accuracy- and IRT-basee baselines, rewarding performance on harder items and enabling meaningful re-ordering among close models (adjacent changes consistent 0.8492): 5) Simplicity and reproducibility: non-parametric graph procedure
with Kendall's with a single damping hyperparameter, no per-item calibration or curve fitting is required, enabling easy, consistent deployments.
What EIP reveals (empirical findings). Applying EIP at scale surfaces actionable empirical patterns: 1) Detusst-specific difficulty profilex: MATH and MMLU-Pro exhibit broader difficulty tails suited for advanced reasoning, whereas GSM8k and HellaSwag skew casier; 2) fadilyyamily consistency: families preserve characteristic difficulty patterns across parameter scales scales-scaling scaling primarily shifts absolute accuracy rather than the relative difficulty structure; 3) Open-weight model potential: difficulty estimated on open-weight pools closely tracks full-pool results (Spearman 0.96, Pearson 0.94, Kendall 0,85), rivaling proprietary pools; 4) Diversity benefits: mixed-scale model pools reduce extreme misestimation by 83% relative to homogeneous pools and best align with human judgments (90% consensus); 5) Size-performance correlation: larger and proprietary models dominate the top competency scores, consistent with observed scaling trends.
We evaluate EIP across 30 models and 35,550 questions spanning BBH, GPQA, GSM8K, HellaSwag, MATH, and MMLU-Pro. Our study covers human alignment, convergence, robustness under model/- dataset perturbations, and controlled simulations. Figure I illustrates the joint estimation of question difficulty and model competency.
Contributions. This work makes three key contributions: 1) A new evaluation framework. We introduce Empirical Interaction Propagation (EIP), a difficulty-aware and paraamamttric approach that jointly estimates question difficulty and model competency. This allows evaluation to move beyond flat accuracy and provide finer-grained, difficulty comparisons of LLM performance. 2) Comprehensive empirical validation. Through experiments six benchmarks covering 35,550 questions and 30 models, EIP demonstrates strong aligmment with human difficulty judgments (90% agreement), scalability to large evaluation settings, and robustness to changes in the model pool and dataset composition. 3) New insights into models and datasets. By applying EIP at scale, we reveal distinctive dataset difficulty profiles, consistent family-level patterns across model scales, and the reliability of open-weight and mixed-scale pools for producing stable difficulty estimates, offering practical guidance for benchmark design and model selection.
2 EMPIRICAL INTERACTION PROPAGATION (EIP)
Method overview. Question difficulty plays central role in evaluating model performance. However, difficulty is inherently abstract and cannot be precisely defined. To address this, we operationalize difficulty through model failure: a question is considered difficult if even competent models are unable to solve it. EIP then evaluates question difficulty and measures model competency through an interactive process. EIP formulates questions and models as interdependent nodes within a directed bipartite graph, where questions' difficulty scores propagate to models that successfully solve questions, and models' competency score contributes to questions that they fail to answer correctly. We model this interconnection as ergodic Markov chain defined on the bipartite graph.
2

---

Published as a conference paper at ICLR 2026
Bipartite Graph & Matrices
Initialize distribution vector 1g *
Iterative Score Propagation
Inputs
Damping a (0,1)
Update Ri2
Update MM
Questions Q
af M- (L-a)
Row-stochastic transitions metrix:
Stationary distributtin: Eig
Competens//ifficulyy PQ-2M Gang (Alar) PM-- cluel Alg)
not converged
connorggd
Check convergence
&g Isq
(c+1)
lb- dir 1iaat
(te))
Binary answers correct/ incorrect
bg
R.c
Initialize Comp/Diff matrices 1)0- (195M -AT
Figure 2: Detailed process demonstration of score propagation in EIP, which includes Evaluation Inputs. Bipartite Graph & Matrices, and Iterative Score Propagation.
We prove that this iterative process converges asymptotically to unique stationary distribution, yielding a set scores: Tm for model competency and Ty for question difficulty. higher 7m value signifies model that consistently solves challenging questions, while higher my value indicates question remains difficult even for competent models. These scores are mutually dependent, providing self-reinforcing assessment of both models and questions.
2.1 SCORING IN EIP
We define the difficulty score Ty for questions and the competency score Tm for models based on the stationary distribution coupled random walk process. This formulation reflects the intuition that question difficulty is not an inherent property but rather emerges from the opposing patterns of model successes and failures. To enable bidirectional propagation between models and questions, establish two reciprocal information flows. First, when model successfully answers question the difficulty weight of g is distributed to m in proportion to q total solvers. Conversely, when model m fails to answer question q, its competency weight is transferred q in proportion the total number of failures m has encountered. Thus, the model competency 7m and question difficulty Tg are defined as:
X
S(g)
F(m)
(1)
(m)
(a)l()(g))
where S(q) represents the number of models that correctly solved q, and F(m) denotes the total number questions that model m has failed.
2.2 DIRECTED BIPARTITE GRAPH AND PERFORMANCE MATRICES
We formalize LLM-question interactions using a directed bipartite graph g = (V,E). The vertex set V = .M U Q comprises M models M = {m,.... m and Q questions Q (9......90) (typically M & Q). The edge set Ecomp F signifies model performance: Competency Edges (CCom)): ): A directed edge (q: + my)) E &Comp Q x M indicates that model m, correctly answered question qi Difficulty Edges (Eran): Conversely, a directed edge (m, - q:) E Epail S M x Q
indicates that model my failed answer question 4s.
These interactions are encoded into two complementary binary matrices, constructed after filtering questions: Competency Matrix (A): This matrix A {0, 1]@XM has entries A., II(( my) model mj correctly answered question q. Difficulty Matrix (Å): This matrix À E {0, M MxQ, expressed as Å = (1 QxM A)T, is the transposed complement of A. Its entries A,x- T[(m, + q:) E Erai] indicate if model m, failed question q:-
Ecomp), where A;
Prior to matrix construction, we exclude questions universally solved by all models or failed by all models, ensuring 0 E < M for all retained questions. This step guarantees graph connectivity and avoids trivial solutions. Such extreme cases are rare (e.g., in our experiments with 35,550 questions and 30 models, only 2% of questions were universally solved or failed, while no models showed 100%/0% accuracy) and are assigned conceptual lowest/highest difficulty. These matrices map the graph's 's edges to linear operators.
3

---

Published as a conference paper at ICLR 2026
2.3 PROPAGATING SCORES VIA ITERATIVE REFINEMENT
Having established the directed bipartite graph, formalize the score propagation. EIP employs an iterative refinement process to mutually determine question difficulty scores (TQ) and model competency scores (TT) The core of this process relies two row-stochastic transition matrices:
Competency transition (PQ- PQ-M MM = (A1M)) Given a question q, the probability of transitioning to model m is proportional m's success on q. The g-th row normalized by (g), the number of models that correctly solved
Difficulty transition (Par-1):: Pas-QQ diag(All)~ Å. Given a model m. the probability of transitioning to a question q is proportional m failure on q. The m-th row normalized by F(m). the number of questions failed.
Scores are updated iteratively. An underlying random walk on the bipartite graph (Ω ++ M) would be 2-periodic. To ensure convergence to unique stationary distribution, we introduce damping factor a E (0, 1), representing a "teleportation" probability, The iterative update equations for the scores step & are:
(1) (++1) = a.T ar M + (1 a1- Q
lQ Q
(++1) iT M
(t+1) + (1 a) a Q-
1M
(3)
M
Here, 1Q and 1M are all-ones vectors of appropriate dimensions, Q = 1Q). and M = MM The transposes Pæ and P facilitate score propagation. "M
(++1) uses the just-updated TQ 1++1)
This iterative process, where competency and difficulty scores are mutually reinforced, guaranteed to converge to unique stationary distributions тे and т. This is because the damped system forms an ergodic Markov chain, whose properties are established by the Perron-Frosd Frobenius theorem (Perron, 1907). A formal proof of existence, uniqueness, and convergence is provided in Appendix C.
2.4 EXTENDING EIP TO CONTINUOUS SCORES
Benchmarks that grade free-form answers often provide partial credit rather than binary outcomes. EIP accommodates this setting by allowing the response matrix take values in the unit interval. Let A, E [0, 1] M denote this continuous analogue of the correct-response matrix (c stands for "ccontinuous") All subsequent quantities mirror the binary case after replacing with A.: The row sums SQq) = I (Ac)gm represent the total score achieved on question q, with questions satisfying 5(q)= = 0 removed as before. The complement matrix A 10xM Ag and corresponding column sums F F(m) (A,) maintain the same interpretation, with F(m) > 0 ensured in practice. The transition matrices retain the same form:
P = diag(S)=1
(4)
PM- = diag(F))))FF -
(5)
The iterative updates in Equation 2-3 and convergence guarantee of Theorem C.1 apply unchanged.
3 EXPERIMENT
3.1 EXPERIMENT SETUP
Datasets & benchmarks & models. We select di- verse and representative set of datasets that encompasses diverse domains, including math, science, natural lan- guage understanding, and programming. We show the details of selected datasets in Table 1. We aggregate a diverse selection of 30 LLMs, covering a wide range of sizes from 0.5B to hundreds of billions parameters The models cover a diverse range and incorporate both open-source and proprictary models to ensure a comprehensive evaluation. The details of selected models shown in Appendix H.
Table 1: Details of selected datasets.
Name
Questions
BBH (Surgun et al., 2023)
6511
GPQA (Rein
GSMS (Cobbo et al., 2021)
1319
HellaSwag (Zellers al.,
10042
MATH (Hendrycks et al., 2021b)
5000
MMLU-Pro (Wang al. 2024)
12032
4

---

Published as a conference paper at ICLR 2026
100
90
EIP
Accuracy
Nom Eip &
70
60
40
30
20
10
Deepseak
Gemini-1 PrPro
Gemini-1 1.mini-1.5-fach fach
GPT-4
GPT-40 mini
Yi-1 German2 2 27b
Owen2 5-78
Phi-3-medium -eddlu
Phi-3 Phi-3.5-minn 5-mini
Gemma-2 8b
Qwen2-78
DeepSeak- Coster-Lu Lite
Llama-3. 1-88
Qwen2 5-38
Phi-a B-mmal
Owen Owen2-Matt-- Math
Owen2-Math- 1.58
Vi-1.5-98
5-98
Y-- 1,5888
OwenZ .55
QwenZ Gavnn M-Math-11.. 1.
Mistral- Aastral-7B--1..
DeepSaek Shekk-hh-htt-L Chat-L Liee
QwenZ-0 5B
Qwen2-1 1.58
Owan2 2-1. 58
Gemma-2 2b
Liama-3 2.338
Llama-3.2 2-18
Figure 3: EIP scores and accuracies of models, both normalized to a fixed maximum of 100 to facilitate direct comparison. The full names of the models are provided in Appendix H.
3.2 MAIN RESULTS
EIP offers finer-grained, difficulty-aware insights beyond traditional accuracy-based ranking. A key distinction between EIP and traditional accuracy-based evaluations lies in how question difficulty is incorporated. Traditional accuracy metries treat all questions equally, which can lead to underestimating models whose overall accuracy is limited but whose strength lies in solving difficult questions. In contrast, EIP explicitly incorporates quantified question difficulty, enabling a more nuanced evaluation of model performance.
The positive correlation between EIP scores and accuracy (Kendall's Tau T 0.8492; see Ap- pendix G) aligns with our expectations, as stronger models typically answer more questions correctly and consequently receive higher scores. This validates that EIP serves as an enhancement to tradi- tional accuracy-based ranking rather than a complete departure from it. However, Figure 3 reveals that significant discrepancies in model rankings frequently emerge, particularly among adjacent models. While the overall trend preserves the general hierarchy between clearly superior and inferior models, substantial rank changes frequently occur between closely-performing models, highlighting EIP's ability to provide finer-grained differentiation. In particular, models with high raw accuracy may receive lower EIP scores than their neighbors, while some lower-accuracy models are ranked their stronger performance more difficult questions. For instance, Qwen2-0.55 (20.2%) is ranked higher than DeepSeek-Chan Chate--hhhh-ttttte (30.49%). This re-ranking stems from HIP's central design: assigning greater credit to correct answers on more challenging questions. Figure 11 supports this observation: Qwen2-0.5B correctly answered 5.5% of hard questions, compared to only 2.4% for DeepSeek- DeepSeek-Chat-Lite, which substantially boosts its EIP score. A similar pattern is observed among top- perpforrrning models: while GPT-4o achieves higher overall accuracy, Gemini-1.3 1.r- receives higher EIP score due to answering 5% more medium- and high-difficulty questions. This illustrates how EIP identifies strengths on difficult questions that are otherwise obscured by flat accuracy scores-a a finding consistent with our simulated Case Study (section 4).
Model families maintain consistent difficulty patterns despite scaling effects. As shown in Figure 5, our leave-one-out evaluation demonstrates that models within the Llama-3 family (Llama- 3.I-8B, Llama-3.2-33 2it.-3..233B-nssttte
Llama-3.2- 1-B---B exhibit stable difficulty-specffic answering distributions across different parameter scales. Despite substantial differences in overall accuracy (51.2% 21.0% + 10.2%) and EIP scores (44.3% - 23.9% + 15.4%), all variants preserve similar distribution of correct responses across difficulty levels (hard / medium / easy). Notably, the 8B model demonstrates slight disadvantage in hard-question accuracy (11.1% 11 compared to 15.7% and 15.6% for the smaller variants), yet maintains nearly identical easy-question accuracy (62.9% compared to 60.4% and 61.5% for the smaller variants). This trend extends across other model families, including Qwen and Yi (Appendix F), indicating that: 1) Accuracy scaling primarily influences absolute performance rather than altering relative difficulty distributions. 2) Traditional accuracy metrics obscure essential behavioral consistencies across model scales.
IIn EIP, a model is categorized small if has fewer than 10B parameters, medium if its size ranges between 10B and 50B parameters, and large if it exceeds 50B parameters.
21 Sewee-One-uui is vross-validation method that iteratively uses each sample in the dataset as the validation set, while the remaining samples form the training
5

---

Published as a conference paper at ICLR 2026
Laama-311-8
Hamm--32-36--nme
Lamm--322-88-
Gemt: 15
RanKLLM Scoro:44.3
Accuracy:51.2
RankLLM Score:23.9
Accuracy22.0
RankLLM Score:15.4
1
Awerd
09: mer
Barma2
188
10.%
Deopleet
eg Fore
Y--155-448
Y-1.5-98
(+-15-68
Oeerg 2al2-
Accuracy60.2
Accuracy:44.0
Accuracy:38.3
RanKLLM Score:53.3
RankLLM Score:39.5
RankLLM $core:36.7
Over
200
Sward.655
Deepleec
Geen2 5-6 650
Lanss
$2A9
12.%
20
100
Figure 4: Relationship between model perfor- mance and parameter scale.
Figure 5: The proportion of Fasy/Medium/HHat questions within correctly answered samples across Llama/Yi variants.
Spearman
Pearson
Kendall
69)
2.00
00
5:00
0.18-
00
OW Ve
1.00
0.52
481 1.00
0.4
bothorge
0.32
00
0-70
Bath
10 00
000
3.70
0.80
9 16 1.32
0.24
2.79 0.65
0.640070 60
2.00
Prtmmee
oetural
Batferked fethtural
Offabee
Prtmmua
Autforal
Kathaeodd
Prtsess
Group
Group
Figure 6: Correlation analysis across model group combinations (Pro. means proprietary, and OW means open weight).
Open-weight models show strong potential in estimating difficulty, rivaling proprietary models. To evaluate the consistency of question difficulty distributions across model groups, we employed three correlation methods: Spearman, Pearson, and Kendall. These methods, capturing monotonic, linear, and rank-based correlations respectively, yielded highly consistent results, indicating strong agreement in the diffieulty rankings produced by different model sets. As shown in Figure 6, the difficulty distributions derived from various model groups exhibit significant correlations. The difficulty distributions generated by open-weight models of all sizes (OW (OW(whole) (Whole)) demonstrate a strong positive correlation with those generated by the entire set models (Both( Quantitatively, the Spearman, Pearson, and Kendall correlations correlation values across all three metrics indicate a strong agreement between the difficulty rankings produced by open-weight models and those produced by the full model set, suggesting that open- weight models alone are capable of capturing the overall difficulty landscape as represented by EIP. The difficulty distributions derived from all proprietary models (Pro,)hole)) show correlation with those derived from all models (hote), with a correlation of 0.78, 0.84, and 0.64. While still demonstrating agreement, there is a noticeable drop in the Spearman correlation compared to the weight models.
0.96, 0.94, and 0.85, respectively. These high
Consensus (941
Caher's
Canseroux 188
61.4%
3.8
75
BP S.ora
50
6%
25
0.13
0.2
3.0
Large
Small
Medum
Large
Whole
Figure Consensus alignment with human judg- ments on ElP-estimated question difficulty by model size group.
Figure Difficulty distribution grouped by
model size.

---

Published as a conference paper at ICLR 2026
Table 2: Alignment analysis with human evaluation on question difficulty (all metrics in %).
V/3
V/
V/
Consensus (%)
Simple Rank
41.4
54.3
50.0
60.0
51.4
54.3
54.3
50.0
50.0
...
52.9
62.9
IPL-IRT
44.3
52.9
2PL-IRT 35.7
45.7
41.4
54.3
52.9
42,9
44.3
47.1
47.1
42.9
51.4
Multi-IRT'
50.0
50.0
48.6
54.3
48.6
57.1
47.1
47.1
44.3
48.6
52.9
70.0
71.4
61.4
64.3
57.1
diverse model pool mitigates extreme question-difficuly estimates in EIP. As shown in Figure 83, analyze the impact of model diversity difficulty estimation by running EIP with four different annotator model pools: small-only, medium-only, large-only, and a combination of all (Whole). Our findings highlight that the composition of the model pool used by EIP plays crucial role in determining the quality of difficulty estimation. Homogeneous model groups tend to exhibit more extreme difficulty distributions. Specifically, the Large group classifies 58% of questions as "overly easy". while the Small and Medium groups contain over 30% "impossible" questions. These extremes introduce high proportion of "dead nodes" in EIP's algorithm, where models either consistently overestimate underestimate question difficulty. In contrast, the Whole group, which includes models of varying scales, achieves a more balanced difficulty distribution. Complementary error patterns across different model scales reduce extreme cases to below 3%. Moreover, the mixed- scale ensemble maintains meaningful bidirectional score propagation in EIP, reducing dead nodes by 83% compared to homogeneous groups. These results align with the "wisdom of the crowd" principle, demonstrating that incorporating diverse models not only mitigates mutual biases but also preserves sufficient granularity to prevent systematic overestimation or underestimation in difficulty assessment.
Difficulty distribution varies by dataset. As shown in Figure 9, the six benchmarks exhibit distinct dif- ficulty profiles. GPQA has a relatively uniform distribution, making it ideal for evaluating perfor- mance across a range of difficulty levels. In contrast, GSM8k, HellaSwag, and BBH are skewed toward easier questions, with most items concentrated at lower EIP difficulty scores. This skew may limit their utility in assessing performance on harder tasks but provides insights into models' fundamental capabil- ities. MATH and MMLU-Pro, on the other hand, display broader and more dispersed distributions, with MATH emphasizing harder questions and MMLU-Pro displaying bimodal pattern, blending challenging items. These benchmarks, therefore, better suited for evaluating advanced reasoning capabilities and the robustness of models.
CONEK
HeelaGowgg
0.03
0.30
Score
Figure 9: Question difficulty distributions for six benchmarks.
3.3 HUMAN EVALUATION
To validate the alignment of EIP with human judgment, we conducted controlled blind trial involving 20 human evaluators (V1-V20). Each evaluator compared two questions from the same subject category (e.g., mathematies) to judge which more difficult. There 70 randomly ordered question pairs in total. A detailed evaluation protocol is in subsection H.3.
Baseline. As comparisons, selected the Simple Rank method primary baseline. Additionally, we evaluated three Item Response Theory (IRT) based models: Ipl-irt, 2pl-irt, and multi-irt. Configu- ration details for these IRT models are provided in subsection H.4. Simple Rank determines question difficulty based solely on the number of models that answered incorrectly, assigning higher difficulty to questions with higher error counts.
EIP achieves high alignment (90%) with human consensus. As shown in Table 2, EIP achieves superior agreement with human judgments for 18 out of 20 evaluators compared to Simple Rank, with an average margin of +9.3% % in individual alignment scores. The "Consensus" column in Table 2 aggregates information via majority vote and resolves near-ties. Overall, EIP achieves 90% agreement with the human consensus, highlighting its strong alignment with human judgment.
more diverse model pool improves human alignment difficulty judgments. As shown in Figure 7, the whole group, comprising models with various sizes, achieves a consensus alignment of
Area above 100 represents questions that all models failed, helow all correct answers.

---

Published as a conference paper at ICLR 2026
90.0% with human judgments question difficulty, along with Cohen coefficient of x of 0.80. In contrast, the homogeneous size groups from the SOTA large model to the 0.3B small models exhibit markedly lower consensus 105.6%%-.4.39 and weaker agreement levels (x 0.13 0.29. This strongly suggests that diversity in model size leads to richer and more robust difficulty assessments, reducing the bias often observed when the annotator pool is restricted to single-size scale.
3.4 EFFICIENCY AND SCALABILITY ANALYSIS OF HIP
Given the rapidly growing number of models (M) and extensive question sets (Q) used in benchmarks, computational efficiency and scalability of the evaluation framework are paramount. We therefore analyze these aspects of EIP both theoretically and empirically.
EIP demonstrates strong computational efficiency and scalability. EIP has time complexity O(IQM) where / is the number of iterations required for convergence (proof in Appendix D). The iteration count remains stable with preset damping factor a and tolerance e, independent of Q or M (Appendix C). Since M < Q in typical LLM evaluation scenarios, the per-iteration cost ((QM) remains manageable for large-scale applications.
Empirical validations further underscore EIP's effi- ciency and scalability. In a direct comparison on
Table 3: Convergence speed of EIP on consumer-grade hardware (Intel i7).
dataset of 30 LLMs and 35,550 questions. EIP achieved convergence substantially faster (in 0.00597 seconds) than all Item Response Theory (IRT) base- lines, including 1PI--RRT, 2PL.IRRT, and Multi-IRT (Table 3). Scalability was further assessed using synthetic response matrices with Q ranging up to 1,000,000 and M up to 2,000. Across all these set- tings, EIP consistently converged in constant number of iterations (9 in experiments), and its average per-iteration time increased linearly with the total number of interactions Q x M, aligning with its theoretical complexity (Table 4).
Method
Convergence Time (s)
EIP
0.00597
IPL IRT
1782.75
2PL IRT
3787 .03
Multi-IRT (3D)
18.76
These combined theoretical and empirical results confirm that EIP not only converges quickly but also scales efficiently with the size of the evaluation. Given the low computational overhead, deployment costs are negligible even for extreme-scale evaluations. On AWS EC2 (approximately $0.05 per vCPU-hour), EIP enables cost-efficient deployment for large-scale LLM evaluation. We have established Hugging Face leaderboard platform that supports continuous updates of new models and benchmarks, capable of handling hundreds of daily updates while maintaining cost efficiency. This makes EIP a practical and robust solution for ranking large numbers of language models across extensive question sets.
Table 4: EIP convergence time under varying numbers of models and questions.
Questions (Q)
Models (M)
Q x .M
Iterations (T')
Total Time (s)
Avg. Time / Iter. (s)
1,000,000
2,000
2.00 x 10
5.28
0.5867
1,000,000
1,000
1.00 x 10
9
3.14
0.3489
500,000
1,000
0.50 10
1.93
0.2144
1,000,000
500
0.50 x 10"
9
2.18
0.2422
500,000
500
0.25 10
9
1.13
0.1256
500,000
250
0.125 x 10
0.64
0.0711
250,000
500
0.125 x 10
9
0.52
0.0578
250,000
250
0.0625 10
0.30
0.0333
3.5
ROBUSTNESS AND EXTENSIBILITY ANALYSIS OF EIP
Given the rapid pace at which new models are developed and integrated into evaluation pipelines, it is crucial that evaluation systems remain both stable and extensible. To this end, we examine the robustness and extensibility EIP under dynamic model pool configurations, ensuring that its scores remain consistent even models are added. To simulate the introduction of new models, we emulate pool variation by randomly removing & models (ranging from to 15) from the original set of 30 models. For each value of k conducted 50 independent trials and computed the mean and standard deviation of the Spearman correlation p between the resulting scores and those obtained with the full model pool.
8

---

Published as a conference paper at ICLR 2026
Even under large-scale modifications to the model pool, EIP preserves stable rankings of both questions and models. This is substantiated by the results in Table and visualized in Figure 10. Even when half of the model pool (15 out of 30 models) where a large influx of new models suddenly enters the system-the mean for question difficulty scores remained high at 0.9382 when compared rankings from the full pool. Model competency rankings exhibited even greater resilience under the same 15-model removal condition. These findings indicate that EIP's relative assessments of both questions and models are largely preserved despite considerable variations in the evaluation pool's 's size. Concurrently, a significant practical benefit observed was the reduction computation time, demonstrating improved efficiency with smaller model sets without a critical loss in ranking fidelity. For other detailed experiments and observations such as the removal of datasets, please refer to Appendix E.
removed -imirvadiii scenario
Table Impact of randomly removing k models on EIP's stability and computation time (averaged over 50 trials).
Models
Question Correlation
Model Correlation
Computation Time
Removed (k)
Mcan p
SD
Mcan p
SD
Avg. Time (s)
% Reduction
1
0.9978
0.0021
0.9997
0.0002
0.0079
14.09
5
0.9850
0.0073
0.9988
0.0009
0.0080
13.57
10
0.9684
0.0107
0.9977
0.0019
0,0066
28,28
15
0.9382
0.0187
0.9934
0.0069
0.0062
33.02
CASE STUDY: VALIDATING DIFFICULTY-BASER EVALUATION
Due to the lack of ground truth in large-scale evaluations, identifying model capabilities on difficult questions challenging. We address this through a controlled simulation where model performance is defined by question difficulty, enabling clear assessment of how different evaluation methods capture performance differences across difficulty levels.
Simulation Setup. We created five hypothetical models (M1-M5) and 100 questions (70 easy, 21 medium, 9 hard) to assess whether evaluation methods can detect subtle performance differences among models with similar overall accuracy. Models were configured to establish ground truth ranking: MI M2 > M4 > M5 M3. MI/2 had equal accuracy, but MI answered more hard questions. M4/5 had equal accuracy, but M4 performed better medium-diificn items.
We applied EIP alongside baseline methods to simulated responses: 1) Standard accuracy: Raw percentage of correct answers. 2) hitttstt-diifieultt--eeehhee
Baseline.
accuracy: Weighted average of accuracies on "Fasy" and "Hard" subsets, with weight wa = - aa for dataset d, where âa is average accuracy of all models that dataset. 3) Item Response Theory ((Van der Linden, 2018)): IPL, 2PL, and multidimensional IRT models. Configuration details are in subsection H.4.
Table 6: Model rankings under different evaluation methods, with v indicating agreement with the ground truth ranking.
Model Salved Qs
EIP
Accuracy
WeightedScore
IPL-IRT
2PL-IRT
Multi-IRT
Truth
Score
Score
Score
Rank
Score
Rank
Rank
MI
70/10/5
100.00
85
77.01
100.00
100.00
38.91
3
1
M2
70/11/4
95.50
85
78.79
100.00
84.14
0.00
5
2
M3
4//13/0
56.88
60
5
$8.86
0.00
5
19.66
25.30
2
5
M4
46/15/0
59.85
61
50.28
3.02
0.00
100.00
]
3
M5
47/14/0
58.21
61
$8.86
3.02
19.30
16.52
MIsM2 Correct?
/
M4-M5=M3 Correct?
X
Only ETP achieved rankings consistent with ground truth (Table 6). EIP distinguished MI from M2 by capturing hard question performance differences and ordered M4 > M5 M3 based on medium-difficule distinctions. Baseline methods showed limitations: Standard accuracy treats all questions equally, failing to distinguish models with identical accuracy but different hard question competency. Dataset-weigee weighted accuracy uses coarse-grained weighting
Table Difficulty scores of simulated questions calculated by EIP.
Category
Quantity
Mean
Easy Medium
70
18.24
0.45
21
73.44
1.56
Hard
9
98.59
1.34

---

Published as a conference paper at ICLR 2026
that ignores intra-dataset variation, allowing models score high by answering easier items within "hard" sets. IRT-based methods (1PL, 2PL, Multi-IRTI showed inconsistent rankings, with Multi- IRT diverging significantly from ground truth.
Analysis of Simulation Results.
These findings highlight EIP's unique ability to capture fine-grained performance differences through instance- difficulty estimation, while conventional methods fail due to uniform weighting assumptions, aggregation bias, or insufficient data robustness.
CONCLUSION
We introduced EIP, a difficulty-awan framework for evaluating large language models. By jointly modeling question difficulty and model competence, EIP provides more fine-grained insights than traditional accuracy based metrics. It achieves 90% agreement with human difficulty judgments, surpasses IRT-based and heuristic baselines, and maintains stable rankings under model or dataset perturbations. Moreover, it converges efficiently in large-scale settings. These results establish EIP as a scalable, robust, and human-aligned alternative for LLM evaluation.
ACKNOWLEDGEMENT
We thank Hanchi Sun for mathematical support, including insights on the random-walk formulation and help with the derivations and proofs underlying damped bidirectional score propagation mechanism. We thank Yuan Guo for early involvement in the project and for contributions to the initial exploration and prototyping that informed this work. We also thank Guanshen Meng for assistance with the visual design of our figures, including color palette refinement and aesthetic improvements. This work is also supported by the Delta/DeltaAl systems at NCSA and the Bridges-2 system at PSC through allocation CIS240308 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants 2138259, 2138286, 2138307, 2137603, and 2138296.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We are committed to ensuring that our results can be independently verified and extended. All source code and prompt templates used in our experiments are included in the supplementary materials. The repository includes: (i) a step-by-step README describing the environment setup and execution commands; (ii) hosted links all model version and formatted benchmark datasets; and (iii) configuration files specifying random seeds, seperr-arameters. and compute resources. Human-evaluation protocols are likewise included. Together, these artifacts enable an end-to-end replication of our experiments any machine with standard GPU resources.
ETHICS STATEMENT
EIP is an evaluation framework and does not generate new user-facing content. All benchmarks employed are publicly available and distributed under permissive licenses. personally identifiable sensitive data processed. Nonetheless, difficulty-awan rankings could conceivably be misused to dismiss models that prioritize safety or fairness over raw competence. We therefore release our code under open license and encourage responsible adoption that complements, rather than replaces, broader alignment assessments. We disclose all model weights evaluated and provide guidelines for reproducing the study without circumventing dataset usage terms.

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: //arxiv. org/ abs/ /2//2444 220444044000 1.8..12211 14219.
01. AI. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang. Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01 01.ai, 2025. URL https: aaraiiiiivv.//.. org/ abs 2403. 04652.
Anonymous. A survey of ai scientists. arXiv preprint arXiv:2510223045, 2025. URL https: //laxii.orrg/aee htargg/bb//55/0 23045.
Marthe Ballon, Andres Algaba, Brecht Verbeken, and Vincent Ginis. Estimating problem difficulty without ground truth using large language model comparisons. arXiv preprint arXiv:2512, 14220, 2025. URL https:/ / arxiv. org/ abs/ 2512. 14220.
Han Bao, Yue Huang, Yanbo Wang. Jiayi Ye, Xiangqi Wang, Xiuyin Chen, Mohamed Elhoseiny, and Xiangliang Zhang. Autobench-v: Can large lssounlanguuae models benchmark themselves? arXiv preprint arXiv:2410.2 1259, 2024.
Andrew M. Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A. Chi, Ryan Chi, Scott A. Hale, and Hannah Rose Kirk. Lingoly: A benchmark of olympiad- linguistic reasoning puzzles in low-resource and extinct languages, 2024. URL https: wwtx/vavvivve///a///aaaaaaaa org/ abs 0.06666196
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang, Kaijic Zhu, Hao Chen, Xiaoyuan Yi. Cunxiang Wang, Yidong Wang. Wei Ye, Yue Zhang, Yi Chang, Philip Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15 (3), mar 2024. ISSN 2157-6904. doi: 10. 10.1145/3641289, URL https: /ttp://oo...o...//////o 1145/ 3641289.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yanchen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: Assessing multimodal lin-as-a-judee with vision-language benchmark. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235, pp. 6562-6595. PMLR, 2024a. URL https:t////// htttt::://ppcccedngge. mlr.press / v25/ che//2ch22////// 4y html.
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. Gui-world: A dataset for gui-oriented multimodal llm-based agents, arXiv preprint arXiv:2406. 10819, 2024b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. Evaluating large language models trained on code, 2021. URL ptattss///axiv.... 2107. 03374.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagenthench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2410.05080, 2024c.
Junhyuk Choi et al. Diagnosing the reliability of Ilm-as-a-judge via item response theory. arXiv preprint arXivv2602.0052), 2026. URL / wwaprrg/ab/////// http://///aaa...oo///////220.000 00521..21..
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https: / //tpxivvvv//////////////.//aa org/ abs/2111 14168.
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, et al. Deepseek-v3 technical report, 2024a. URL https: a/axxi w.t.go/bs/////222/ 19437.
12

---

Published as a conference paper at ICLR 2026
DeepSeek-AL, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, al. Deepseck-coder-v2 Breaking the barrier of closed-source models in code intelligence, 2024b. URL https:/ arxiv.org abs 41441.1111..
Mucong Ding, Chenghao Deng. Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Anima Anandkumar, and Furong Huang. Easy Bhsy2hard- bench: Standardized difficulty labels for profiling IIm performance and generalization, 2024. URL
https://arxiv: org/ alss 2409. 2409.18433. 18433.
Wanyong Feng and Andrew Lan. Gencat: Generative computerized adaptive testing. arXiv preprint arXiv:2202.22002, 2026. URL https: http:///aavvvvveoeeeeeaa org/ abs///a2..222// 20020.
Dennis Frauen al. Nonparametric Ilm evaluation from preference data. arXiv preprint arXiv:2601.21816 2026. URL
Chujie Gao, Siyuan Wu, Yue Huang, Dongping Chen, Qihui Zhang, Zhengyan Fu, Yao Wan, Lichao Sun, and Xiangliang Zhang. Honestllm: Toward honest and helpful large language model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Mark E Glickman. Example of the glicko-2 system. Boston University, 28, 2012.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama herd of models, 2024. URL https: / arxiv, www.:/org/aaaoo 227 21783.
Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662-59688, 2023.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021a. URL https: /wtp///ppeeeew..eee/eeeee net http://eeet/oommm////e forum?id= d7KB 3GmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL https::////./c.. /awxiiiiivv////ai///aaaaaaaaaavv org/ abs/ 03874.
Valentin Hofmann, David Heineman, Ian Magnusson, Kyle Lo, Jesse Dodge, Maarten Sap, Pang Wei Koh, Chun Wang, Hannaneh Hajishirzi, and Noah Smith. Fluid language model benchmarking. In Proceedings of the Conference on Language Modeling (COLM), 2025. URL https:// arxiv. org/ abs 2509 11106.
Jenny Y. Huang, Yunyi Shen, Dennis Wei, and Tamara Broderick. Dropping just handful of preferences can change top large language model rankings. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. URL https:: / openrevineeeeeeeeeeeeeeeee n
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, al. Metatool benchmark for large language models: Deciding whether to use tools and which use. arXiv preprint arXiv:2310.03128. 2023a.
Yue Huang, Qihui Zhang, Philip S. Yu, and Lichao Sun. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint a5Xiv:2306.11507, 2023b. URL https: // //arxiiv. //aaxivv.rg/aaaa.............. wwt.//////aaa//aa/aaa/...at.aaet /2306 11507.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark for large language models: tools and which to use. In The Twelfth International Conference Learning Representations (ICLR), 2024a. URL https:/ http////pee/veiieeee/e//e https:://openeewiie RDc2qtalgG.
Deciding whether
net feet/forumtt forum?ide
13

---

Published as a conference paper at ICLR 2026
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang. Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang. Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing, Furong Huang. Hao Liu, Heng Ji, Hongyi Wang. Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, Joaquin Vanschoren, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip Yu, Pin- Yu Chen. Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong... Chen, and Yue Zhao. Position: Trustllm: Trustworthiness in large language models. In Proceedings of the st International Conference on Machine Learning (ICML), volume 235, pp. 201 60-20770. PMLR, 2024b. URL https: / proceed ngs..... ml w.w..pres...... / v235/ h23.///aa/////22//// xx.. html.
Yue Huang. Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Trustllm: Trustworthiness in large language models. arXiv preprint (XX.v::2401.05561.1 2024c.
Yue Huang, Zhengqing Yuan, Yujun Zhou, Kehan Guo, Xiangqi Wang, Haomin Zhuang. Weixiang Sun, Lichao Sun, Jindong Wang, Yanfang Ye, et al. Social science meets llms: How reliable are large language models in social simulations? arXiv preprint arXiv:2410.23426. 2024d.
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang. Shichao Sun, Yang Xiao. Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang, Dahua Lin, Yu Qiao, and Pengfei Liu. Olympi- carena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai, 2024e. URL https: /arxivv.rrivvv///aa///a/ o abs 2400 12753.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al. Mistral 7b, 2023. URL https: /ttp//aavvvv//v////a////a//aaav/ org/ abs 2210. 06825.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arX1v:2307.10169 2023.
Peiyu Li. Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, and Nitesh V Chawla. Adaptive testing for I1m evaluation: A psychometric alternative to static benchmarks. arXiv preprint arXiv:2511.04689, 2025. URL /attts::/arrxee org/ abaaaaas///11 /211.. 04689.
Chumeng Liang and Jiaxuan You. Evaluating Ilm llm-generated diagrams as graphs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 12678- 12690. Association for Computational Linguistics, 2025. URL. https:: / /acll anthology
org/ / 2025. emn cmmnlpmmm 1p-mal-mmmmm n.. 640/.
Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18744 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating 1lms as agents. arXiv preprint arXiv:2308. 03688, 2023b.
Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11621-11640. Association for Computational Linguistics, 2024. URL https::////........ htt///aaaaaattooooopoooo wwt. acl- long.. 624/.
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-ssfetybench: A benchmark for safety evaluation of multimodal large language models. In European Conference on Computer Vision pp. 386-403. Springer, 2025.
14

---

Published as a conference paper at ICLR 2026
Alejandro Lopez-Ling and Yuehua Tang. Can chatgpt forecast stock price movements? return predictability and large language models, 2023. URL https::tp////... /atttp::/axxiv.org/// 07619.
Frederic Lord and Melvin Novick. Statistical theories of mental test scores. IAP, 2008.
Yingzi Ma, Jiongxiao Wang, Fei Wang, Siyuan Ma, Jiazhao Li, Jinsheng Pan, Xiujum Li, Furong Huang, Lichao Sun, Bo Li, Yejin Choi, Muhao Chen, and Chaowei Xiao. Benchmarking vision language model unlearning via fictitious facial identity dataset. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https:ttpp/.occ / 1411.03555
Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models inference tasks, 2023. URL
https://arxiv.ot org// 20to0//200..15555
OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, and Aditya Ramesh others.
Gpt-4o system card, 2024a. URL https: hwwph::tttaaaatt http:::://axxiv..rrro. orbbbs/////////ooooooo/////ss/
abs 2410 a241/ 21276.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. Gpt-4 technical report, 2024b. URL
abs /2t/2/233333 0133330.8880 774.
OpenAI, Josh Achiam. Steven Adler, Sandhini Agarwal, Lama Ahmad, al. Gpt-4 technical report, 2024c. URL https::t//// http://axxvvvv..oo.. hbtp:/bb//23000 08774.
Lawrence Page. Sergey Brin. Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Stanford InfoLab, 1999. URL https: dippbss st stanford. edu/pub/ 19999-9209 66. Technical Report.
Oskar Perron. Zur theorie der matrices. Mathematische Annalen, 64:248-263, 1907. URL http: //eudml.. wor.orgg//oooooo...o htrrg/doc/1583111
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https: / www.i.vvoro... aba/ 12022.
Ofrr Ben Shoham and Nadav Rappoport. Medconceptsqa: Open source medical concepts qa bench- mark, 2024. URL https: wwwii.iiiivv./..m www.rxii...rraaaaoommo abs///////255 07348.
Swadesh Sistla al. Evaluating Iims in open-source games. Advances Neural Information Processing Systems (NeurIPS), 2025. URL httttttss///xiiii org/ abw.////1... 00371.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. URL https: /aaxivv.vv.. wrg/ 04615.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them, 2022. URL https: : //tpxx/.x.oo///////ooo.../ org abs 2210. 09261.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens context, 2024a. URL https: /arxiv. org/abs halaab//244/ /2003 05530.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, et al. Gemma 2: Improving open language models at practical size, 2024b. URL https: /arxivvv.vv..////////a.a/aaaaav. org/ alss /2008 00118.
Arun James Thirunavukarasu. Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8): 1930-1940, 2023.
15

---

Published as a conference paper at ICLR 2026
Guiyao Tie, Xueyang Zhou, Tianhe Gu, Ruihang Zhang, Chaoran Hu, Sizhe Zhang, Mengqu Sun, Yan Zhang, Pan Zhou, and Lichao Sun. MMLU-Reason: Benchmarking multi-task multi- modal language understanding and reasoning. arXiv preprint arXiv:2505.1645, 2025. URL https: : / tttpriawwwrgg//eeeee.ee./../o.ot alos/ 2205. 6459.
W.J. Van der Linden. Handbook of Item Response Theory: Three Volume Set. Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences. CRC Press, 2018. ISBN 9781351645454, URL
bttps::/bbodt woogloo.oooooooooo.oooooooooooee com/ wwt.cookc/ceeeccccooooooooooo..t
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurlPS, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark, 2024. URL htwph::tt//.c....... /ttppss////xvvv////
abs 2406. 01574.
Zhuohan Wang, Ziwei Zhu, Ziniu Li, Yizhou Han, Yufeng Lin, Zhihang Lin, Angyang Gu, Xinglin Hu, Ruoyu Sun, and Tian Ding. Orgeval: Graph- evaluation of Ilms in optimization modeling. arXiv preprint arXiv:2510.27610, 2026. URL htthstt::: pectps://arxta /ttp///aavvv////////////eela.aaa org.iiioo.r...oooooooo..oo. abs 2510. 27610.
Siyuan Wu, Yue Huang, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xian- gliang Zhang, Jianfeng Gao, Chaowei Xiao, et al. Unigen: A unified framework for textual dataset generation using large language models. arXiv preprint arXiv:2406. 18966, 2024.
Zhiling Yan et al. Livemedbench: contamination free medical benchmark for Ilms with automated rubric evaluation. arXiv preprint arXiv:2602.10367 2026. URL https: : //axxaxxvvvv.o/.///aaa org/ /abs/ 2602. 10367.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 106.6 2024.
Kai-Cheng Yang and Filippo Menczer. Accuracy and political bias of news source credibility ratings by large language models, 2024. URL htt.t:////// rxiv.orr/abaa.... ab///30////343/ 00228.
Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. Recent trends in deep learning based natural language processing [review article]. JEEE Computational Intelligence Magazine, 13(3):55-75, 2018. doi: 10.1109/CC22018.244000
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence?, 2019. URL ht tps /ttp://aavvv//o///a//ee http///wwwwwooo///////////ooaa// 1905. 07830.
Ruohong Zhang, Yau-Shian Wang, and Yiming Yang. Generation-driven contrastive self-training for zero-shot text classification with instruction-following 11m. 2024. URL htttts:://axxi..rrrrt abs/2304 11872.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. Sentiment analysis in the era of large language models: A reality check, 2023. URL httppa/////xvvv.o//o/. abs/ 2305. 15005.
Kangyu Zheng, Kai Zhang, Jiale Tan, Xuehan Chen, Yingzhou Lu, Zaixi Zhang, Lichao Sun, Marinka Zitnik, Tianfan Fu. and Zhiding Liang. Beyond affinity: A benchmark of ld, 2d, and 3d methods reveals critical trade-offs in structure-based drug design. arXiv preprint arXiv:2601. 14283, 2026. URL https::///// // Parxit org/ aa///bb///000 14283.
16

---

Published as a conference paper at ICLR 2026
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. A comprehensive survey pretrained foundation models: A history from bert to chatgpt, 2023. URL https:://// /arxttxi rxivxivvv org/ abss////// 2302. 09419. 9.
Hongli Zhou et al. Lost in benchmarks? rethinking large language model benchmarking with item response theory. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026. URL https: www.v. /otgg//bs/2555 15055. Accepted for Oral Presentation.
Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, and Lichao Sun. Libero-pro: Towards robust and fair evaluation of vision-language- action models beyond memorization. arXiv preprint arXiv:2510.03827. 2025. URL https:
//arxiv. .rgg aba/ /2500 03827.
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Graph-informed dynamic evaluation of large language models. arXiv preprini arXiv:2309.1716. 17167,
2023.
Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dyval 2: Dynamic evaluation of large language models by meta probing agents, arXiv preprint arXiv:2402. 14865, 2024.
Yubo Zhu, Dongrui Liu, Zecheng Lin, Wei Tong, Sheng Zhong, and Jing Shao. The Ilm already knows: Estimating llm-perceived question difficulty via hidden representations. In Proceedings of
the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1160- 1176. Association for Computational Linguistics, 2025. doi: 10.18653/w120055.mmmlp
URL https::///// / /acl/aclleee anthology.coo 2025. emn ma main in. 61/.
17

---

Published as a conference paper at ICLR 2026
A
APPENDIX
APPENDIX CONTENTS
Related Works
20
Proof of Convergence
21
Time Complexity Analysis
22
Robustness Analysis
23
E.1 Robustness to Single/Group Model Addition
23
Robustness to Dataset-Level Perturbation
25
Answer Distribution Across Difficulty
25
G Analysis Correlation and Difference between EIP and Accuracy-Based Model Rank- ings
25
G.1 Distribution of Model Rank Changes
27
G.2 Local Rank Displacement within Accuracy-Defined Tiers
27
Experiment Details
28
H.1 Dataset & Prompt.
28
H.2 Model Abbreviation.
29
H.3 Evaluation Protocol
29
H.4 IRT Baseline Configurations
30
I Sensitivity Analysis of the Damping Factor
31
Robustness to Weak Model Inflation
K Fevily-Level Blind Spot Analysis
31
L Robustness to Super Model Addition
32
M Extended Computational Efficiency Analysis
32
N Anchor Question Sampling for Rapid Evaluation
33
Asset Licensing and Terms of Use
33
0.0.1 Software Library Licenses
33
O.0.2
Dataset and Benchmark Licenses
34
0.0.3
Language Model Licensing and Terms of Use
34
Verification Test Example
38
18

---

Published as a conference paper at ICLR 2026
Disclosure of LLM Usage
39
19

---

Published as a conference paper at ICLR 2026
B RELATED WORKS
Benchmarking and evaluating LLMs. Owing to the advanced capabilities of LLMs, sophisticated benchmarking is essential to comprehensively assess their performance in both general and specialized domains (Chang et al., 2024). The evaluation of LLMs spans multiple fields, beginning with traditional core natural language processing (NLP) tasks such sentiment analysis (Lopez-Line Lira & Tang, 2023; Zhang et al., 2023), text classification (Yang & Menczer, 2024; Zhang et al., 2024), and natural language inference (McKenna et al., 2023). In recent years, the development of diverse benchmarks has significantly expanded aspects of evaluating LLMs, enabling more comprchensive assessments of their capabilities. For example, the MMLU (Hendrycks et al., 2021a) and MMLU-Pro (Wang et al., 2024) assess models on multi-disciplinanl language understanding and reasoning, and MMLU- Reason further benchmarks multi-tast 2025), while Big- tig-Bench Hard (BBH) (Suzgun et al., 2022) evaluates their multi-step reasoning and algorithmic reasoning abilities. The GSM8K (Cobbe et al., 2021) benchmark tests performance on grade-school-level mathematics, and HvmanEval (Chen et al., 2021) measures model's 's capability in algorithmic reasoning. AlignBench is designed to evaluate the aligmment of Chinese LLMs (Liu et al., 2023a; 2024), and HellaSwag (Zellers et al., 2019) focuses on commonsense reasoning by requiring LLMs to select the most plausible continuation of given context. Guo et al. (2023) and Huang et al. (2024d) assess the abilities of L.LMs in science domains, and recent benchmarks also extend evaluation to medicine with contamination-awall protocols and automated rubrics (Yan et al., 2026) and structure-based drug design with systematic trade-off's across representations (Zheng et al., 2026). With the development of agentic AI, there are many benchmarks designed to evaluate the agent- related capabilities of LLMs (Liu al., 2023b; Chen et al., 2024c;b; Huang et al., 2023a; 2024a), and robust evaluation beyond memorization has also been studied for vision vision-language-ace action models (Zhou et al., 2025). Some benchmarks are also focusing on evaluating the trustworthiness of L.MMs (Huang et al., 2024c; Wang et al., 2023; Gao et al., 2024; Liu et al., 2025; Huang et al., 2023b; 2024b), and benchmarking unlearning is another complementary direction for trustworthy evaluation (Ma et al., 2025). Besides capability evaluation, some recent works propose many evaluation paradigms, including multimodal judge-based evaluation (Chen et al., 2024a) and broader perspectives on AI scientist systems that motivate evaluating complex agentic workflows (Anonymous, 2025). For instance, flexible protocols for dynamic evaluation have been advanced, exemplified by the recent initiatives Val (Zhu et al., 2023) and Vy 2 (Zhu et al., 2024). Wu et al. (2024) and Bao et al. (2024) both propose a dynamic evaluation framework powered by generative models. Beyond dynamic protocols, fluid and continuously updated benchmarks have been proposed better track distribution shift and rapid model progress (Hofmann et al., 2025). Psychometric alternatives based on adaptive testing further aim to reduce evaluation cost while maintaining measurement fidelity (Li et al., 2025), and generative computerized adaptive testing extends this direction by generating or selecting items online (Feng & Lan, 2026). Recent IRT-based analyses revisit benehmark conclusions by explicitly modeling item parameters (Zhou et al., 2026). In preference-based evaluation, rankings can be highly sensitive to small perturbations of comparison sets (Huang et al., 2026), motivating more robust inference from preference data (Frauen et al., 2026) and reliability diagnosis of LLM- as-a-judge using IRT (Choi et al., 2026). Finally, task-specific benchmarks extend evaluation to structured settings such as optimization modeling with graph-theoretic scoring (Wang al., 2026), graph-structured diagram generation (Liang & You, 2025), and open-source game environments (Sistla al., 2025).
mudti- language understanding and reasoning (Tie et al.,
Difficulty measuring. Most previous works rely on human effort to classify question difficulty levels. For example, in benchmarks like MATH (Hendrycks et al., 2021b) and GPQA (Rein al., 2023), human experts annotate questions into predefined difficulty tiers. LingOly (Bean et al., 2024), a linguistic reasoning benchmark, categorizes question difficulty into five levels by considering both semantic similarity to English and the complexity of required reasoning. The labeling process is time-consuming and lacks scalability. Unlike human labeling, OlympicArena (Huang et al., 2024e) employs GPT-4V as an annotator to categorize question difficulty levels. MedConceptsQA (Shoham & Rappoport, 2024), on the other hand, determines difficulty levels based on the relative distances among answer choices within undirected graph built by medical code. However, these approaches either introduce bias in difficulty assessment due to reliance on a single model or are domain-speciife limiting their generalizability to other fields. Recent work also estimates question difficulty from model-internal signals, e.g., hidden representations capturing 4LM--erceived difficulty (Zhu et al., 2025). Complementarily, difficulty can be inferred without ground truth by leveraging comparative
20

---

Published as a conference paper at ICLR 2026
signals across LLM outputs (Ballon et al., 2025). Recent work. Easy2Hard- -eeard--Becchh- (Ding al., 2024), applied two independent methods to quantify question difficulty: Item Response Theory (IRT) (Lord & Novick, 2008) and Glicko-2 (Glickman, 2012). Unlike IRT, which assumes static difficulty parameters, EIP dynamically adjusts scores based on model performance, providing a more adaptive and accurate representation of difficulty. In contrast to Glicko-2, which relies pairwise matchups, EIP employs bidirectional score propagation approach, allowing question difficulty to be inferred from a broader set of solver interactions rather than isolated 'matches', which provides a dataset-wide estimate of difficulty.
C PROOF OF CONVERGENCE
"M 1 measure the change between iterations. The algorithm terminates when A e for predefined tolerance For irreducible chains, convergence occurs at geometric rate O(a"), typically requiring log 1 iterations. An important consideration is the convergence speed in large-scale settings. our experiments with 30 models and 35,550 questions, we find that about 9 iterations of the bipartite propagation process are sufficient to reach stable distribution of difficulty and competency scores. Each iteration primarily involves matrix-vector multiplications on A and A, making the approach computationally feasible even with tens of thousands of questions. The relation between the damping factor and the speed of convergence is shown in Table 8.
-11-1111 Convergence. Let (0) = - "Q ||1 + |
Table 8: A for different damping factors (a) across iterations, indicating the difference between the stationary distribution and convergence.
Iteration
a=0.10
a=0.20
a=0.30
a=0.40
a=0.50
a=0.60
a=0.70
a=0.80
c=0090
a=l
8.39e-02
1.68--01
2.52e-01
3.37e-01
4.22e-01
5.07e-01
5.93e-01
6.80e-01
7.67e:01
8.55e-01
1.86c-03
7.65c-03
L.77-002
3.24c-02
5.20c-02
7.70c-02
1.08c-01
144c-01
1.88c-01
2.38c-01
I 1,20c-06
2,00c-05
1.06c-04
3,48c-04
8.84c-04
1.91e-03
3.66c-03
6.49c-03
1.08c-02
1.70c-02
le-10
4.14c-08
5.00c-07
2.97e-06
1.20e-05
3.76e-05
9,99c-05
2.34c-04
4.99c-04
9.84e-04
4 4.59c-13
L119-10
3.10c-09
3.17e-08
1.94--07
8.54c-07
3.01c-06
9.02c-06
2.38c-05
5.70c-05
Converge
6,28e-13
3.69c-11
6.63e-10
6,26c-09
3,92c-08
1.86c-07
7.16e-07
2.36c-06
6,85e-06
Converge
Converge
3.41e-13
1.10e-11
1.65e-10
1,50e-09
9,78c-09
4.7c-088
2,09c-07
7.58c-07
Converge
Converge
Converge
1,44c-13
3.33e-12
4,42c-11
3.94e-10
2.64c-09
1.41c-08
6.37c-08
Converge
Converge
Converge
Converge
5.38e-14
1.03e-12
1,27e-11
1.12e-10
7.67e-10
4,30e-09
Converge
Converge
Converge
Converge
Converge
Converge
3.27e-13
3.79e-12
3.32e-11
2.32e-10
10
Converge
Converge
Converge
Converge
Converge
Converge
Converge
1.74e-13
1.87e-12
1,58e-11
Theorem C.1 (Existence and Uniqueness of the Stationary Distribution). Let PQ-- E RQQM and Pas-sQ E Tr. Xx be the row-stochastic matrices defined in subsection 2.3. For any damping factor a E (0, 1), the iterative process defined in Equation 2 and Equation converges to a unique stationary distribution RM) satisfying
1
PT PQ-MTT + TM
(1 +(1 a)
(6)
M
TQ a aT QM- TM + (1 - a) 1Q
(7)
Q
Furthermore, aa and R have strictly positive entries.
Proof sketch of Theorem C.I. Step 1: Reformulation Single Markov Chain. The alternating updates between गo and TM in Equation 6 and Equation 7 can be unified into a single Markov chain
on an augmented state space of size Q + M. Define the combined state vector 70)
The
iterative updates become: n (1+1)
TTTTT (() + a)v,
PQ-M 0
1
where the block matrix P PM-+Q represents uniform teleportation.
preserves the bipartite transitions, and v
M
21

---

Published as a conference paper at ICLR 2026
Step 2:
Damped Markov chain interpretation. The combined dynamics correspond to Markov
chain with transition matrix:
T T= (1-aav11+t
where 1 is the all-ones vector. At each step, the chain either follows PT (with probability a') or restarts uniformly via v (with probability 1 - a),, mirroring the damping mechanism in Equation 6 6-Equation 7.
Irreducibility and aperiodicity. 1) Irreducibillit; The teleportation temn ensures every state can reach any other state one step, as (1 ajv > 0 componentwise. 2) Aperiodicity: Self- transitions occur with probability least (1 a) min c >> breaking periodicity inherent in the bipartite structure. Thus, T is irreducible and aperiodic.
Step 3:
Step 4: Perron-Frobenius theorem application. By the Ferron-Frobenius theorem for irreducible
n with > 0, satisfying
and aperiodic Markov chains, T has unique stationary distribution
TM
T Substituting T into this equation recovers the coupled updates in Equation 6-Equation 7, confirming (TQ; TM) as the unique solution. Crucially, convergence iteration depends on the damping factor and the desired precision c, but it is independent of the scale of the problem, i.e., the number of questions Q or models M.
This proof rigorously establishes Theorem C.I using notation consistent with the main text. The use of P, v, and damping factor a directly corresponds the iterative updates in Equation 6-Fquation ensuring notational coherence.
D TIME COMPLEXITY ANALYSIS
In this section, we calculate and prove the time complexity of EIP in theory.
Let be the number of questions and M be the number of models. The Competency Matrix is denoted
by A e f0, 1]@xM, and the Difficulty Matrix (transposed) is Å (1 A)T {0, 1] MxQ. The transition matrix PQ-- E R is derived from A, and PMI-Q E RxMxQ is derived from Á.
Each iteration of EIP primarily involves two sparse matrix-vector multiplications defined in Equation 2 and Equation 3 of the main paper. The operation T involves multiplying the transpose of P (a Q x A sparse matrix) by the M-dimensional vector T the computational cost of this product is proportional to the number of non-zero elements in PM-QQ plus the dimension of the output vector, i.e., (nnzz(MM-Q) + Q). Similarly, the cost of the transposed product is ((nnz(P + M).
The sum non-zero elements across both original transition matrices, nnz(PQ-MM) + anz(PM--Q)) represents the total number of correct and incorrect answers, which sum to Q.. Specifically, unz(Pq--u) is the total number entries where Ai = 1, and ( is the total number of entries where A₁ 1 (i.c., A.j 0).
Additional per-iteration operations, such as vector additions and scalar multiplications for incorporat- ing the damping factor and the uniform teleportation term, contribute O + M).
Combining these, the total complexity for one iteration OQQM + +Q M) ((QM)
In typical scenarios the QM term dominates Q + Q... Therefore, the per-iteration complexity is effectively O(QM). Thus, each iteration runs in O(QM) time. Combining the number of iterations T and the per-iteration complexity, the overall time complexity of EIP is:
+ (QM = O(log(1/e)) (QM + Q + M)) = O(tQM)
(8)
Where is the predefined tolerance value, and / is the convergence iteration count.
22

---

Published as a conference paper at ICLR 2026
E ROBUSTNESS ANALYSIS
E. ROBUSTNESS TO SINGLE/GROUP MODEL ADDITION
To assess EIP's ranking stability the model pool composition evolves, simulating the integration of new models, we analyze its performance under perturbations. We consider scenarios analogous incorporating single new model a small group of new models into existing evaluation set. This is achieved by examining the Spearman correlation (p) of question difficulty and model competency scores derived from reduced model pools (N-1 models via Leave-One OnreeOOn--ut, and N-5 models via random subset removal) against the scores from the original full pool of N models. High correlations indicate that EIP's assessments remain consistent. The statisties for the N-5 scenario reflect averages over 10 trials. Results are summarized in Table 9.
Table Comparison of EIP robustness: Leave-One Out (LOO) vs. Random Removal of 5 Models. LOO was conducted for each model in the pool (30 trials). Random removal was conducted for 10 trials, each removing 5 models. Values are averaged Spearman correlation (p) over trials.
LOO (Remove 1)
Random Removal (Remove 5)
Question Difficulty Correlation
Mean Spearman Max Observed Drop
0.9978
0.0024
0.9863 + 0.0076
0.0100
0.0271
Model Competency Correlation
Mean Spearman
0.9998
0.9987
Question Set Reduction
Mean Questions Removed
32.9
217.9
Max Questions Removed
103
307
Mean % Reduction
0.09%
0.62%
The analysis presented in Table 9 underscores EIP's robust stability when its model pool composition evolves, akin to integrating new models.
Model Competency rank exhibited greater stability: The mean Spearman correlation ceptionally high, starting at 0.9997 (SD=0.0002) with one model removed and remaining at 0.9934 (SD=0.0069) even with 15 models removed. This highlights EIP's capability maintain consistent relative model rankings despite considerable variations in the composition and size of the model evaluation pool. A more intuitive visualization is shown in Figure 10.
EIP maintains high consistency in question difficulty rankings despite significant model pool variations. When simulating the addition of single model (N-1 pool correlated with the N- model pool), the Spearman correlation (p) for question difficulty remains exceptionally high at 0.9978 + 0.0024. Even with more substantial change, such as integrating a group of 5 models (N-5 pool correlated with N-model pool, 17% pool size change), the correlation for question difficulty remains very strong at 0.9863 0.0076. This indicates that the relative difficulty assessment of questions is largely preserved.
Model pool perturbations cause minimal changes to the effective question EIP filters out questions universally solved or failed by the active model pool, and Table 9 shows that adding new models (simulated via I.O0 or random removal) has limited effect on this filtering. Even when integrating five models, the average change a maximum of 307. This negligible impact ensures that nearly all questions continue contributing to the evaluation, and the exclusion of this small fraction does not disrupt the overall difficulty rankings, which remain highly stable.
only 217.9 questions (about 0.62% of the total), with
When viewed from the perspective of an evolving model pool, the system demonstrates strong resilience, ensuring its evaluations are dependable as new models are incorporated into the assessment framework, with model competency rankings being particularly robust.
23

---

Published as a conference paper at ICLR 2026
1.00
Spearman Correlation
0.98
0.96
0.94
Question Difficulty Correlation
0.92
2
4
10
12
14
1.00
1.00
Spearman Correlation
1.00
0.99
0.99
0.99
Model Performance Correlation
0.99
2
4
6
8
10
12
14
Number of Models Removed
Figure 10: Impact of Random Model Subset Removal EIP Stability. Top panel shows p for question difficulty, and the bottom panel shows for model competency rank, both comparing results from the reduced model pool the full model pool models removed. Error bars represent the standard deviation over multiple random removal trials for each k.
24

---

Published as a conference paper at ICLR 2026
E.2 ROBUSTNESS TO DATASET-LEVER PERTURBATION
To assess the influence of adding individual benchmark datasets on the final model rankings, we performed saave-onedattsettou analysis. The results are shown Table 10.
Table 10: Robustness of EIP model competency rankings to dataset removal. Spearman p correlation calculated between rankings obtained after removing a dataset and the original rankings based on all datasets.
Dataset Removed
# Questions
Model Rank Correlation (p)
HellaSwag
10042
0.9604
MMLU-Pro
130
0.9626
MATH
5000
0.9884
GPQA
646
0.9973
GSM8k
1319
0.9973
BBH
6511
0.9973
Average Correlation
0.9839
Standard Deviation
0.0162
EIP model competency rankings exhibit high overall stability across all different dataset combinations. The high average Spearman correlation between model rankings derived from N-1 datasets and the full N-dataset ranking(00.933) (0.9839 = 0.0162) indicates that the relative model hierarchy established by EIP largely consistent, even when any single dataset is newly introduced to or excluded from the evaluation pool. When simulating the addition of substantial datasets like MMLU- Pro (12, 032 questions) or HellaSwag (10, 042 questions) to the remaining pool, the model rank correlations were observed to be more perturbed compared to adding smaller datasets. However, the correlations in these scenarios (0.9604/0.9620) still indicate strong preservation of the relative model ranking.
These findings suggest that EIP provides model evaluations that are robust to variations in the specific suite of benchmark datasets used, irrespective of whether large, broad dataset or smaller, more specialized one is being integrated. This consistency points to its utility in generating more generalized assessments of model capabilities, less dependent on the idiosyncrasies of individual datasets.
F ANSWER DISTRIBUTION ACROSS DIFFICULTY
As shown in Figure 11 (Highlighted in square box), similar pattern emerged aligned with the case study in section 4: although GPT-4o answered slightly more questions correctly overall, Gemini-1.5. Pro outperformed on difficult questions, achieving a 0.6% higher accuracy rate in this category. Consequently, despite answering 2.9% fewer mediun-difficulty questions and achieving lower overall accuracy rate, Gemini-115-Prr scored higher than GPT-4o under EIP, further emphasizing the impact of performance on high-diffiulty questions. This real-world example corroborates the findings of our simulation, demonstrating EIP's ability to diffcrentiate models based on their performance on challenging tasks.
G ANALYSIS ON CORRELATION AND DIFFERENCE BETWEEN EIP AND ACCURACY-BASER MODEL RANKINGS
While EIP's competency scores exhibit strong overall correlation with traditional accuracy- based rankings (Kendall's Tau T 0.876, p < 0.001, as detailed in Table 11), a closer examination reveals significant and insightful differences in how individual models and groups of models are ordered. This section delves into these inter-mode ranking variations, underscoring EIP's ability to provide a more nuanced perspective model capabilities than accuracy alone. As summarized in Table 11, the mean absolute rank change between the two methods is 1.60, with median change of 1.0 rank position. Crucially, the maximum observed rank change for model 4 positions, indicating that for
25

---

Published as a conference paper at ICLR 2026
instuct Accuracy30.7
Owen22 natruct Aocuracys33. Remell Scare-M,
Qwen2.5 Accur
Accuracy Scoeest0.)
Accuracy:35. oc:35.4
Accurscy 24,4
Accuracy264
RankL Scone
Renitl Srore:15.3
R Srome-17.3
Bank:
81.0%
$8.7%
82.25
Owen2 S-Math- S.iisstract Accurecy:22 M
Qwen2-4, 5/8 Accarary201 RonKLLU
Owend 5-a.8i--redmuc Accuncyy RanKLLM Scorc:28.2
Owenz 3-3BBitddrcctt Accureer RanKLLM
Llama-3 Arcanery:112 RanKLUM Score
Usme-d 2-30-imstaat
Acconecy yo2
RankLUM
14.4%
5.5 18.7%
37.5%
13.00
10.0%
75.4
75 MP%
71.80
Lama 3.2.8 instrutt BankLL Scone:154 15.4
Gemini 1.5 Pro
Gemtni fatthhh
Phi medi 128k ReneLLM Scotesfa.3
Phila 328k RankLLM Sconed11
Pii mink
BericLLM Score-92.8
BankLLL Scoee:75.4
Rank Scomeffa.
5.0%
2.0% 20.0%
311% 17.85
201
24.8%
$2.0%
71.85
E1.5%
76.1%
$2.4%
Accurscy50
Apcurscy44-0
Acourccy 38
Accuracy 64.4
Acc Arcurrsy:35.
Accurary:33
Score:45.2
2120.2%
1
ELEE%
8..3%
83.0%
Mistral-7B-nnstuu In- Accuracy:34. HuniLLM Score519
Deepseck-V3 Accuracysia. RankLLM Score:100.
DeegSerk-Code Cadeeek--Cde///-uu Ulte Instrct Deep Llt Chat
6P7-40 Accaracy:00 RankLLM Scoredds.3
GPT-lomil Accuracy:74.
Accarecy/S..5 HankLLM Scores43.66
scc.r KankiLi Score:st.0
Nank Sconubb..
13.7%
6.3%
22%
10%
10.4%
65.9%
76.7%
Easy
Hard
Figure 11: Distribution of correctly answered questions by models across the whole dataset (leave- one-out).
26

---

Published as a conference paper at ICLR 2026
Table 11: Summary of model ranking comparison metrics between EIP and accuracy.
Metric
Value
Mean Absolute Rank Change
1.60
Median Rank Change
1.00
Max Rank Change
4.00
Kendall's Tau Correlation
0.876(p=0.001)
Bank-Biased Overlap (RBO)
0.896
ICCI (Absolute Agreement)
0.974
some models, the evaluation outcome under EIP can be substantially different from an accuracy-only assessment. The high Intraclass Correlation Coefficient (ICC1 0.974) suggests strong overall agreement in the rankings, yet the rank changes highlight the specific instances where EIP provides distinct evaluation.
G.1 DISTRIBUTION OF MODEL RANK CHANGES
The distribution of absolute rank changes (see Fig- ure 12) reveals that while majority of models expe- rience small shifts (e.g., 6 models, or 20%, have no rank change), a notable portion shift by or more po- sitions. These larger shifts particularly interesting as they point to models whose performance on ques- tions of varying difficulty disproportionately affects their EIP score compared to their simple accuracy.
Numbor of Modoo Mode
G.2 LOCAL RANK DISPLACEMENT WITHIN ACCURACY-DEFNNG DEFINED TIERS
Absolut Rank Diference
Figure 12: Absolute model rank difference distribution.
To further investigate where these ranking differences are most pronounced, we analyzed the local rank displacement of models within windows defined by their accuracy ranks. Figure 13a, Figure 13b, Figure 13c, and Figure 13d illustrate the mean and maximum rank displacement for models when grouped into tiers by their accuracy ranking, using window sizes of 1, 3.3,5 5, and 10 respectively.
The trends observed in Figure 13 indicate varying degrees of rank displacement across accuracy- defined tiers:
Relatively High Stability in Top- fop-Tier Model Rankings. For the top 10 models ranked by accuracy (window "1-10" in Figure 13d), the mean displacement in EIP rank is comparatively low at 1.0, with maximum displacement of 2.0. This suggests that among the highest-performing models by accuracy, EIP's rankings largely concur, although some re-ordering still occurs. The Kendall's Tau correlation within this specific window very high (0.867, p < 0.001), indicating strong local rank agreement.
Increased Rank Volatility in Mid-Tier Models. The subsequent group of models, those ranked 11- 20 by accuracy (window "1 1-20" in Figure 13d), shows increased rank displacement when evaluated by EIP. The mean displacement rises to 1.4, and the maximum displacement observed within this tier reaches 3.0. The local Kendall's Tau correlation (0. T0.73,, p R 0.002) remains statistically significant but lower than that of the top tier, reflecting more substantial re-shuffling by EIP.
Pronounced Rank Displacement in Lower- Tier Models. Models in the lower third of the accuracy ranking (window "21-30" in Figure 13d) exhibit the highest mean displacement (2.4) and the overall maximum observed displacement of 4.0 rank positions. The local Kendall's Tau correlation drops further to 0.467 (p zs 0.043), indicating considerably weaker local rank agreement between accuracy and EIP in this tier. This suggests that for models with lower overall accuracy, EIP's difficulty- weighting mechanism has more pronounced effect, leading to more substantial re-rankings based on performance on hard questions versus reliance easier ones.
27

---

Published as a conference paper at ICLR 2026
Mon
Dipkorrey
12.2
2027
Rank Window
Ram Window
aa) Window Size
(b) Window Size
Vear Digincanen
Mas
11.50
(c) Window Size =
(d) Window Size = 10
Figure 13: Mean and Maximum Rank Displacement of EIP scores compared Accuracy scores within different Accuracy-Raa Rank based windows. Windows are defined by accuracy rank (e.g., "1-10" refers to models ranked 1st 10th by accuracy).
Individual Model Rank Fluctuations Illustrate Local Reordering. When examining displacements at the individual model level (window size 1, Figure 13a), fluctuations are evident across the spectrum. For instance, the model ranked 2nd by accuracy is displaced by 1 position in EIP, the model ranked 9th by accuracy is displaced by 2 positions, and models ranked 21 26th, and 28th by accuracy are all displaced by 4 positions in their respective EIP rankings. This highlights that even when broader trends might align, EIP provides distinct local orderings based on its difficulty-awan assessment.
These windowed analyses demonstrate that while EIP broadly agrees with accuracy at the very top of the performance spectrum, its differentiation based on question difficulty becomes more apparent and impactful in the middle and lower tiers of accuracy rankings. This is where the ability to correctly answer challenging questions, or the failure to do so, most significantly revises model's perceived competency compared to a simple count of correct answers. The maximum displacement of rank positions underscores that EIP can lead to meaningfully different conclusions about relative model strengths, particularly for models with nuanced performance profiles across varying question difficulties.
H EXPERIMENT DETAILS
To ensure optimal performance and compatibility across our experiments, we employed a combination of advanced software libraries and frameworks, including Together AI 1.2.1, OpenAl 1.30.3, vLLM 0.5.5, FlashInfer 0. 1.1.5tcclll 124, Flash-Attm 2.6.3, Torch 2.4.0, Google Generative AI 0.7.2, torch 2.5.1 and CUDA 12.4.
H.1 DATASET PROMPT.
We adhered to the standard configurations outlined in the respective original benchmarks used in our experiments. For the benchmarks BBH, MMLU-Pro, GSM8k, GPQA, MATH, and HellaSwag, we utilized 5-shot prompting combined with Chain of Thought (CoT) reasoning. These strategies were chosen to align with the original settings for each respective benchmark. Temperature was set to 0 to ensure reproducibility.
28

---

Published as a conference paper at ICLR 2026
H.2 MODEL ABBREVIATION.
For simplicity, we use some model abbreviations in figures and tables. Specifically, DeepSeek-
Coder-Lite is DeepSeek-Coder- Bonder-V--Lite-nnsttue Qwen2.5-3B is Qwen2.5-3BB-nstrric Qwen2.5- is
1.5B is 1.5--n5t..BB v0.3 is Mistral-7B BB-nnttructt-O..3 DeepSeek DeepSeck-Chat-Lie is DeepSeek--2
Mistral-7B-
W22-Lite-Chak, Qwen2.5-0.5B
is Qwen2.5-0.50 I--ttructttt Llama- 1amma-..2--2 3.2-3B is 3eaia--.2--B--msee Llama-3.2-11 is Alamm-332--IB
Instruct.
Table 12: Information on the selected models used in EIP.
Name ChazGPI-to (OpenA] ct al., 2024a1
Size
Developer
Version
Context Length
Open Weight
N/A
OpenAl
Aug 2021
128K
ChatGPT ChatGPT-fo-ning (OpankI et al., 2024e)
N/A
OpenA1
July 2024
128K
DeepSSeez (3jeeeppeee et
2024a)
DeepSeak
Do: 2024
64X
DeepBaek-
2ode=-v2-1iie Instrce (Desg3eek AAI et al., 2024b)
15.78
DeepSeek
June 2024
32K
neepSeek 19-1--------tteee (heepSeek-AT
al... 202461
15.78
DespSeck
May 2014
13K
Gamini-1.3-Pre (Tean et 31., 20243)
May 2024
Saadii-1.5-Faaph (Tean et a2., 2024a]
N/A
Google
May 2024
1M
Gomma-2-27b-15 (Toan al., 20242)
27:28
Google
June 2024
8K
Gemma-2- 2-50--2-5---- [Tean et al., 2024ha]
Google
June 2024
Germe- 2-25-1t [Tean et al., 202461
228
Google
June 2014
8K
(Gratiafiori
al., 2024)
Meta
July 20/44
12NK
6 clamm-3.222 3.- rust (Srattafiori et al., 2024)
321B
Meta
Sep. 2074
128K
1era--,2-1--nsttie (Grattzfiori ct al., 2024)
1224
Meta
Sep. 2024
128K
Mistral-7B-
(Jiang
2023)
7:288
Mistral
May 2024
32K
(Abdin et ai., 2024)
148
Micresofi
May 2024
128K
Phi Phi-3-amall amalll 128k instruct (Abdin al., 2024]
7,39
Microsoft
May 2024
128K
Dhi-3.8-mini-1288
(Abdin et
2024)
Microoof
2074
128K
Gren2-1B-instxuct (Yong et si., 2024)
762B
Qwen
June 2024
131K
wwen--kata-7--Instroe
20241
763B
Qwen
June 2024
131K
(vaa2-1.53-Iastuue (Vang et al., 20261
Qwen
2024
131K
Oven2 1.58 Instese- (Yang et al., 2024)
1:588
Qwen
June 2024
131K
(wan2-3.53-Iaam)
20261
0
June 2024
32K
Qven2.5 73 cnst (Yang el al., 2024]
Qwen
Sep. 2024
131K
0wen2.5- 5-Matt In--trru- (Tang et al., 20241
7668
Qwen
Sep. 2014
131K
Qwen2. 5-..tt-ttrttt (Yang (tang et al.. 2024)
2024)
1,5BB
Sep. 2024
131K
Owen2 1--tatt-1.$---nneeees
1,548
Qwen
Sep. 2024
B3IK
Qwan2. 5-13-Iastruct [Yarq et al., 20261
3,99
Qwen
Sep. 2024
131K
Yi 1.5
2025]
34.48
May 2014
16K
11-1.3-98-Chat (AI at al., 2023)
3338
01-A1
May 2024
16K
Yi 1.5 6B Chat [AT al., 2025)
6068
01-A1
May 2024
16K
H.3 EVALUATION PROTOCOL.
To ensure robust human ovaluations, we implemented two key design principles: Discipline selection: Varied domains (c.g., mathematics, computer science, commonsense reasoning) with matched pair disciplines. 2) Blind judgments: Evaluators were unaware of model or peer judgments.
Each of the 20 evaluators assessed 70 randomly ordered question pairs via verification interface (Appendix P) to mitigate order bias. We computed two primary alignment metrics: 1) Individual Alignment: For each evaluator, the percentage of their non-skipped judgments (Vi-V20) aligning with method predictions. 2) Consensus Alignment: The proportion of questions where a method's prediction matches the majority of human judgment. For the set of non-skipped questions Q, this is defined as:
Consensus
1
Σ gEQ
pred, arg max count, JEV,
[Q|
Evaluator Information. Our human evaluation involved total of 20 participants. This pool included the two authors and 18 current students not holding terminal degrees, ranging from associate-leve to PhD candidates. To ensure diverse range of perspectives, participants were recruited from various academic backgrounds, and gender balance was also considered. All evaluators possessed strong English proficiency, enabling effective task completion. Table 13 summarizes the academic fields and educational levels of the 18 student evaluators.
Evaluators with backgrounds in computer science, mathematics, and engineering had all completed formal coursework in calculus, probability, and statistics, providing them with solid quantitative foundations. In contrast, participants from the arts, social sciences, and humanities may not have
29

---

Published as a conference paper at ICLR 2026
Table 13: Diversity of Human Evaluator Pool (N=18, excluding authors).
Academic Field
Undergraduate Graduate (Master's)
PhD Candidate
Associate Degree
Total by Field
Computer Science
6
Other Engineering
Arts
0
2
Business
1
0
2
Humanities
0
1
4
Social Sciences
0
Total by Level
4
2
20 (incl. 2 authors)
received such training. To accommodate differences in subject knowledge, participants were allowed to voluntarily skip domain-spectie questions if they felt unqualified to answer them. This mechanism helped minimize the influence unfamiliar content on the evaluation process.
Domain Domain-specific specific questions requiring technical expertise were handled by participants with the ap- propriate background, while common sense reasoning tasks were completed by all evaluators. This design ensured that human evaluation captured both specialized capabilities and general reasoning performance across a diverse participant pool.
H.4 IRT BASELINE CONFIGURATIONS
To provide a robust comparison, we implemented and configured three Item Response Theory (IRT) baseline models. These configurations, detailed below, were chosen to give the IRT models ample opportunity to converge and perform optimally, even on the relatively small 100-question dataset used in controlled simulation (section 4).
IPL IRT (Rasch Model) and 2PL IRT: Both the 1-Parameter Logistic (1PL) IRT model, also known the Rasch model (where item discrimination is fixed at 1), and the 2-Parameter Logistic (2PL) IRT model were fitted by maximizing the log-liked likelihoood of the observed response matrix. Optimization was performed using the L-BFGS-B algorithm, quasi-Newton method suitable for bounded parameter estimation. A maximum of 1000 iterations was permitted for the optimization process. To enhance parameter stability and prevent extreme values, L2 regularization with coefficient of 0.01 was applied to both model abilities (0) and item difficulties (8). For the 2PL model, L2 regularization was additionally applied to the item discrimination parameters (ar) centered around 1 (i.e., penalizing (a - )),,, encouraging discriminations to be in a standard range. Parameter bounds were enforced during optimization: abilities and difficulties were constrained to the interval [-5, 5), and discrimination parameters for the 2PL IRT were constrained to [0.1, 5] to ensure positivity and interpretability.
Multidimensional IRT (Multi-IRT): We implemented Multidimensional IRT (MIRT) model configured with D = latent dimensions. This choice was made to explore a more complex latent trait structure, though
acknowledge that estimating parameters for three dimensions from 100-question dataset can be challenging. The MIRT model was trained for 1000 epochs using the Adam optimizer with learning rate of 0.01. Parameter estimation employed variational inference (VI), where the true posterior distributions of abilities, difficulties, and discriminations were approxi- mated. Standard Normal distributions, .N(0, 1), served priors for all model parameters (abilities, difficulties, and per-dimension discrimination values). For the variational posterior approximation, we utilized a mean-field approach where the posterior for each parameter was modeled as a Normal distribution with a learnable mean and a fixed standard deviation of 1. The model was trained by maximizing the Evidence Lower Bound (ELBO). The reported ability scores for Multi-IRT are the L2 norm of the mor-m multidimensional ability vectors, subsequently min-max scaled to 0-100 range, similar to other IRT models and EIP scores for comparability.
30

---

Published as a conference paper at ICLR 2026
I SENSITIVITY ANALYSIS OF THE DAMPING FACTOR
The damping factor a in EIP (cf. Equation 2-Equation 3) controls the strength of teleportation regularization and influences the smoothness of the stationary distributions over questions and models. To verify that our conclusions are not sensitive to this choice, we recomputed EIP on the full MetaBench response matrix (30 models x 35,550 questions) across broad grid of a values and compared the induced model- and question-rankings against reference run with a = 1.00 using Spearman correlation. Table 14 shows that both model and question rankings remain highly stable across this range.
Table 14: Sensitivity of EIP rankings to the damping factor a. We report Spearman correlations of model and question rankings relative to a 1.00 on the full MetaBench matrix.
Pmodd
Pquestion
0,50
0,9911
0,9976
0.60
0.9942
0.9985
0.70
0.9978
0,9992
0.85
0.9987
0.9998
0.95
1.0000
1.0000
0.99
1,0000
1,0000
Even under a conservative setting (e.g., a = 0.50), model rankings remain highly stable (Proodel = 0.9911) and question rankings are even more stable (Pqucstion 0.9976). For a 2 0.95, the induced rankings are effectively identical (p 1.0000 for both models and questions). In all experiments in the main paper, we fix = 0.85, which balances fast geometric convergence with a sufficiently expressive stationary distribution (see Table 8).
J ROBUSTNESS TO WEAK MODEL INFLATION
natural concern that expanding the evaluation pool with many weak models could inflate question difficulty scores and perturb rankings. EIP mitigates this effect by attributing failing model's marginal contribution to question difficulty proportionally to nm/ Fm/F((m. where Tm is the model competency score and F(m) is its total number of failures; weak models tend to have both small Tm and large F(m), limiting their per-question influence. We stress-test this scenario by duplicating the 10 weakest models (two identical copies each), expanding the pool from 30 50 models, and recomputing EIP on the full response matrix.
Table 15: Weak-moder inflation stress test (30 - 50 models by adding 20 weak copies). We compare question difficulties (Pearson) and the ranking of the original 30 models (copies excluded).
Statistic
Value
Pearson r (question difficulties; 30 vs. 50 models)
0.9972
Spcarman P (model ranks; original 30 models)
0.9969
Unchanged in Top-20 models (by rank)
20/20
Max absolute rank change among Bottom-10 models
2
Table 15 confirms that both the question-difficuly landscape and the induced model ranking are highly stable under this extreme weak-mode model inflation: the Top-20 models remain unchanged, and the Bottom- 10 models shift by at most two positions. To further illustrate why weak models do not dominate difficulty estimates, Table 16 provides a concrete per-question contribution breakdown. For wrong-side
question 14703, 15 models answer incorrectly; GPT-4o alone accounts for 32.69% contribution mass, whereas the 7 weakest failing models combined contribute only 18.60%.
K FAMILY-LEVEL BLIND SPOT ANALYSIS
Another concern is whether "family-level blind spots" could systematically bias difficulty estimates, e.g., if single architecture family disproportionately fails set of questions and thereby dominates
31

---

Published as a conference paper at ICLR 2026
Table 16: Wrong-side contribution breakdown for question 14703. For each failing model m. we report its EIP competency score manx-nnomallzee to 100), its total failure count F(m) over the 35,550-question matrix, and its percentage share of the question difficulty, proportional to km/ (m/F(()) among failing models.
Model
Model score
wrong F(m)
% of difficulty
GPT-44
88,3262
7308
32.69%
Yi-1.5-34B
53.2638
14518
9.92%
Swen2.5-7B-Insttree
50.2553
14699
9.25%
Phi-3-medium-1285
48.3485
17207
7.60%
Qwen2-7B
44.4336
16800
7.15%
Qwen2-Math-7B
40.1448
20416
5.32%
Yi-1.5-9B
39,4718
20176
5.29%
Qwen2.5-1..B-Insee.
35.2976
22845
4.18%
Mistral-7B
Distral-7B-Instruct--dd
31.8986
23617
3.65%
Qwen2.5-Math-1.5B
33.5447
27613
3.29%
Qwen2-0.55 0..5
31.1981
28460
2.96%
Qwen2.5-0.33 2........-IIsiii
28.2489
28859
2.65%
gemma-22 2-2b
24.7172
27192
2.46%
3.2-3Btlnttruct
23.8847
28205
2.29%
Llama 3.2---3.-111 1-2-1B-Imne BB-Instruct
15.3590
31967
1.30%
their difficulty mass. We quantify this effect by aggregating each question's wrong-side contributions by model family (Qwen, Llama, Yi, etc.) and marking a question family-dominated when single family accounts for more than 90% of the wrong-side contribution mass and has at least two failing members that question. Table 17 shows that such extreme cases are in our diverse 30-model pool: only 246 out of 35,550 questions (0.69%) are family-dominated. Moreover, none of these dominated questions appear among the globally hardest items (none are the Top-1.00 1,001,000 hardest questions), and their difficulty scores fall in the moderate-to-easy range (median 1 11.7; maximum 23.07 on the EIP question scale).
Table 17: Family-dominant questions under 90% dominance threshold (and requiring at least two failing models within the dominating family).
Family
dominated questions
Share of all questions
Qwen
0.65%
Llama
13
0.04%
Phi
0.00%
Total
246
0.69 %
L ROBUSTNESS TO SUPER MODEL ADDITION
To further probe robustness to pool perturbations, we simulate adding a hypothetical "super model" that answers large fraction of questions correctly (99%. 95%, 90%, 80% accuracy) and recompute EIP on the expanded pool. Table 18 reports that, after excluding the added synthetic model, the Spearman correlation between the original 30-model ranking and the post-addition ranking remains extremely high (pmodel E [0.9956, 0.9973)), with maximum absolute rank change of 2 across all settings.
M EXTENDED COMPUTATIONAL. EFFICIENCY ANALYSIS
We provide an extended ceppuuattionaleffiiiennnn comparison for EIP and IRT baselines on the full MetaBench scale (30 models x 35.550 questions). Table 19 reports wall-clock runtimes consumer CPUs and high-perfone performaaee GPUs. EIP's update dominated by sparse matrix-vector multiplications and converges in 9 iterations (see Appendix C), resulting in millisecond-level total
32

---

Published as a conference paper at ICLR 2026
Table 18: Robustness adding hypothetical super model. We append synthetic model and recompute EIP; correlations and rank changes are computed on the original 30 models (excluding the added model).
Super model accuracy
Prodel
Mean |rank]
Max Arank
99%
0.9964
0,40
2
95%
0,9956
0.47
2
90%
0,9973
0.33
80%
0.9973
0.33
2
Table 19: Computational efficiency comparison on the full response matrix (30 models x 35,550 questions). EIP totals correspond to iterations until convergence; IRT runtimes report end-to-end fitting times for IPL and 2PL baselines.
Hardware
EIP (1 iter.)
EIP (total)
IPL-IRT
2PL-IRT
Intel 17 CPU
0.00597s
0.05373s
1782.75s
3787.03s
MPS (Mac MI)
0.012296s
0.110804s
A40 GPU (CUDA)
0,000581s
0.005263s
24.09s
459s
B200 GPU (CUDA)
0,000510s
0.004634s
5.338
0.59s
runtime even on CPU; GPU acceleration further reduces the runtime. In contrast, IRT requires iterative likelihood maximization and orders of magnitude slower this scale.
N ANCHOR QUESTION SAMPLING FOR RAPID EVALUATION
Finally, study whether EIP rankings be recovered from small subset of questions, which relevant to adaptive or budgeted evaluation settings. We perform difficult--stratfiee sampling (deciles by EIP difficulty) and select only 0.5%, 1%, 5% of questions anchors, recompute EIP on the anchor-only matrix, and compare the resulting model ranking against the full-question ranking. Table 20 shows that with 0.5% anchors (180 questions), the induced model ranking remains strongly correlated with the full ranking (Pmodel = 0.8901), improving further as the anchor budget increases.
Table Difficulty- anchor sampling for rapid evaluation. We recompute EIP using only the sampled anchors and compare the resulting model ranking against the full-question ranking.
Anchor rate
anchors
Avg Arank
0.5%
180
0.8901
3.07
1 %
360
0.9199
2.47
5%
1780
0.9729
1.40
O ASSET LICENSING AND TERMS OF USE
This appendix details the licensing information for all software libraries, datasets, benchmarks, and language models utilized in the experiments presented in this paper. Ensuring compliance with the respective terms use is critical for reproducible and ethical research.
O.0.1 SOFTWARE LIBRARY LICENSES
The software libraries employed in this research governed by variety of open-source licenses, predominantly permissive ones, which facilitate their use in academic and research settings. Table 21 provides summary.
Notes on Software Library Licenses: Together AI Python SDK 1.2.1 is licensed under Apache 2.0, confirmed by its official repository. Community SDKs may vary. OpenAl Python Library
33

---

Published as a conference paper at ICLR 2026
Table 21: Software Library Licenses
Library
Version
License
Together AI Python SDK
1.2.1
Apache License 2.0
OpenAl Python Library
1,30.3
Apache License 2.0
vLLM
0.5.5
Apache License 2.0
FlashInfer
0.1.5+cu124
Apache License 2.0
Flash-Attn
2.6.3
BSD 3 C-Cluuse
Torch (PyTorch)
2.4.0
BSD 3 Clause
Google Generative Al SDK
0.7.2
Apache License 2.0
torch (PyTorch)
2.5.1
BSD 3-Clause
NVIDIA CUDA Toolkit
12.4
NVIDIA CUDA Toolkit EULA
1.30.3 is licensed under Apache 2.0 according its official repository, though older PyPI versions might indicate MIT: the current repository is considered authoritative. vLLM 0.5.5 is licensed under Apache 2.0, as per its PyPI page and official repository. FlashInfer 0.1.5+cu124 licensed under
Apache 2.0, with the "+cu 124" denoting CUDA 12.4 compatibility. Flash-Attn 2.6.3 is licensed under BSD 3-Clause, according to its official repository. Py PoTor (Torch 2.4.0 2.5.1) is licensed under BSD 3- 3-Clause, confirmed by its official repository and PyPI for version 2.5.1. Google Generative AI SDK 0.7.2 is licensed under Apache 2.0, as per its official repository. Newer developments may be in "google-genai' under the same license. NVIDIA CUDA Toolkit 12.4 is governed by the NVIDIA CUDA Toolkit EULA, a proprictary license permitting development and specific redistribution.
O.0.2 DATASET AND BENCHMARK LICENSES
The datasets and benchmarks employed are governed by various open-source licenses or public domain dedications. Table 22 summarizes this information.
Table 22: Dataset and Benchmark Licenses
Dataset/Benchmark
Stated License(s)
BBH (Big-Bench Hard) (Suzgun
Apache 2.0 (for BIG-Bench (Srivastava et al.. 2023)) MIT (for specific BBH repository by Suzgun al.)
et al., 2022)
MMLU-Pro (Wang et al., 2024)
Apache License 2.0
GSM8k (Cobbe et al., 2021)
MIT License (likely, from original OpenAl repository)
GPQA (Rein et al., 2023)
MIT License
MATH (Hendrycks et al., 2021b)
MIT License
HellaSwag (Zellers et al., 2019)
MIT License (original author's 's repository)
Notes on Dataset and Benchmark Licenses: BBH is derived from BIG-Bench (Srivastava et al., 2023), which is under Apache 2.0. The specific repository for BIG-Bench Hard by Suzgun et al. (2022) uses an MIT license. The source utilized determines the applicable license. MML.U-Pro clearly licensed under Apache 2.0 by TIGER-AI-LA (Wang et al., 2024). GSM8k, originally released by OpenAl (Cobbe et al., 2021), likely under an MIT License. It's important note that variants like GSM8k-Platinum may CC-BY-4.0; the specific source license is key. GPQA by Rein et al. (2023) is under an MIT License. The SuperGPQA variant uses ODC-BY. MATH dataset by Hendrycks al. (2021b) is clearly MIT licensed. HellaSwag, from the original authors' repository (Zellers et al., 2019), is under an MIT License. Other platforms hosting HellaS (e.g.. Kaggle: CCO; Hugging Face/jon-tow: CC BY NC 4.0) may have different licenses; the original MIT license was considered authoritative for the version used.
0.0.3 LANGUAGE MODEL LICENSING AND TERMS OF USE
Language models are subject to API service agreements or specific seight licenses. Table 23 provides overview.
34

---

Published as a conference paper at ICLR 2026
Table 23: Language Model Licensing and Terms Overview
Model Family/Pruvider
Primary Governing Terms
OpenAl (ChatGPT series, GPT-4o (OpenAl et al., 2024a), GPT-4o-mini (OpenAl et al., 2024b))
OpenAl Usage Policies & Services Agreement
Google Gemini (API: Gemini- 1.5-nii-1,.-Poo, Jemiini-1.5-Flashe (Team et al., 2024a)) Google Gemma (Open Weights: Gemma-2 series (Team et al., 2024b))
Google APIs ToS & Gemini API Additional ToS
Gemma Terms of Use
Meta Llama (Llama 3.1. Llama 3.2 series (Grattafiori et al., 2024))
Llama Community License Agreement & Acceptable Use Policy DeepSeek Model License / MIT License (varies by model)
DeepSeek AI (DeepSeck-V3 V3 (DeepSeek-Al et al., 2024a), Coder-V2-Lieg V2-Lite-Chat (DeepSeek-Al al., 2024h))
Mistral Al (Misstal-7B-IIsstrcctv.... (Jiang
Apache License 2.0
al., 2023))
Alibaba Qwen (Qwen2, Qwen2.5 series (Yang et al., 2024))
Apache License 2.0 (most) / Qwen RESEARCH LI- CENSE
01.A1 Yi (Yi-1.5 series (Al al., 2025))
Apache License 2.0
Microsoft Phi (Phi-3, Phi-3.5 series (Abdin al., 2024))
MIT License
Notes on Language Model Licenses: OpenAl Models' API use governed by OpenAl's Usage Policies and Services Agreement. Customer Content (input/output for paid API tiers) is not used for training OpenAI models, as per their Services Agreement. Google Gemini API is governed by Google APIs Terms of Service and the Gemini API Additional Terms of Service. Data usage for improvement depends on the service tier, according to the Gemini API Terms of Service. Google Gemma Open Weights are governed by the Gemma Terms of Use, allowing modification and distribution with attribution and adherence to a Prohibited Use Policy. Meta Llama Models are released under version-specific Llama Community License Agreements, requiring attribution and adherence to an Acceptable Use Policy (AUP). Commercial use by entities with over 700 million monthly active users typically requires separate license, as detailed in the Llama 3 Community License. for example. DeepSeek AI Models' code is often MIT licensed. Some model weights are also MIT (e.g., V3-0324 release), while others are under a custom DeepSeek Model License with use restrictions. The specific license for each model must be checked. Mistral AI Models, such as Mistral-7B-IIsstuee-vvo
(Jiang al., 2023), are available under Apache 2.0. Premier models may have different licenses. Alibaba Qwen Models mostly Apache 2.0. However, specific versions like Qwen2.5-333 --B-trruuttt use a nommcommercial Qwen RESEARCH LICENSE
LICENSE AGREEMENT. The license for each specific model should be verified. 01.AI Yi Models' open-source releases (AI et al., 2025) are under Apache 2.0. Microsoft Phi Models, such as those described by Abdin et al. (2024), are generally released under the permissive MIT License.
Compliance Statement To the best of the authors' knowledge, and based on the diligent research of the licenses and terms detailed herein, the use of all software, datasets, benchmarks, and language models in this study complies with their respective governing terms and conditions. This proactive approach to documenting and adhering to licensing requirements is fundamental to responsible and ethical AI research.
35

---

Published as a conference paper at ICLR 2026
Prompt O.1: Example Prompt of BBH, few-shot CoT Prompts are formed by original research team
You are logic expert, you are given questions that involve enumerating objects and asking the model to count them.
Example 1:
<Example Question
Answer:
<Example Answer 1>
Example 2:
<Example Question 2>
Answer:
<Example Answer 2>
Example 3:
<Example Question 3>
Answer:
<Example Answer 3>
Example 4:
<Example Question 4>
Answer:
<Example Answer 4>
Example 5:
<Example Question 5>
Answer:
<Example Answer 5>
Answer Formatting:
Multiple Choice Questions with Options: Only select from the provided options (e.g., A. B. C, or D). you calculate a numerical answer (e.g., "10") that matches an option, respond with the corresponding option letter (e.g.. A ()), not the number itself. Failing to select the option will be marked as incorrect.
Short Answer Questions: Provide the final answer in the format correct answer. Do not include any additional formatting explanations inside "=". Do not answer only the serial number as well, Remember if answer is I.Henry, respond with
where X is the
Henry but not
2. Answer Example:
For multiple choice: the calculated answer is 10 and "10" corresponds to option C, respond with C
For math questions: the correct answer is "42," respond with 42 For short answer question other than math, Remember if answer I.Henry, respond with
Henry but not
Additional Instructions:
-Place any reasoning calculations outside of the "=" notation if necessary.
Uuee "=" only for the final answer.
Think step by step and answer carefully.
You must think before answering the question.
You must answer step by step.
Question: <question>
36

---

Published as a conference paper at ICLR 2026
Prompt O.2: Example Prompt of GSM8k, CoT Prompts are sampled from original test set
You are math expert, and you are tasked with answering questions math with example to reference.
Example 1:
<Example Question
Answer:
<Example Answer 1>
Example 2:
<Example Question 2>
Answer:
<Example Answer 2>
Example 3:
<Example Question 3>
Answer:
<Example Answer 3>
Example 4:
<Example Question 4>
Answer:
<Example Answer 4>
Example 5:
<Example Question 5>
Answer:
<Example Answer 5>
You'v 'ee've finished reading all the examples
Now read the question carefully and answer according to the following guidelines:
1. Answer Formatting:
Multiple Choice Questions with Options: Only select from the provided options (e.g., A, B. C, D). If you calculate numerical answer (e.g.. "10") that matches option, respond with the corresponding option letter (e.g.. AAA"), "),, not the number itself. Failing to select the option will be marked as incorrect.
Short Answer Questions: Provide the final answer in the format "[_x J.. where X is the correct answer. Do not include any additional formatting or explanations inside "-". Do not answer only the serial number as well, Remember if answer is H.Herry, respond with Henry but not
2. Answer Example:
For multiple choice: If the calculated answer is 10 and "10" corresponds option C, respond with CC
For math questions: the correct answer is "42," respond with 42 h
For short answer question other than math, Remember if answer is I.Henry, respond with
Henry but not
3. Additional Instructions:
Pllace any reasoning or calculations outside of the "-" notation if necessary.
Use "=" only for the final answer.
Think step by step and answer carefully.
You must think before answering the question.
You must answer step by step.
Question: <question>
37

---

Published as a conference paper at ICLR 2026
P
VERIFICATION TEST EXAMPLE
Prompt P.1: Verification Test Example
Welcome EIP Test Program!
Please enter your First Name for identification:
Now Group / 100 (Group ID: 0)
Progress:
0.0%
Question:
Evaluate the expression (751 745) + (748 742) (745 - 739) + (742-73)) + +....+4440 (499--43)) + (496 490).
2. Question:
What is the value of the inflection point of f(x) 10
A. 2.000
1.587
C. 0.693
D. 1.203 E. 3.014
F. 2.718
G. 4.000
H. 3.142
I.
1.000
J. 2.301
Which question is more difficult? Enter 0 if unable to judge, otherwise enter 1 or 2:
38

---

Published as a conference paper at ICLR 2026
Q DISCLOSURE OF LLM USAGE
We used large language models solely for editorial assistance (grammar, phrasing, and clarity). No model used to generate technical content, derive equations, design experiments, analyze results, or write code. All datasets, algorithms, and empirical results originate from the authors' implementations and public benchmarks. No proprietary or sensitive data were submitted third third-party services.
39


Reviews:
[{"review_id": "dzUBi6a0WW", "reviewer": "Reviewer_R6mp", "summary": "This paper proposes RankLLM, a difficulty-aware evaluation framework that jointly estimates question difficulty and model competency via a damped random walk on a model–question bipartite graph, yielding a unique stationary distribution for both scores. The method operationalizes difficulty through model failures and propagates scores bidirectionally between questions and models. Experiments span 30 models over 35,550 questions from six benchmarks, with reports of strong alignment with human difficulty judgments and robustness analyses.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and timely: moving beyond flat accuracy to a difficulty-sensitive ranking that better separates closely matched models. \n\n2. The method is simple yet principled—formulated as a damped Markov chain with a uniqueness/convergence guarantee—and scales to large pools and datasets.\n \n3. The paper is well written and easy to follow, with a clean derivation and a clear pipeline figure that makes inputs, transitions, and stopping criteria explicit.", "weaknesses": "1. Rankings are sensitive to who is in the pool: adding many weak models inflates failure mass, makes those items look “hard,” and artificially boosts any model that solves them. This causes rank shifts without any change in per-item accuracy and breaking cross-study comparability.\n2. The score scale itself changes with the participating models and the domain mix, so adding or swapping peers alters the baseline. There is no common unit across model clusters or data clusters. Results cannot be compared across studies or over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gKOLHYPHXr", "reviewer": "Reviewer_XEuy", "summary": "The core contribution of this paper is the proposal and validation of an evaluation framework named RankLLM. This framework utilizes an iterative algorithm to jointly estimate question difficulty and model competency. \nThis process is modeled as a bidirectional score propagation on a bipartite graph connecting models and questions. The introduction of a damping factor ensures the algorithm converges to a unique stationary distribution, yielding final competency scores for models and difficulty scores for questions. The authors validate the framework's effectiveness, robustness, and scalability through large-scale experiments on 6 popular benchmarks, involving 30 models and over 35,000 questions.", "questions": "please see Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method of jointly modeling question difficulty and model competency via score propagation on a bipartite graph is highly novel and intuitive. Treating question difficulty and model competency as interdependent and co-evolving variables is more dynamic and sound than traditional static methods based on accuracy or IRT.\n2. The paper provides extensive validation across multiple mainstream benchmarks and 30 models. The results are convincing, particularly highlighting the method's outstanding performance in aligning with human judgment and its computational efficiency.\n3. The study not only proposes a new method but also uses it to reveal several empirical findings that are insightful and of practical value to the LLM field.", "weaknesses": "1. The paper proves that the algorithm converges for any α ∈ (0, 1) and shows its effect on convergence speed in the appendix. However, it does not thoroughly investigate the impact of the choice of α on the final scores and rankings for model competency and question difficulty. It is unclear whether different values of α could lead to changes in model rankings. The paper lacks a discussion on a principled method for selecting an optimal α or a sensitivity analysis of the final results to this parameter.\n2. If a specific model family (e.g., the Llama series) shares a common 'blind spot' for a certain type of reasoning, questions targeting this weakness could be erroneously labeled as 'extremely difficult.' Consequently, when a new model with a different architecture correctly answers these questions, its resulting competency score boost might be constrained by the initial competency landscape dominated by the biased model family. \n3. While the method is innovative, the definition of 'difficulty' fundamentally remains dependent on model performance. The essence of difficulty is still derived from model failure rates, and the paper does not introduce external criteria or a theoretical framework to independently validate the soundness of this difficulty definition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ux2OG9PS5G", "reviewer": "Reviewer_urAu", "summary": "The core of this paper lies in addressing a major flaw in current evaluation methods for Large Language Models (LLMs): existing benchmarks (such as MMLU and MATH) typically only calculate overall accuracy, treating all questions equally without considering their actual difficulty levels. This approach may lead to distorted evaluation results. For instance, a model that answers many simple questions correctly might appear stronger than one that answers fewer but extremely difficult questions correctly.\n\nTo solve this problem, the authors propose a new evaluation framework called RankLLM.\n\nThe core idea of RankLLM is to move beyond evaluating models or questions in isolation; instead, it quantifies both \"model capability\" and \"question difficulty\" simultaneously. It is based on an intuitive interaction logic:\n\n- If a model can correctly answer a widely recognized difficult question, its capability score should be higher.\n- If a question is answered incorrectly even by widely recognized strong models, its difficulty score should be higher.\n\nBased on this logic, RankLLM constructs a \"bipartite graph\" between models and questions. It then uses an iterative algorithm called \"bidirectional score propagation\" to calculate the final scores. This process iterates continuously until the model capability scores and question difficulty scores reach a stable equilibrium.\n\nIn conclusion, I think this is a simple but interesting work.", "questions": "1. Regarding the dependence on the \"evaluation model pool\": The abstract mentions that \"using a diverse (mixed-scale) model pool yields the most accurate difficulty estimates.\" I am curious about how the authors define a \"sufficiently diverse\" pool. For a new user looking to adopt this framework, how many models—and of what types—would be required to obtain a reliable (stable) difficulty ranking?  \n\n2. Regarding the 90% \"human agreement\": This is an impressive figure (mentioned in Section 3.3). I would like to delve into the details of this experiment: How were the \"human experts\" selected? How was \"agreement\" specifically calculated? Additionally, what was the level of \"agreement\" among the human experts themselves (i.e., their internal consensus)?  \n\n3. Regarding the limitation of \"single-dimensional\" capability: This method seems to generate a single \"capability score\" ($\\pi_m$) for each model. In practice, however, a model might excel in mathematics but perform poorly in writing. I am interested to know whether the authors investigated this \"unbalanced capability\" phenomenon. Alternatively, can the framework be extended to generate a multi-dimensional \"capability radar chart\" (e.g., a mathematics difficulty score and a writing difficulty score) instead of just an overall ranking?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "By introducing \"question difficulty\" as a core variable, the RankLLM framework significantly outperforms traditional evaluation models that prioritize \"accuracy above all else\".\n\n1. Finer Differentiation (Beyond Accuracy)\nIn traditional methods, models receive the same score for answering a simple question and a difficult question correctly. RankLLM breaks this limitation by assigning a higher \"capability score\" to models that solve difficult questions.\n\n- As shown in the simulation experiments in Section 4 of the paper, when two models have the same overall accuracy, RankLLM can accurately identify the model that solves more difficult questions (M1 > M2), while traditional accuracy-based methods fail to make such a distinction.\n\n2. High Alignment with Human Judgment (Strong Consistency)\nThe \"gold standard\" for an evaluation framework is whether it aligns with the judgments of human experts.\n\n- Section 3.3 of the paper demonstrates that the ranking of question difficulty generated by RankLLM matches the consensus of human experts by up to 90%. This is far higher than that of mainstream statistical models such as Item Response Theory (IRT), confirming the validity and reliability of its evaluation results.\n\n3. Efficiency, Scalability, and Robustness (Engineering Practicality)\n- **High Efficiency**: On a dataset with 35,000 questions and 30 models, the algorithm converges in only 0.006 seconds (Section 3.4). This allows it to be easily deployed in large-scale leaderboards that require frequent updates.\n- **Strong Robustness**: Experiments prove that even if a large number of models (up to 15) are randomly removed from the evaluation pool, the relative rankings of the remaining models and the difficulty rankings of questions remain highly stable (Section 3.5, Table 5).\n\n4. Automated Difficulty Quantification (No Manual Annotation Required)\nTraditional benchmarks (such as MATH) require human experts to pre-label subjective difficulty levels (e.g., Level 1~5).\n\n- RankLLM, by contrast, is fully automated. It \"emerges\" a definition of difficulty from the performance of the model group, without relying on any manual prior knowledge, making it more objective.", "weaknesses": "1. The core of RankLLM lies in the \"relative\" relationship between models and questions. The difficulty of a question is defined \"relative to\" the pool of models participating in the evaluation. This gives rise to an issue: if the model pool itself is biased, the resulting difficulty scores and capability rankings may also be biased.\n- The paper itself demonstrates this in Figures 7 and 8: if only a \"large-scale\" model pool is used, 58% of the questions will be classified as \"excessively easy\"; if only a \"small-scale\" model pool is used, 30% of the questions will instead be classified as \"impossible\".\n- While the paper proves that \"removing\" models from the current pool is robust (Table 5), it does not explore what would happen to the overall ranking if a completely new \"super model\"—with capabilities far exceeding all existing models in the pool is added. Theoretically, the addition of this new model would \"lower\" the difficulty scores of many previously \"difficult questions\", potentially causing drastic changes to the entire ranking list, especially for models in the upper-middle tier.\n2. This framework ultimately computes a single capability score ($\\pi_m$) for each model and a single difficulty score ($\\pi_q$) for each question, which represents an oversimplification of the actual scenario.  \n\n- Model capabilities are inherently multi-dimensional. For example, a model might demonstrate exceptional performance in \"mathematical reasoning\" while showing significant weaknesses in \"historical knowledge\". RankLLM tends to \"average out\" these varied capabilities, yielding a moderate overall score. This averaging effect can lead to a \"well-rounded\" model and a \"specialized (or lopsided) expert\" model receiving similar overall rankings, ultimately masking the true capability profile of each model.  \n- Question difficulty also exhibits multi-dimensional characteristics. A single question may simultaneously demand proficiency in \"knowledge retrieval\", \"multi-step reasoning\", and \"spatial imagination\". RankLLM lacks the ability to decouple these composite difficulty components, making it unable to accurately capture the nuanced difficulty structure of such questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "83cqhdt7oK", "reviewer": "Reviewer_Kyip", "summary": "This paper introduces RankLLM, a difficulty-aware framework for evaluating Large Language Models (LLMs) by jointly estimating question difficulty and model competency. The core mechanism is a bidirectional score propagation process over a directed bipartite graph of models and questions, using model successes and failures to iteratively reinforce difficulty and competency scores. The approach is non-parametric, scalable, and designed for use with large evaluation pools. Empirical studies are conducted on 30 LLMs across 35,550 questions spanning several benchmarks. The authors report that RankLLM exhibits strong alignment with human judgments of question difficulty, outperforms standard baselines like Item Response Theory (IRT), converges rapidly, and scales to large datasets. The framework is further analyzed for stability, extensibility, and computational cost, with robustness claims under dataset/model perturbations.", "questions": "1. Are you implementing IRT yourselves or using a well-established package? If you implement IRT yourselves, did you carefully validate your output with a widely validated package (say, an R package)? In Table 3, it is reported that 1PL IRT takes ~30 mins to fit. I happened to work on IRT for AI evaluation; my 1PL IRT implementation takes ~30 seconds to fit on ~180 LLMs and ~80,000 questions (on an A100 GPU, though).\n\n2. Given that different benchmarks have different measurement objectives, is there a reason you fit all benchmarks together?\n\n3. IRT is well established in psychology; Is your method thoroughly studied in any other fields? Does it have an original reference in psychology or measurement science, or other fields? If so, is there previous work that reaches the same conclusion as you do (i.e., outperforming IRT)? I have heard about Elo-rating systems involving test takers and items (which can be shown to be equivalent to 1PL IRT in theory), but I am not familiar with the details.\n\n4. IRT enables computerized adaptive testing (CAT), which can reduce evaluation compute for new LLMs. To create a leaderboard, the computational cost of querying benchmark questions to LLMs seems much larger than the difficulty/competence update step. Can your method similarly support computerized adaptive testing?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Principled, mathematically sound methodology: The central mechanism relies on a well-formulated ergodic Markov chain on a bipartite graph, with mutually reinforced score propagation between questions and models.\nComprehensive empirical validation: Evaluation extends to 30 models over 35,550 questions. Large-scale empirical studies support many intuitive claims, clearly presented and visualized.\nHuman judgment: This is a great source to measure the goodness of fit for item difficulty.\nOpen and reproducible claims: Claims are signposted as non-parametric and hyperparameter-light, and computational infrastructure (including licensing) is disclosed.", "weaknesses": "No major weakness, however see my questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)