Unearthing Skill-level Insights for Understanding Trade-offs of Foundation Models
Authors: Mazda Moayeri
Vidhisha Balachandran
Varun Chandrasekaran
Safoora Yousefi
Thomas FEL
Soheil Feizi
Besmira Nushi
Neel Joshi
Vibhav Vineet
Venue: ICLR 2025 Poster (2025)
Keywords: evaluation
skills
tradeoffs
frontier models
llm
large multimodal model
rationale
error analysis
interpretability
Primary area: datasets and benchmarks
TL;DR: Automatic skill-based evaluation of LLMs and LMMs by parsing skills from rationales for existing benchmarks

Abstract: With models getting stronger, evaluations have grown more complex, testing multiple skills in one benchmark and even in the same instance at once. However, skill-wise performance is obscured when inspecting aggregate accuracy, under-utilizing the rich signal modern benchmarks contain. We propose an automatic approach to recover the underlying skills relevant for any evaluation instance, by way of inspecting model-generated {\em rationales}. After validating the relevance of rationale-parsed skills and inferring skills for $46$k instances over $12$ benchmarks, we observe many skills to be common across benchmarks, resulting in the curation of hundreds of \emph{skill-slices} (i.e. sets of instances testing a common skill). Inspecting accuracy over these slices yields novel insights on model trade-offs: e.g., compared to GPT-4o and Claude 3.5 Sonnet, on average, Gemini 1.5 Pro is $18\%$ more accurate in \emph{computing molar mass}, but $19\\%$ less accurate in \emph{applying constitutional law}, despite the overall accuracies of the three models differing by a mere $0.4\\%$. Furthermore, we demonstrate the practical utility of our approach by showing that insights derived from skill slice analysis can generalize to held-out instances: when routing each instance to the model strongest on the relevant skills, we see a $3\\%$ accuracy improvement over our $12$ dataset corpus. Our skill-slices and framework open a new avenue in model evaluation, leveraging skill-specific analyses to unlock a more granular and actionable understanding of model capabilities.

Published as a conference paper at ICLR 2025
UNEARTHING SKILL-LEVEL INSIGHTS FOR UNDER- STANDING TRADE-OFFS OF FOUNDATION MODELS
Mazda Moayeri'.2 Vidhisha Balachandran' Varun Chandrasekaran !.3
Safoora Yousefi', Thomas Fel', Soheil Feizi", Besmira Nushi', Neel Joshi', Vibhav Vineet
Microsoft Research AI Frontiers, UUniversity Maryland. University of Illinois Urbana- Choama-Cammpainn "Harvard University Kempner Institute mmoayeri penyyerruune @umd. vivineed @mi crosoft. com
ABSTRACT
With models getting stronger, evaluations have grown more complex, testing mul- tiple skills in one benchmark and even the same instance once. However, skill-wise performance is obscured when inspecting aggregate accuracy, under- utilizing the rich signal modern benchmarks contain. We propose an automatic approach to recover the underlying skills relevant for any evaluation instance, by way of inspecting model-generated rationales. After validating the relevance rationale-parsed skills and inferring skills for 46k instances over 12 benchmarks, we observe many skills to be common across benchmarks, resulting in the cu- ration of hundreds of skill-slices (i.e. sets of instances testing common skill), Inspecting accuracy over these slices yields novel insights on model trade-offs: e.g., compared GPT-4o and Claude 3.5 Sonnet, on average, Gemini 1.5 Pro 18% more accurate in computing molar mass, but 19% less accurate in applying constitutional law, despite the overall accuracies of the three models differing by a mere 0.4%. Furthermore, we demonstrate the practical utility of our approach by showing that insights derived from skill slice analysis can generalize to held-out instances: when routing each instance the model strongest the relevant skills, we see a 3% accuracy improvement over our 12 dataset corpus. Our skill-slices and framework open avenue in evaluation, leveraging skill-specifie analyses to unlock a more granular and actionable understanding of model capabilities.
INTRODUCTION
Recent years have seen benchmarks evolve to keep up with ever-advancing models. While classical benchmarks tested specific capabilities, like recognizing digits (LeCun et al. 1998) or classifying sentiment (Bowman et al., 2015), modern benchmarks measure proficiency in numerous capabilities simultaneously. drawing questions increasing difficulty from more diverse domains (Mialon et al.) 2024 Yue et al.) 2024 Wang et al. 2024). As the questions
test models on have grown more complex, aggregate performance measures provide less understanding about model proficiency in specific abilites. For example, as shown in Figure] I we find that
dozen benchmarks, GPT- 40, Gemini 1.5 Pro, and Claude 3.5 Sonnet (OpenA] 2024 Gemini-Team 2024 Anthropic 2022) achieve overall accuracies within 0.4% of one another, leaving an open question: Are these models all the same, or are valuable insights being averaged away?
Manual annotations of categories across instances (e.g. YYng al. (2024) and Liu al. (2023) in- clude "ability" tags) enable going beyond accuracy, but at cost for benchmark creators that rises with the number and difficulty of test questions. This results in few (if any) and non-standardized an- notations, restricting cross-benchmart aggregation, even though many benchmarks have large over- lap the (implicit) skills they test (Miao al. 2020 Cobbe et al. 202T). Prior works show promise in automatically grouping inputs by attributes to extract image classification failure modes (e.g. 'fails
2020
Work done while interning Microsoft Research

---

Published as a conference paper at ICLR 2025
Roughly equal accuracy averaged over benchmarks testing many abilities at once
Large differences unearthed along skill slices, towards understanding specific strengths and weaknesses le.g. for Gemini 1.5 pro)
interpreting evolution over new releases
36, 10
64.55
predeccsu SU00000
68.3% over SHix git-4o
thets gemee2..pp
35 74
79
rentbench
rorda eoedbeech Popular Brechmarks
26 computting molar masa (:51 inssercea)
20
6
20
41
applying consttutional (215 rntanco)
understarding logal procodurs [110 intanc))
(Teatin many Capabiition at mnnx)
Figure 1: We leverage penell-generate rationales to extract the skills relevant to any evaluation instance. Inspecting accuracy along skill-slices (instances drawn across benchmarks involving the same skill) surfaces fine-grained insights otherwise obfuscated by aggregate accuracy.
on white foxes') ) (Eyuboglu et al. 2022] Rezaei et al. (024), but these methods fail to produce deeper insights on the underlying competencies that models use across contexts (e.g. 'fails on counting"). )
To this end, we propose method to automatically discover skills related to any evaluation instance, toward a finer-grained understanding of model capabilities from existing benchmarks. We consider skills latent features pertaining to "how" model must operate the steps involved in solving a given task- whereas a general attribute relates to "what" is being addressed the observable characteristics of the instance. Moreover, the relevant skills for instance may not be readily apparent through superficial examination, often only revealing themselves upon inspection of the solution. This complexity highlights the challenge of automatically inferring relevant skills and underscores the need to develop new approaches for their effective identification.
To this end, we propose harness the antionale-generating ability of strong models infer skills, as rationales (i.e. explained solutions) elucidate the steps involved in performing a task, and have had many benefits in the past [Wei et al. 2022] Kojima et al. 2022 Singh et al. [024 [hsaa al.) 2019 |Mitra et al. [2023 Hsich et al. 2023). In our work, we use rationales not to directly analyze the rationale- natonaal-generating model, but instead annotate data, on which any model can be evaluated.
Specifically, given an evaluation instance, we instruct a strong model (e.g. GPT-44o) to generate a detailed step-by-step rationale, along with the skill applied in each step. We then parse skills per in- stance for numerous benchmarks and aggregate instances along skill-slices, i.e. subsets of instances drawn from multiple benchmarks that all involve a specific skill, enabling measurement of model proficiency for that skill. After applying our technique to over 45k instances from dozen popular modern benchmarks (primarily multimodal), we find hundreds of skills with slice sizes of 2 100. To validate the relevance of our skills, we devise an automatic verification protocol and observe that our automatically extracted skills are relevant, even in cases where the annotating model fails in solving the query problem. We will release all our rationales, skill annotations, and skill-slices, which term the Skill-Index, to the public at liggtuuu.bb.... com/ mi crwsoft/ ski 11-1 slice- inseeinsiii igtts
Having verified the accuracy of constructed skill-slices, we present multiple ways skill annotations and skill-slices can be used to inform our understanding and usage of current large models. First, we conduct detailed skill-level analyses of 6 state-of-art models from the GPT, Gemini, and Claude families, identifying practical insights into model strengths and weaknesses at previously underexplored granularities. Despite roughly equal average accuracy across our corpus for GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet, we discover skills for each model where accuracy far exceeds that of its counterparts, with gaps as large as 20% (see Figure D]. For example, we find Gemini 1.5 Pro to be much stronger in math and science skills, like 'computing molar mass', while falling far behind for legal skills like 'applying constitutional law'. Next, we show that our skill-slices allow for fine-grained understanding of improvements in new models within a model family: while Gemini 1.5 Pro did not improve over v1.0 on 'understanding legal procedures'. OpenAl and Claude models improved substantially, with Claude 3.5 Sonnet leaping nearly 50% over Claude 3 Opus.
Further, skill-level understanding of models can be directly utilized for instance level model selection based on skills required. When routing each query instance to a model with best perfor- mance relevant skills, we improvements in overall accuracy across dozen benchmarks by
2

---

Published as a conference paper at ICLR 2025
Evaluation Instance (MMMU, #744)
Select type interval.
(A) perfect (B) major (C) minor (D) diminished
Generated Rationale with localized skills
Post hoc Verification by
Step 1. Recognize the image
Mark if each 1isted skil1
Perception, Recognition, Symbel identification, Treble cief recagnition the treble stoff.
La relevant to the given
Concivsion: The image
Note dentification *Counting
identily
Skill: Perception, Visual Recagnition,
Music Theory
Note identification, Pitch recognition
Treble Clef Recognition
Concuusion:
*Scene Understanding
Pitch Relationships
Step 3. Determine the Interval
Skill Knowledge, Music Theory,
*Setting up Equations
X
Interval identification, Pitch relationships
"Randomly Sampled Negative Skills
Figure 2: (left) Sample GPT-4o generated rationale: each skill is listed under multiple names of cascaded gramularity, and localized to specific step and concluding claim. (right) Annotated skills can be verified independently with second model or human. We include randomly sampled negative skills and multiple verifiers to assure the quality of the verification, as detailed in 2.3
3%. including gains of 3.5 to 7% MMLU Pro (Wang et al. 2024 Finally, to assess model's proficiency on skill in isolation, we generate probing questions that directly target a specific skill. We find that model performance the synthetie prohing sets corroborates accuracies structed skill-slices, further strengthening our findings.
Our skill based framework and findings advocate for paradigm shift in foundation model evalua- tion, emphasizing the importance of skill-specific analysis to gain a more granular and actionable understanding of model capabilities. We summarize key contributions below:
Automated Skill Inference: We present scalable method that leverages model-generaan rationales to recover (and validate) the underlying skills relevant to any evaluation instance.
Novel findings from Skill-Slices; Analyzing skill-slices sets of instances testing a com- mon skill reveals new fine-grained insights on the trade-off's across leading models.
Skill Annotations for Popular Benchmarks: We release the Skill-Index, dataset of instance-level skill annotations and rationales, along with code to easily expand to new benchmarks, providing valuable resource for the research community.
CONSTRUCTING SKILL-SLICES USING MENERAGEN RATIONALES
We now detail our method for inferring relevant skills for each evaluation instance and aggregating skill annotations across datasets to form skill-slices (i.e. set of instances that share relevant skill). We utilize rationales, or step-by-step solutions, generated by a strong model (GPT-40) to facilitate the extraction of relevant skills. However, as evaluation instances are challenging by design and generating relevant skills requires a significant level of meta-reasoning, is important to assess the quality of produced skills. Thus, additionally present multifaceted automatic approach to validating the relevance of extracted skills, which we confirm aligns well with human judgements.
2.1 SKILL EXTRACTION VIA RATIONALE PARSING
Given an evaluation instance, we prompt GPT-4o generate detailed rationale (i.e. step-by-step solution), where each step only involves the use of single skill; figure 2 presents an example. We additionally instruct the model to, after each step, list the applied skill with multiple names of cas- cading granularity. For example, the skill 'treble clef recognition" is prepended with 'perception, visual recognition, symbol identification Listing multiple names of different granularity per skill
If some of these skills are unfamiliar to you, you're not alone! Many benchmarks today ask questions that are challenging to anyone without expertise in the relevant domain. This precisely motivates relying stronger models help make sense of these benchmarks, as manual inspection has increasingly limited utility.
3

---

Published as a conference paper at ICLR 2025
Rand Sampled Negative Skills
MMVet
Correctly
MMVP
Answereed
90.10
9.90
MMMU
Incorrectly
"Groundtruth"
RealWard QA
Answered
Dataset
Mench
MMMBeench
VibeEval
Listed Skills
MathVVsta
MME
5.89
94.11
MMLU Pro
SEEDBench
MMC
relevant Verifier Output
Relevant
Skill Relevance
Figure 3: (left) Post-hoc verification shows verifiers reliable, they admit low rates of false positives (marking a randomly sampled negative skill as relevant), (right) Rate that GPT-4o-annotated skills are marked as relevant, separated by GPT-4o correctly answered the underlying evaluation instance (blue) not (orange). Empirically, annotated skills have high relevancy rates even when the annotator incorrectly answers the question.
skills are relevant, and that automatic
a simple way to increase the number of slices evaluation instance falls into, leading a greater quantity and diversity of skill-slices. Further, each granularity has unique and complementary ad- vantages: fine-grained skills are more specific, while coarse-grained skills lead to slices with larger number of samples, making an accuracy estimate over such slices more reliable. Also, in order to standardize the response structure, we include an in-context example in our prompt, which enables simple parsing of rationales to extract skills. See Appendix for complete details.
Note that other methods could possibly be used to annotate relevant skills, like directly prompting GPT-4o to list them. We prioritized maximizing the total count and size of resultant skill-slices. In comparing the skills per instance obtained by direct prompting vs. our rationale parsing, we indeed find that rationale parsing results in higher average count and diversity in grain (Appendix C.I
2.2 CURATING SLICES ACROSS BENCHMARKS
With parosin,.,.,iing. we tag relevant skills for 46k evaluation instances from 12 benchmarky that are popular (i.e. commonly featured in evaluations for recent model releases) and unsaturated (SOTA under 90% accuracy), resulting in 690k total skills (128k unique) over 202k rationale steps. Importantly, we observe that skills cut across benchmarks. That is, many skill-slices of non-trivial size are formed when including instances across benchmarks, because models re-use skills in diverse contexts. Namely, 278 skill-slices have at least 100 unique instances each. This number rises to 332 after de-duplicating via a tight clustering on the text embeddings of the skills. Aggregating across benchmarks enables us to analyze skills that in the long tail for one benchmark, but get resur- faced when many different benchmarks are joined together. Indeed, slice count increases by 51% when slicing across benchmarks instead of only drawing instances from one benchmark time.
2.3 AUTOMATED VALIDATION OF SKILL RELEVANCE
In the absence of fine-grained ground truth skill annotations that would confirm whether generated skills are relevant or not, we design automatic validation approach. Namely, we propose two automated methods to directly validate the relevancy of listed skills: post-hoc verification and inter-(skillaannotuh agreement. The latter checks the overlap in skills listed for the same instance by two different annotators. Here, we focus on the former and defer full details of both approaches to Appendix C.2 Post-hoc verification consists of second 'verifier' model which inspects an evaluation instance with a list of skills and marks whether each skill is relevant or not. We form the list of skills by combining the skills for the instance with equally sized list of negative skills, randomly sampled as follows: with S denoting all skills from our corpus, negative skills for an
"See Appendix for complete details on datasets annotated and the slice formation process. We employ high minimum cosine similarity threshold of 0.95 per cluster de-duplicate skills listed under slightly different names; e.g., 'trigonometry' and 'trigonometric calculation' have similarity of 0.95.
4

---

Published as a conference paper at ICLR 2025
GPT-4v to GPT-4o
Gemini 1.0 Pro to Geminii 1.5 Pro
Claude 3.0 Opus to Cloude 3.5 Sonnet
5.1% Avenge Improvernnt
20 5% Inprowement
7.7% Improvement
Arrunay Imprommett
ingorsmeccc funstoons
sggonett calulaiioe
rati and proprttone
stocatical esiiaation theoy
lighl pricoppe
procesluee
lagal undentardnng
lighl procedire
systhess informrion
ruees evidence
undestarding
electricll theory
equantions
bask dividon operation
freetion computerion
algpbr
optics theory
compuration
thermody mumuiss
numeicall cosoittttooon
propont ins
ded deductionn
logal piiniil
niees exidence
Intorprctationn
applicatoo prapant
property
idual gas
match numencal
sill
Figure 4: Skill-slices shed insight on how models evolve over new releases. For GPT and Claude models, skills related to law see the largest increases in accuracy, while Gemini models improve most in performing math and science skills.
instance x with skills S. are drawn from {s S es, sim(s,s; < T}. Here, 'sim' is the similarity of two skills, computed via the cosine similarity of text embeddings from frozen text embedder. The threshold r is selected based on the text embedder so that no negative skill is equivalent up to paraphrasing to an annotated skill for the instance. We include negative skills as quality controls to ensure that the verifier model does not simply mark all skills as relevant.
Figure BaureBsshowe shows the results of post-hoc verification of skills for 100 skills per dataset averaged over three verifier models (Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-4.D). We observe 94.1% of skills annotated by GPT-4o are verified as relevant. A smaller human validation of 640 total skills results in 95.7% relevancy rate, well as 92.8% agreement with automatic verifiers, indicating that post- hoc verification reliable automated metric for assessing skill relevancy. The right panel shows that the relevancy rate is roughly equivalent, regardless of if the annotator correctly answers the derlying evaluation instance; crucially, this enables using automated skill annotations on instances that are interesting to evaluate (i.e. hard enough to stump some models). Thus, while rationales may at times be partially incorrec f they can still be leveraged to shed insight on data at scale.
3 ANALYZING FOUNDATION MODELS WITH SKILL-SLICES
We now leverage the skill-slices obtained in the previous section to better understand and utilize frontier models from OpenAl, Google, and Anthropic. First, we identify skill-slices with vastly different accuracies across releases from the same family, as well as across families, which are obfuscated when inspecting overall accuracy. Then, having discovered models differ in their specific strengths and weaknesses, we show how overall accuracy can be improved by choosing the best model per instance or dataset, leveraging inferred skills and computed model-wise skill accuracies.
3.1 FINER-GRAINED INSIGHTSS FROM EXISTING EVALUATION INSTANCES
Understanding Model Evolution. Figure shows average improvement, along with skill-slices with greatest and least improvement, for the last two releases in the Open AI GPT, Google Gemini, and Anthropic Claude f For all model families, we find skills where improvement is more than twice the average, as well skills where the model does not improve at all. Notably, both GPT and Claude models see greatest increases in skills related to law, with the Claude-Sonnnet 3.5 im- proving over Claude-Opus 3.0 by staggering 50% along numerous law skill-slices, suggesting improving legal abilities of their models may have been a recent priority for OpenAl and Anthropic.
*GPT-4v denotes GPT-4 Turbo 2024-04-09, and GPT-4o denotes GPT-4o 2024-05-13 throughout our paper. In fact, there's an error the rationale Figure 2 the notes are Bb C. However, the skills are accurate. 'Because Gemini-1 1.0-po is not multimodal, present results for skill-slices curated from 12k language- only evaluation instances (instead of entire corpus) with least 100 instances within each slice.
5

---

Published as a conference paper at ICLR 2025
Strenghi
Strengths
Strongths
houncinng (:30
stynicl (+10 3%5)
system (+10.15))
licating an object the
setting equations [+22 (+22.2%
comparing data [- (-16.0%))
image (-10.3%) 3%)
depth perreption (+166)
consersane cakoriation
(+30 6501
divectons determization
(+015)
matheraatia substhution calou 20
[e 35S
valcic on 25)
(gft color oifleretittoon
motar mont (+117
itendiying categorie
anstion [+900)]
synchesiiing information (46.5%)
[-0 25
(+17.15))
traffc sigrol slentfization
[+16 (l))t
aigalea (+19 shitt
prayeicall reasoringg (+7.3%)
vocda (+9.0%)
understanding
guittflnegg color on otjects
Fraction computattion
breed aralyds
coperraiin conoaet
[-8.7%)
(-7.75))
Cectie [+8 1id
extreting sumerical
gnaph ideroioaaiinn 8868%)11
80 Apounayy
Accursty
Accurnty
GPT-4o
Gemini 1.5 Pro
Claude 3.5 Sonnet
Wegknespes
Comini
Claude 3.5
solving varialle (-13 850)
evidance (-21.55)11
undorsanding caunting (-11.1%
solving squrtiors. (-15 05011
legal procsdurr (-21 45611
fina smover salection (5 45%
legal
procosingg
Irggonometicc functions (-1655%)
inicpll (-16. 556
visual difecrniaaionn :-
Intiuutiin wabing
lagal undenstadd
muttiple penson identiication
manchie values (:16 25001
surface detal (7.9%)
(-16, 0%611
aquare noot (-21 155)
properte to law (-16.5 556111
architcciral iage
recogration [-6 (6.2%)
distiggsshing animal character
050)1
(-15 65611
(-7.6%
Hectrical theory Iisportii (:36 79317
othical thery (:18. 0%611
animol bekowior (13.1%01
urdersanniigg
Jaggootic analysi
undertandinng ecologr
[-6
75
Accurwty
Accuracy
Accunay
Figure 5: Unique strengths (top) and weaknesses (bottom) of GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet, relative to one another. We present skills where each model's 's slice accuracy is highest / lowest (respectively) relative to the average of the other two model accuracies. t denotes significance (p < 0.01) via Wilcoxon signed-rank test; it means both gaps vs. the other models are significant.
Finding I: Skill-level improvements across model releases greatly differ (i) from average improvement and (ii) between families. The biggest leaps for GPT and Claude families were for legal skills, while Gemini improved most for math and science skills (figure 4
Interpreting Trade-offs. We now directly compare GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet on all skill-slices with at least 100 unique evaluation instances. While overall accuracies for the three models across our evaluation corpus falls within a half percent of one another, we obxerve differences high as 25% for certain skills, as shown in figure 5 Further, some patterns emerge: Gemini 1.5 Pro is distinctly strong for math and science, while GPT-4o's relative strengths pertain to visual skills like color differentiation and object localization, signal identification, where GPT-4o is on average 16. 16.7% more accurate. Interestingly, Gemini 1.5 Pro's 's greatest relative weaknesses nearly all concern law, which we found above be the key skills that GPT-4o and Claude 3.5 Sonnet improved upon compared to their respective predecessors.
well as the real-world skill of traffic
Finding 2: Models have unique strengths and weaknesses: e.g., GPT-4o excels in visual skills, but lags behind on various math skills. Conversely, Gemini 1.5 Pro outperforms both other models math and science skills, while falling far behind for legal skills (figure 5)

---

Published as a conference paper at ICLR 2025
Claude Sonnet 3.5
58.49
Skill-wise Accuracies on
MMLU Pro (Language- only))ly) )
0.8
GPT-Ao
y 0.M6, 3e
0.81,
Gemini Pru
67 71
50:47
Claude 3.5 Sonnet 0.79,
0.6
0.5
GPT-4o
68.
56.09
0.4
Scill
70
Routing
0.3
70
55
0.4
0.5
0,8i
0.7
01.8
12.9
Accuracy over 12 Datasets
MMLU Pro Accuracy
Skill-wise Accuracies on Multimodal Benchmarks
Figure 6: (left) A simple routing scheme that assigns each instance to the model with best accuracies for the skills relevant that instance can lead to noticeable accuracy gains. (right) Accuracies over skill-slices can generalize to unseen data. In fact, skill-wise accuracies computed over multimodal benchmarks correlate strongly with accuracies obtained over a language-only benchmark.
Unfortunately, since we have limited visibility into the training data and processes for these private models, we cannot offer explanations for the strengths and weaknesses we observe. Nonetheless, this illuminates an advantage of our method: by annotating evaluation instances. skill-slice analysis enables fine-grain insight on model capabilities, even when only granted black-box access.
3.2 GENERALIZATION OF SKILL-SLICE INSIGHTS
Our skill-slice analysis operates under the premise that the insights drawn from inspecting sufficiently-large slices from a wide range of sources can generalize to new instances. the in- sights generalize well, then understanding skill trade-offs between models enable employing the models in a more calculated manner. That is, if we have knowledge of the skills relevant to new instance, as well each model's skill-wise strengths, we can improve accuracy by routing that instance to the model whose strengths are most aligned with those skills.
To test this, we route each instance in our corpus to one of GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet, based on the skill annotations for that instance and the skill-wise accuracies per model com- puted over the remaining corpus (i.e. without the test instance). To obtain single score per instance model, we take a weighted average of skill-wise accuracies, where the weight for each skill is the inverse of its slice size (so to upweight finer-graine of figure 6. routing increases accuracy by up 3.2% compared to each of the frontier models alone on 12 datasets combined, including improvements 3.5 to 6.8% for MMLU Pro WWang et al. 2024
more specific skills). As shown in the left panel
We highlight the MMLU Pro results because the vast majority of the instances in the reference corpus (over which skill-wiss accuracies are computed) are from multimodal benchmarks, while MMLU Pro is language-only, making it good candidate to showcase the generalization of our approach. To study this deeper, we now partition each skill-slice based on the instance comes from MMLU Pro
of our multimodal benchmarks. Then, for each model obtain two paired sets slice-wise accuracies: one accuracy score per skill-slice, per partition. As shown the right panel of figure 6 observe strong correlations 2 0.79. p le - 7) between these two of scores for all three models. In addition to showing generalization, this result further validates our idea that skills pertain to deeper property of given instance (namely, what must be done solve it) than surface level attributes, like its modality.
Finding 3: Skill-slice accuracies enable instance-wise model selection. Routing each in- stance to the model strongest on the relevant skills results in a 3% accuracy gain, indicating that skill-slice analyses offer generalizable insight (see figure 6)
We note that this proof of concept utilizes accuracies computed over slices defined by a single skill, not accounting for the added difficulty created when certain skills used together. Nonetheless, framework opens the door to studying these second order effects, e.g., by surfacing error error-induss -oduiing slices defined by combination of skills, as done by Rezaei et al. 2024 for image classifiers.

---

Published as a conference paper at ICLR 2025
Rate of Inconsistency in Probe Answers
70
Average Rate of Inconsistency
Surface instance and rationalc-ste for skill to probe
60
[Tng questleat 3
Step Identify the Wotes
34.
Skilit Note identification
15
-Conclusion the notes are 8 and
8.33
Reframe concluding to probing question
The notes ore B and E.
What the notes?
30
Check consistency over multiple responses
20
10
Affot
Fond
eand
above answers consistent?
Median
Highest Accuracy
X
Accuracy
Accuracy
Figure 7: (left) Skills who's slices yield lowest model accuracy have the highest rates of incon- sistency when probed. Each point corresponds to one model's consistency rate for single skill. (right) Probe questions are generated by reframing rationale steps associated with a skill of interest.
CORROBORATING SLICE ACCURACIES WITH PROBING QUESTIONS
While skill-slices allow for approximating a model's proficiency at a skill by averaging accuracy over any instance where the skill is relevant, our rationale parsing framework enables second independent analysis to more directly probe single skill, without the effect of co-occurring skills.
Namely, rationale parsing localizes each skill to a specific step of the generated solution, as well as the resulting claim, which we instruct the vationale-generatiin example, as shown on the right in figure 7 the resultant claim to a step where the skill "[musical] note identification" is applied is "The notes are B and E". A claim can then be reframed as a question (e.g. "What are the notes?") probing a single skill, unlike the original question that often requires nu- merous skills and steps, any one of which could cause incorrect answer. We use the rate of incon- sistency, like in Wang et al. (003)), over multiple responses to each probing question as a measure of proficiency at the probed skill: if a model contradicts itself, it must have been wrong at least once.
model to include in its response. For
We now probe three sets of 20 skills: those with lowest, median, and highest accuracy based on the analysis in section 3] to sample skills with variety of slice accuracies. For each skill, we (i) gener- per probe per model, and (iii) evaluate consistency of the responses with GPT-4c7 As shown in 7 models contradict themselves a significantly higher rate for low accuracy skills, and inconsistency rises as slice accuracy falls, corroborating the skill-slice analysis. Overall, inconsistency correlates well with slice accuracy (r -0.675; Appendix Further, we find skills where all models contradict themselves more often than not, such as "[musical] note identification". "eyee direction analysis", and "enumerating objects", the last of which is well-documented limitation of VLMs YYuksekgonul et al. 2023 Paiss et al.) 2023).
ate 20 probing questions, (ii) obtain 5 responses per
Finding 4: Rationale-parsing enables targeting skills in isolation. This independent analy- sis yields corroborating assessments of skill proficiency as skill-slice accuracies (figure 77.
In addition to independently verifying the insights from slice accuracies, probe inconsistency offers complementary signal: since some evaluation instances engage multiple skills, low accuracy over one skill-slice could be due to a deficiency on another highly co-occurring skill. However, if a model proficient a skill, will always answer consistently (correctly) to probing questions. Thus, con- sistency on probing questions can be used to refine insights from skill-slice analyses, as skill with low slice accuracy and high probe inconsistency is almost surely deficient. We discuss automatic diagnosis of skill deficiencies further in Appendix E.3 leaving it as future application of our work.
While we use GPT-4o, forming probing questions and evaluating consistency are likely simple enough to he done reliably by open-source I.I.Ms (as both tasks text-only). See appendix E. for all prompts used.
8

---

Published as a conference paper at ICLR 2025
5 RETRIEVING EVALUATION INSTANCES FOR CUSTOM QUERY SKILLS
Finally, towards (a) direct comparison of our work to prior methods and (b) more flexible use of skill annotations, introduce the task of skill-based retrieval: given open- vocabulary query skill, can we retrieve relevant evaluation instances, so to build a custom skill-slice for specialized evaluation? This task boils down defining efficient metric that assigns similarity score between text query and an evaluation instance. Here, we focus on multimodal evaluation instances R
We consider baselines that leverage embeddings of the instance (either of just the image using CLIP Radford et al.] (2021), just the text question, or the average of both image and text), b. attributes of the input, either inferred by GPT-46 or provided as ground-truths by benchmark creators. In this latter case, we embed text attributes, along with the query skill, using text encoder, with which we compute similarity of an attribute to the query skill as the cosine similarity of their embeddings. To obtain a single similarity score for the GPT-4o attributes baseline where multiple attributes exist per instance, we average the top highest similaritie Our method is identical to the attribute-based approach, except that we leverage annotated skills instead of attributes. The baselines encapsulate how prior methods propose to group inputs to go beyond overall accuracy (Eyuboglu et al. 2022 Rezaei al. 2024), enabling direct comparison our method.
Setup: We take 20 skills from one bench- mark (MMBench), and retrieve 20 instances per skill per method from two sources: a. a single other benchmark: MMTBennch, selected because it has the highest number of ground- truth categories (162) out of our corpus, b. all benchmarks aside from MMBench, though we exclude ground-truth and GPT-4o inferred tributes baselines in the second case, for avail- ability and cost reasons respectively. Metrie: We evaluate each retrieved instance by present- ing along with the query skill to GPT-4o and prompting it to decide if the query skill is rele- vant or not; 'precision @ k = 20' is the fraction of the 20 retrieved instances deemed relevant.
Ground Truth Attributes
53.00
GPT-4o Attributes
56.75
Retrieval Method
Text Embeddings
29.25
Retrieved from:
27. 25
MM
Only
Image and Text Embeddings
43.50
Entire
48.25
Corpus
50,50
Image Embeddings
55 50
Skills (Ours)
74. 75.75
88. 00
As shown in figure g retrieval using skill an- notations far exceeds the precision for any other baseline in both settings, with improve- ments of 18-45.5% and 50. 50.75-32.5% respec- tively. The attribute baselines are stronger than the embedding-basse
Precision @ =
Figure 8: Retrieving instances based on query skill is far more effective when utilizing skill an- notations than using direct representations of each instance, as done in prior work.
approaches, though ob- taining attributes is also more expensive, The embeddding based baselines struggle, reflect- ing that frozen-embedding models may prior- itize encoding surface-level information over skills, skills may only become more apparent when inspecting the solution to the instance. We also find that searching over a broader corpus benefits the skill-based approach far more than either embedding-baad based approach. In summary, while allowing for direct searching over corpus, these skill-based retrieval experiments underscore how our skill annotations offer novel signal, enabling the grouping of instances in ways that can be complementary to prior methods.
Finding 5: Skill annotations open the door to custom evaluation sets via skill-based re- trieval. Similarity of attributes or direct embeddings for two instances does not reflect the similarity of the underlying skills tested by each instance, leading to poor skill-based retrieval performance (figure
AA multimodal instance consists of one image along with text question.
9 We obtain surface-level 'tags': the type of image, subject of the question, etc. See App.
As in Moayeri al. 2024a), find top-3 averaging be superior to top-1 or full averaging.

---

Published as a conference paper at ICLR 2025
6 REVIEW OF LITERATURE
Prior efforts to gain fine-grain insights often involve intensive manual efforts to construct multi- task benchmarks (Liang et al. 2023; Wang et al., 2019] Srivastava et al. 2023; Al-Tahan et al.)
2024 Balachandran et al. 22024) datasets with annotations beyond ground-truth truth labels (White) ct al. 2024| Fu et al. 2023 Yu et al. 2024b Lu et al. (024)). On top of quantifying specific abili- ties, extra annotations have illuminated biases (Buol amwini & Gebru 2018 Moayeri al. 2024b) and robustness issues (Idrissi et al. 2022 Koh et al., (021). Automated alternativss include synthe-
sizing benchmarks with known attributes Zhang et al. 2024 [Bordes et al.)
2023) automatically
uncovering sub-groups within existing data (Luo et al. 2024 Murahari et al. 2023). The term slice
The
discovery (or otherwise referred to as data cohort or subgroup) was coined (Chung et al., 2019]
Eyuboglu et al., (2022) to describe these methods which enable comparisons Eyuboglu et al. 2024
and error analysis (Rezaei al., [0244 Slyman al., 2023] Nushi al., 2018 [Singla et al. 2021
Dunlap et al. (2024). While these works focus on slices formed by surface-level attributes for visual recognition tasks, we curate skill-slices, which pertain to the underlying ability each slice requires from model, and as such extend to more tasks and modalities.
Skills are akin to the 'ability' tags present in some modern benchmarks (Liu et al. 2023 Ying et al.) 2024). though they can be finer (or coarser) grained (Whitel 1973) than what is typically anno- tated. The importance of acquisition of skills has been studied in human cogmition (Gagne 1962.
Koedinger et al.) 2023) as well as language models (Arora & Goyal 2023). Yu et al. (2024a) utilizes skills as imput to an LLM to generate challenging evaluation instances requiring a composition of the provided skills. Recent Murahari al. (023)) and concurrent Didolkar et al. (2024) work also
plore skill inference, albeit in more narrow domains, toward improved prompting or finetuning. This important shift of methods from attributes to skills indeed motivated by the increasingly general- purpose nature of state-of-the art models. In this work, we scale up such analysis and insights by expanding to 12 benchmarks and by introducing rationale parsing as a means infer more skills.
Rationales have been studied extensively in the context of prompting (Wei et al. 2022 Kojima et al., 2022
Yao et al. )023). Other works show the value of rationales as richer training signal (Hsich)
et al.
2023; Mitra et al., 2023] Zelikman et al.] 2022 Krishna et al. 2023) or model explanations
(Ehsan et al.
2019: Hu & Yu |024)|uang et al. 2023), ) though some question their faithfulness to
underlying model processes (Madsen et al. [2024] Fayyaz et al.) (024). We utilize rationales not to interpret a model directly, but instead to improve the relevance and diversity of the annotated skills
Singh et al. 2024), which become apparent in the solution steps present in rationales.
7 DISCUSSION AND FUTURE WORK
It is often said, "We can only improve what we can measure." As models grow in complexity and capability, traditional evaluation metrics like aggregate accuracy over entire datasets become insuffi- cient. Simply averaging accuracy across datasets hides important insights needed to understand and enhance both the datasets and the models themselves.
To address this gap, our work introduced a skill-slice analysis approach for model evaluation. By examining mooeel-generated rationales, we developed an automated method to extract the underlying skills required to solve individual evaluation instances. This shift from focusing on overall accuracy to conducting a more detailed analysis provides a more granular picture of model's strengths and weaknesses. Furthermore, our reliance on strong model help humans understand evaluation data may prove to be increasingly pertinent as models get strong that only a small handful of people will be qualified provide manual annotations on evaluation instances. In this sense, our work relates to scalable oversight (Bowman et al. (022),, providing new avenue to evaluation leveraging human-Al collaboration to understand models that will encapsulate advanced skills and knowledge.
Looking ahead, our approach can assist in creating more focused, skill-based evaluation datasets, by accelerating the process of discovering skills where models are collectively deficient or exhibit disparate performance. Our skill-based retrieval method can play pivotal role in generating valu- able training data for model development. By concentrating on specific skills, one can create more effective training sets that address identified weaknesses, thereby accelerating the advancement of Al models. This aligns with the principle that meaningful progress is achievable when we have clear insights, reinforcing the notion that can only improve what can measure.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENTS
We sincerely thank Siddarth Joshi, Alessandro Stolfo, and Natasha Butt for invaluable conversations and feedback throughout this project.
This project supported in part by grant from NSF CAREER AWARD 1942230. ONR YIP award 800014-22-1-2211 ARO's Early Career Program Award 310902-00001, Army Grant No. W91 19F2206,,,,7,, the NSF award CCF2212458, NSF Award No. 2229885 (NSF Institute for Trustworthy Al in Law and Society, TRAILS), an Amazon Research Award and an award from Capital One.
REFERENCES
Haider Al-Tahan, Quentin Garrido, Randall Balestriero, Diane Bouchacourt, Caner Hazirbas, and Mark Ibrahim. Unibench: Visual reasoning requires rethinking vision- Asioonlanguage beyond scaling,
2024. TRR..https:////
abtrg//b///// /240///..//2220002 04810
Anthropic. Introducing claude 3.5 sonnet.
htcpss//ww.anttrrpicc..co/// claude - 3-5---55-00---- 3--sonnet] 2024. Accessed: 2024-09-18.
Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models, 2023. htLlattts://arxtii orq abs
Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James WoffindenLLuey. and Safoora Yousefi. Eureka: Evaluating and understand- ing large foundation models, 2024. URL https: http://aaivvv.... abs /24/9/4099..mm 10566]
Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari S. Morcos. Pug: Photorealistic and semantically controllable synthetic data for representation learn- ing, 2023.
Sam Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, John Kernion, Jamie Kerr, Jared Mueller, Jeff Ladish, Joshua D. Landau, Ka- mal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noem'1 Mer- cado, Nova Dassarma, Robin Larson, Sam McCandlish, Sandip Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom B. Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfiel-Doodds Benjamin Mann, and Jared Kaplan, Measuring progress on scalable oversight for large language models. ArXiv, 1..0540,, 2022. URL sttp:::/api..emmntta http://o.....ooooooooooooo//a//.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods Natural Language Processing (EMNLP), pp. 632-642 2015.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.), Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pp. 77-91. PMLR, 23-24 Feb 2018.
Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. Slice finder: Automated data slicing for model validation. In 20/9 IEEE 35th International Conference on Data Engineering (ICDE), pp. 1550-1553, IEEE, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemy Tworek. Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14400 14168, 2021.
11

---

Published as a conference paper at ICLR 2025
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of Ilms: An exploration in mathematical problem solving, 2024. URRpttss:///xx abs/2405. 2205
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences image sets with natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2024.
Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O. Riedl. Automated rationale generation: technique for explainable ai and its effects human perceptions. In Proceedings of the 24th International Conference on Inteligent User Interfaces, IUI '19, pp- 263-274. Association for Computing Machinery, 2019. doi: 10.1 00.1145/301227533003311
Sabri Eyuboglu. Maya Varma, Khaled Saab, Jean- Delbrouck, Christopher Lee-Messee Jared Dunmmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross- modal embeddings, 2022.
Sabri Eyuboglu, Karan Goel, Arjun Desai, Lingjiao Chen, Mathew Monfort, Chris Ré, and James Zou. Model changelists: Characterizing updates to ml models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAceT '24. Association for Computing Machinery, 2024.
Mohsen Fayyaz, Fan Yin, Jiao Sun, and Nanyun Peng. Evaluating human alignment and model faithfulness of Ilm rationale, 2024. URL BtLlptts:://aa: http::///xxivvorr.. abs /2407 00219
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Robert M. Gagne. The acquisition of knowledge. Psychological Review, 69(4):355-36) 1962. doi: 10.1037/H0042550.
Gemini-Teamt Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text, 2024. URL htttps://aaxiv....r httpr//abb2243..000
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Cheng- Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujüi, Alexander Rat- ner, Ranjay Krishna, Cu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes. arXiv preprint 1iXiv:3005.02000 2023.
Shuaibo Hu and Kui Yu. Learning robust rationales for model explainability: A guidance-based approach. Proceedings of the AAAI Conference on Artificial Intelligence, 38, 2024. doi: 10. /Rthhttsss:////s..seeet .org indeiindxx... http://ntexxxpppeeeeeeepe http//AAA//rttic/
view/29783
Shiyuan Huang. Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H. Gilpin. Can large language models explain themselves? a study of Ilm-generated self-explanations, 2023.
abtp//bs//233 11207
Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nico- las Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim. Imagenet- x: Understanding model mistakes with factor of variation annotations, 2022. URL https:
//arxiv.org htt.aa/////...//22211
Kenneth R. Koedinger, Paulo F. Carvalho, Ran Liu, and Elizabeth A. McLaughlin. An astonishing regularity in student learning rate. Proceedings of the National Academy of Sciences, 120(13): e2221311120 1220,, 2023.
12

---

Published as a conference paper at ICLR 2025
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pp. 1199-22213, 2022.
Satyapriya Krishna, Jiaqi Ma, Dylan Z Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. Post hoc explanations of language models can improve language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openreview. wwt.:nett///mee.. detmmmmdd=3373733 3i UEv
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 23244, 1998.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
Haotian Liang, Zhe Gan, Linjie Li, Jingjing Zhang, Jason Baldridge, Ce Liu, Jianfeng Yang, and Jianfeng Gao. Eyes wide shut? exploring the visual shortcomings of multimodal llss. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr. Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha Niladri Chatterji. Omar Khat- tab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas leard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of lan- guage models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL Certification.
htw.rrewwww.eet/oooournddddo...d 1 Featured Certification, Expert
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. MMC: Advancing multimodal chart understanding with large-scale in- struction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics, 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? vrXiv:2307.0628// 2023.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Yulin Luo, Ruichuan An, Bocheng Zou, Yiming Tang, Jiaming Liu, and Shanghang Zhang. Llm as dataset analyst: Subpopulation structure discovery with large language model, 2024. URL
https::/arxiv..o wwtp:/iiwwr//as///oo///ooa 2405. 02363
Andreas Madsen, Sarath Chandar, and Siva Reddy. Are self-explanations from large language mod- els faithful? In Findings of the Association for Computational Linguistics ACL 2024, 2024. URL https://act antho Sacholloge ogy. f ndi addings-aci.
13

---

Published as a conference paper at ICLR 2025
Rui Meng*, Ye Liu, Shafiq Rayhan Joty. Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedddngg2: Advanced text embedding with multi-stage training, 2024. URL https:
//huggingfaace. Ste/Salesfoon / SFR- Bn--Eeddeding22.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: benchmark for general Al assistants. In The Twelfth International Conference on Learning Representations, 2024. URL https::/openreett httpee/wwwwweeeeeeeeeeeeeeeeeeee Forum?
Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984, 2020.
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes Ribeiro,
Sahaj Agrawal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal,
Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadal-
lah. Orca-2: Teaching small language models how to reason. arXiv, Novem-
orca- 2- -etahching small 11 anguage-mode wttp://ruu/eeeeeer/o/////lceaee
ber 2023. URL https: /tt/://wwwww. www. www.osoott...mmmmmmmmmmmmmww...t
hww- reason/
Mazda Moayeri, Michael Rabbat, Mark Ibrahim, and Diane Bouchacourt. Embracing diversity: Interpretable zero-shot classification beyond one vector per class. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 2302-2321. Association for Computing Machinery, 2024a.
Mazda Moayeri, Elham Tabassi, and Soheil Feizi. Worldbench: Quantifying geographic disparities in Ilm factual recall. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAceT '24. Association for Computing Machinery, 2024b.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark. arXiv preprint arXiv:2210.07316, 2022. doi: 1048550ARRXW221000731.
httt:http//aaaaa wwwpaaiwwww..gagaaaa attg//bs/222// 07316
Vishvak Murahari, A. Deshpande, Peter Clark, Tanmay Rajpurohit, Ashish Sabharwal, Karthik Narasimhan, and A. Kalyan. Qualeval: Qualitative evaluation for model improvement. ArXiv,
abs/2311.02807, 2023. URL https::/ semant sentticccolars wwwpoolarrrrr/ooouuuooeee 265033603
Besmira Nushi, Ece Kamar, and Eric Horvitz. Towards accountable ai: Hybrid human- machine analyses for characterizing system failure. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 6, pp. 126 135, 2018.
OpenAl. Hello gpt-4o. https://oppen http://ooppaa..coo/o.oooooo cww from/index/lne 1aaeello-pppp 1o-gpt-4oo 2024. Accessed: 2024-09-18.
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Dono- van Ong. Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua. Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng. Cyprien de Masson A'tumme Dani Yogatama, Mikel Artetxe, and Yi Tay. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.02287, 2024.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3170-3180, October 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: ///psii///xxv.. org/ 1908. 10084
14

---

Published as a conference paper at ICLR 2025
Keivan Rezaci, Mehrdad Saberi, Mazda Moayeri, and Soheil Feizi. PRIME: Prioritizing inter- pretability in failure mode extraction. In The Tweifth International Conference on Learning Rep- resentations, 2024.
net wtt.://eeeeoooneeennnroaeoa
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the of large language models. ArXiv, abs/2402.0176. 2024. http:: //api. /emant
www.stemantiio. i www.chcchca....o..
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12853-12862 2021.
Eric Slyman, Minsuk Kahng, and Stefan Lee. Vislice: Interactive vision-and-language slice discov- cry. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp- 15291-15301, October 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alone Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray. Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, An- ders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, An- drew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Her- rick, Avia Efrat, Aykut Erdem, Ayla Karakas. B. Ryan Roberts. Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski. Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch. Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro. Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Gar- rette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, De- nis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Eka- terina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Eliza- beth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Ev- genii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo. Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch- López. Gregor Betz, Guy Gur-Ari Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannanch Hajishirzi, Harsh Mchta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hong- ming Zhang, Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Ji- acheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Orallo. Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishhan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chia- fullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Contreras-Ochando, Louis Phillpph Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He. Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Fa- rooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria
15

---

Published as a conference paper at ICLR 2025
Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduma Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michat Swedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng. Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar El- baghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang. Pe- ter Eckersley, Phu Mon Huut, Pinyu Hwang. Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mci, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Ro- man Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang. Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang. Sahib Singh, Saif M. Moham- mad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter. Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghaz- arian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann. Sebastian Schus- ter. Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upad- hyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Des- bordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Rau- nak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang. Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. Transactions on Muchine Leurning Research, 2023. https: wttpor///ppeeeevie.....om.. net/
ISSN 2835-8856. URL
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: a stickier benchmark for general-purpose purpose language understanding systems. 2019.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://cpeereviie .et. worpmrod=uimnNINIIII
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024. URL https: / arxiv. org abs/ 2406. 01574
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL httptt/sss//opeeeeeeee..... http//ppeeeeeeeeeeeeeeeeee net/
Colin White, Samuel Dooley. Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein,
16

---

Published as a conference paper at ICLR 2025
Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination- free 1lm benchmark, 2024. (RLhttps:// wrwia. www.v.orgaab.//24444.. 1931
Richard T. White. Research into learning hierarchies. Review of Educational Research, 43(3);
361-375, 1973.
ISSN 00346543, 19351046. URL http: /www. http////wwoooor///tttbiiiieeee
1170110
xAL Grok 1.5v: Understanding ai's latest leap, 2023. URL
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openzeview.o wwt. www.ee/oooro.ddeeeeeeeee....
Kaining Ying, Fanqing Meng. Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang. Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision- vision-language models towards multitask agi, 2024.
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohent Anirudh Goyal, and Sanjeev Arora. SKILL-MIX: a flexible and expandable family of evaluations for AI models. In The Twelfth International Conference on Learning Representations, 2024a. httphhttpss//ppeeeeeeeee
net//Eorumiicd-JEfspp....
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR, 2024b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why languae models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023. 1d==RRUvxh8uax
net forum??
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL httlttta//pppeeeeiieet net/ Eorum? id= 3ELRdq2sqI
Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. arXiv preprint arXiv:2406.11775 2024.
17

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
(somewhat surprising) key hypothesis our work hinges on that models can articulate relevant skills, even for evaluation instances that they fail answer correctly. While we empirically validate this claim for (a subset of) the skill annotations we present, it is possible that on harder benchmarks, the listed skills may no longer be accurate. Thus, we recommend always employing at least our post-hoc verification when collecting skill annotations on new set.
Perhaps what is more likely is that while listed skills are relevant, they may be incomplete. Note that we do not present evidence for completeness in the main text. High overlap between two skill annotators promising sign (see Appendix [C.2). the fact that overlap not perfect suggests some skills may be missed. Moreover, most skills can always be broken down into smaller skills, making completeness perhaps an unnecessary objective.
Lastly, generating rationales is more expensive than directly prompting strong model to list skills, or simply using encoder to embed an entire instance at once. show rationales have some ad- vantages compared to each of these approaches in Appendix C.I and Section respectively though the added cost of our method is worth mention.
B COMPOSITION OF SKILL-SLICES IN OUR STUDY
Here, we enumerate the datasets we compute skill annotations for, as well as provide more details on the resultant slices form. All annotations and slices, which we call the Skill-Index, will be released to the broader community to enable skill-level analyses on models of their choice. We will also release code easily add the Skill-Index.
Datasets. We include datasets from 12 benchmarks in our study, consisting of multimodal (image and text) datasets and language-only dataset. These datasets are:
MMLU Pro, language-only benchmark by Wang et al. (2024). intended to be harder version of MMLU (Hendrycks et al. 2021
MMMU, multimodal benchmark with college level questions from many academic sub- jects intended to test 'expert' AI, by Yue et al. (2024)
Math Vista, a mathematics visual undersanding benchmark by Lu et al. 2024)
MMC, a chart understanding multimodal benchmark by Liu et (2024)
MMVP, benchmark specifically focusing on failure modes of VLMs, by Liang et al.
(2024)
Many general multimodal benchmarks testing numerous abilities:
MMBench, by Liu al. 2023
-MMTBench by Ying et al. (024)
MME, Fu et al. (023)
MMVet. by Yu al. 2024bb
SEEDBench, by Li
(2023)
Realworld-QA, a benchmark that claims to test many realworld visual understanding questions, by XAI (2023)
VibeEval, by Padlewski et al. 2024) (also referred reka vibe, as it was produced by the company Reka).
The two largest datasets in our benchmark suite are SEEDBench and MMLU Pro, which have 14k and 12k instances respectively. For example instances from some of these benchmarks, along generated rationales, see Appendix
We primarily sought out multimodal benchmarks, many detailed benchmarks exist for language- only tasks. We aimed to select popular benchmarks by prioritizing benchmarks that were featured in recent reports for model releases those tha had large number of recent downloads hug- gingface. We stress that benchmarks can easily be added to our corpus - our experiments suggest prompt is effective in diverse settings.
18

---

Published as a conference paper at ICLR 2025
Forming skill-slices. To consolidate the 128k unique total skills extract, perform tight clus- tering over embeddings of the skill names. We use the EPRREmbedding-22
model (Meng* et al, 2024). as it the leader on the huggingface MTEB leaderboard at the time (Muennighoft et al.
We utilize the fast clustering from sentences canceeta ranns rans innforrre formers (Reimers & Gurevych, 2019 with minimum similarity threshold of 0.95, intended to de-duplicate skills. One could use a lower threshold, though in that case, naming resultant skill cluster would be more challenging simply take the skill name in the cluster that closest the centroid, though we qualitatively observe nearly all skills in the cluster have very similar names. To be sure we have completely de-duplicated, we run the clustering algorithm until all but at most 0.5% of cluster names have similarity below 0.95.
2022
Throughout our paper, we use skill-slices with least 100 instances, though this number is arbitrary. Picking a lower threshold increases slice count (often including finer-grained slices), but accuracy over theses slices may be less reliable.
C DETAILS ON SKILL EXTRACTION AND VALIDATION
We present the prompt used to generate rationales below. It features detailed instructions and an in-context example. We generate this prompt by iteratively asking ChatGPT to refine the prompt based specific desiderata, along with manual tweaks along the way. The in context example is also generated by GPT-4o; it was produced using an earlier version of the prompt, and then modified slightly to correct some errors. Note that by GPT-4o, we always mean GPT-4o 2024-05-13.
Prompt to generate rationales (part 1)
System Prompt: Detailed Step-by-Step Response with Skills Labeled
Objective: List the skills and evidence for each step the given question. responding to each question in detailed, step-by-step manner, where each step utilizes only single skill. Clearly state the skill being used and the evidence applied at each step. Guidelines:
1. Single Skill Per Step:
step involves only one skill. Skills categorized tiple names, each getting more specific. For example:
Knowledge:
Music Theory, Scale Identification, Reciting F Major scale
Biology, Photosynthesis Process, Calvin cycle
History, World War II Events, Battle of Stalingrad
Perception:
Visual Recognition, Musical Note Identification
Chart understanding, information extraction, determining the length bar bar chart
Visual understanding, Spatial reasoning. Understanding depths of ob-
jects
Auditory Recognition. Sound Identification, Recognizing child's
voice
Video Analysis, Episodic memory, Remembering where object
placed
Reasoning:
Logical Deduction, Process Elimination
Mathematical Calculation, Algebraic Manipulation, Resolving fractions
2. State the Skill: Clearly identify and state the skill being used in each step.
pieces of relevant evidence:
Specify the evidence used in each step. This could include:
Information from the question (e.g.. specific phrase).
Duta or findings from previous steps. List these findings clearly if they are required contest for the claim of the step!
Regions in an attuched image, applicable, clearly stating WHERE the imnge the information being extracted from.
4. State the Conclusion of the Step-
5. List your final matted as ANSWER: O
after your step-by-step explanation line for-
Here is an exumple of the desired response structure given a question.
19

---

Published as a conference paper at ICLR 2025
Prompt to generate rationales (part 2)
Example Question: "Based on the following image, can you determine what the two sounds
belong to? jimage
A. #F major scale
B. c melodic minor scale
C. harmonic minor scale
D. melodic major scale"
Example Response Structure:
Step Recognize Type Image
Skill: Perception: Visual Recognition, Image Classification, Sheet music recognition
Evidence: The image has lincs and symbols, which resemble sheet music.
Conclusion: The image is of shoet music.
Step Identify Clef
Skill: Perception: Visual Recognition, Symbol Identification. Treble clef recognition
Evidence:
There is staff' in the middle the image, and on the left, there is treble clef.
Conclusion: The music shown on treble clef.
3. Step 3: Identify Symbol Positions from the Image
Skill: Perception: Visual Recognition, Spatial reasoning, Locating musical
symbols
present middle image: with sharp in the second space from the bottom, and one in the space above the line. The notes F# and A.
Evidence:
Two
Conclusion:
Step Read Provided Options
Skill: Knowledge: Reading Comprehension. Understanding the question
Evidence:
The provided options written after the question. minor scale,
Conclusion: options Fi major scale, monic minor scale, and E melodic major scale.
Step Eliminate Incorrect Options
Skill: Reasening:
Logical Deduction, Process of Elimination
Evidence:
notes shown are and so they must be correct scale,
F# major includes F#
melodic minor does include Fw.
harmonic minor does not include F#. melodic major is not standard scale.
Conclusion: melodic minor. b harmonic minor, and E melodic minor are incorrect.
Step Conclude the Correct Scale
Skill: Reasoning: Logical Deduction, Conclusion
Evidence: F# major scale the correct answer, it includes both F#t and A.
ANSWER: A
Final Note: Ensure that the skills are described in detail: LIST AT LEAST THREE NAMES FOR EACH SKILL..
C. ABLATIONS ON RATIONALE PARSING FOR EXTRACTING SKILLS
We now perform an ablation study to compare how skills inferred by rationale parsing (our pro- posed method) differ from those obtained by directly prompting strong model to list relevant skills. Specifically, the 'direct prompting' alternate method consists of presenting an evaluation in- stance along with the prompt "List skills that are relevant to the given question. Do not answer the question, only provide the list skills. Be detailed, but only use short specific phrases throughout your response only use few words per list item. I will provide you with an example first." As the prompt suggests, we additionally provide an in context example, primarily to ensure standard response format to facilitate automatic parsing of the skill annotator model's 's outputs. We compare the annotated skills produced by GPT-4o over 1 1195 instances from all 12 benchmarks (roughly 100 per benchmark) using direct prompting and rationale parsing.
20

---

Published as a conference paper at ICLR 2025
Method
Method
0.20
Fraction Inferred Skills
0.125
Fraction of Instances
Direct Prompting
Direct Prompting
Rationale Parsing
0.100
Rationale Parsing
0.15
0.075
0.050
0.05
0.025
0.00
0.000
0
10
20
30
0
10
100
1000
10000
# of Skills Instance
Slice Size for Inferred Skill (log scale)
Figure 9: (left) Rationale parsing results in significantly higher number of annotated skills per instance than using direct prompting. (right) Skills inferred with direct prompting are generally finer-grained, resulting in higher fraction of very small slices (e.g. with < 10 instances), and lower fraction of all other slice sizes (see table for exact numbers).
Slice Size
[0, 10)
[10, 25)
(25,50) 50)
[50, 100)
[100, 250)
[250, 1000)
2 1000
Skill Inference Method
Direct Prompting
2494
525
341
318
288
185
123
Rationale Parsing
2399
560
451
409
491
290
164
Table 1: Comparison of slices formed by skills inferred via direct prompting vs. rationale parsing. Direct prompting results in a higher number of skills that are very specific to the question, resulting in more slices with less than 10 instances than rationale parsing. Aside from this hyper-fine grain, rationale parsing results in more slices at every level of granularity, with largest improvements in medium grain slices. Details in section C.I
First, we observe that rationale parsing results in a substantially higher number of unique skills annotated per instance than direct prompting. The left panel of figure shows the distributions of number of annotated skills per instance with the prompting methods. On average, we obtain 9.98 skills per instance with rationale parsing, a 58% relative improvement compared to the average of 6.32 skills per instance with direct prompting. We note that for about 4% of instances, direct prompting leads to 0 inferred skills because the skill annotator directly answers the instance instead of listing the relevant skills. In the rationale parsing method, the skill annotator both answers the given instance and list skills simultaneously, alleviating issues with conflicting instructions.
Next, we seek to compare the granularity of inferred skills from the two methods. Recall that there is no single 'optimal' gramularity: amer-erained skills are more descriptive, while coarse grained skills result in larger skill-slices, which in turn improve the reliability of average accuracy taken across such a slice. Thus, ideally, we can obtain skill annotations of various granularities. However, skills that are exceedingly fine-grained cannot be used, as skill-slices consisting only a few instances may not have sufficient sample size to offer reliable measurements of model proficiency.
To proxy skill's granularity, we count the number of instances in our corpus annotated (via aetionale-parsing with a highly similar skill (i.e. similarity of at least 0.95 between text embed- dings of the skill names). Figure | (right) and table [ show the distributions of the # of relevant instances per skill inferred via direct prompting and rationale parsing. First, we observe a larger fraction of skills inferred with direct prompting to be relevant for a very small number of instances, suggesting that many of the skills inferred by direct prompting may be too fine-grained to re- sult in sufficiently large skill-slices. At other levels of granularity, rationale parsing consistently yields more skills for each grain. For example, 491 skills inferred with rationale parsing have be- tween 100 and 250 relevant instances, while only 288 skills inferred with direct prompting have the same number of relevant instances
reduction of 41%. We note that the improved diversity of grain of inferred skills using rationale parsing vs. direct prompting may in part be attributed to our instruction to list skills with multiple names of cascading granularity.
21

---

Published as a conference paper at ICLR 2025
Agreement GPT.40
Skill Verification
0..5
Second Verifier Model
claude-sonnet
gemini-1 5-pro
KDt-4v
0.0
mmmop
vibe
sendbench
Dataset
Agreement GPT GPT-4o TT-40
Skill Annotation
0.75
0.50
0.25
Second Annotator Model
claude sonnet
gemini 1.5 pro
EPT
0.00
mimpp
realworld.qa
mmlu.pro
Dataset
Figure 10: Diverse models agree when verifying the relevancy of an annotated skill (top) and anno- tating skills given instance (bottom).
In summary, our proposed rationale parsing method results more annotated skills per instance and greater diversity granularity of annotated skills, when compared to direct prompting, Also, we qualitatively observe both methods to yield skills that are overwhelmingly relevant to the given in- do not expect much difference the relevance
stance do not quantitatively test this facet, of skills annotated by either method. Another aspect worth noting is that rationale parsing enables skill localization, which offers second, complementary avenue to corroborate and refine our skill slice findings (see sections 4 and E.3
We emphasize that better methods to extract skills certainly may exist. Here, we present one method and demonstrate the utilities it may hold. Many of these utilities can be extended and potentially expanded by improving our skill extraction method. Having demonstrated that skill inference is possible and useful, we leave further iteration skill inference and its applications to future work.
Lastly, we again note that rationales are leveraged not to directly gain insight on the rationale- generating model, but instead to annotate evaluation data, with which any model's proficiencies can studied more closely. This distinction important as it side-steps the potential concern of rationale faithfulness. While the rationale-generating model may not be faithful in its written chain of thought, we are solely concerned with whether or not the skills listed this chain of thought relevant to the underlying evaluation instance. Thus, validation focuses understanding relevance of skill annotations to data, and not to hidden model internal mechanisms.
C.2 DETAILS AUTOMATIC VALIDATION
We now detail second automatic validation approach: inter-(skillaamnotttr agreement. As the name suggests, we simply extract skills by feeding the same prompt to new rationale-generating try Claude 3.5 Sonnet, Gemini 1.5 Pro, and GPT-4 Turbo 2024-04-09. We compute agreement between two sets of skills listed by different models as the overall fraction of skills such that match exists in the other of skills. Here, we define two skills being match their text embedddings have cosine similarity of at least 0.85. Note that this is also the threshold we set for T in sampling 'negative" skills for post-hoc verification. This threshold lower than our clustering threshold because in this case, we do not want to admit paraphrased versions of the same skill, where as in our clustering, we only wish to ensure extremely high similarity within each slice, since it ultimately only takes on name. Figure 10 shows the inter-annotator agreement, both across different skill annotators (bottom) and verifiers. For verifiers, agreement is very high (> 0.9) across the board, while agreement is slightly lower for skill annotators. We believe this may relate to the fact that our skill discovery procedure may not result complete list of skills, which we note in section A We argue precision (skill relevance) is more important than recall (completeness), as
model. Namely,
22

---

Published as a conference paper at ICLR 2025
Dutaet Instunes))
SEREDBecch (14252)
MMLLU (12002)
MMBBesc (4329)
MMTBeash (5123)
MME
MMC
Mathista (1000)
MMMM (900)
ReulWodd
ViteEval (269)
MMVer
(2374)
(215)
Model
Genini 1.5 Pho Claude 2.5 Sonnet
69.94
59.47
84.18
02.36
78,64
75.40
56.19
56.33
68.10
74,00
36,43
60.55
69:43
58.49
$4,50
65.17
$4.33
82.27
48.60
58.67
71.76
65.00
44,98
59.63
GPT 4o
70.62
30.09
$7.46
64.52
82.73
77.66
50.70
00:44
75.95
35.62
$9.81
02.54
Random choice Skill Routing (cur))
99.99
58.02
85.36
64.03
81.90
78.64
$1.80
58-48
71.94
74.78
43.74
61.00
71.81
62.94
87.13
65.42
85.89
82.13
55.50
58.59
26.34
80.82
49.07
59.63
Table 2: Per-dataset accuracies for each of the three foundation models study, as well as accu- racies achieved when randomly selecting a model per instance or when routing each instance to the model strongest on the relevant skills.
omitted skills do not affect the veracity of skill-slice accuracy (so long as they are omitted randomly), while incorrect skills do corrupt the signal.
D DETAILS ON ROUTING PROOF OF CONCEPT
We route each instance independently, based skill-slice accuracies computed over the remaining corpus (i.e. everything except for the current instance). For an instance, we first take all of its listed skills, and then remove any for which we do not have a skill slice with least 100 instances on the remaining corpus. If no skills remain, we default to Claude 3.5 Sonnet, which is (barely) the best model over the entire corpus. For the remaining skills, we consolidate accuracies over those slices to obtain single score per model via a weighted average. Namely, the weight for each skill is the inverse of the number of instances in that skill slice. This way, more specific (i.e. lower frequency) skills for the instance carry higher weight.
This routing protocol very simple and does not account for numerous sources of variance, such as different levels of difficulty across instances for the same skill, and how skills can be harder when applicd in the context of other skills. Nonetheless, we see accuracy improvement using our simple scheme, suggesting that skill-slice insights generalize.
To provide a full breakdown of the effectiveness of our simple approach, we present the per-dataset results in table 2 We show accuracy per dataset for each of the three foundation models in our study. Also, we show the accuracy obtained with skill routing (our method) or by selecting a model random in the latter case, we show the expected accuracy. For 11 out of 12 datasets, skill routing yields higher accuracy than random model selection, with the sole exception occurring for the smallest dataset in our study (where accuracy measure may not be as reliable due to the smaller sample size). The accuracy gains exceed 3% for 7 of the 12 benchmarks studied. Compared to the best model over our entire suite (Claude 3.5 Sonnet), skill routing improves accuracy for 10 out of 12 benchmarks, with a tie in one benchmark. GPT-4o, however, beats skill-routing for 5 benchmarks, though three of these are the smallest benchmarks in our study.
Overall, we find that our simple approach to skill routing offers accuracy gains under many settings. We interpret this as positive signal in support of the generalizability of skill-slice insights, as well direct potential application of our analysis to improve performance. As mentioned above, the simplicity of our scheme may limit its effectiveness. Potential avenues to improve this approach include considering accuracy over slices formed by multiple skills used together, perhaps by training a lightweight router to predict the best model given skill annotations (akin to an interpretable mixture of experts). We leave these explorations to future work.
E DETAILS ON PROBE CONSISTENCY ANALYSIS
We now provide complete details for our probe consistency analysis.
E. GENERATING PROBING QUESTIONS AND CHECKING CONSISTENCY
To generate probing questions and check consistency, we require multiple calls to an LLM we use GPT-4o. Importantly, neither task requires multimodal LLM,, thanks skill localization.
23

---

Published as a conference paper at ICLR 2025
First, we rephrase claim linked to skill of interest with the below prompt. Notice that form probing questions in a few slightly varied way, including two yes or no questions testing contra- dictory questions. Qualitatively, this increases the chance that model contradicts itself when it is unsure, as opposed to giving the same exact incorrect answer (which our consistency analysis can- not detect). We omit prompts for answering probe questions and checking consistency, these straightforward and can be viewed in our code release.
Probing question generation
I will provide a specific skill excerpt from step-by-step solution some problem. Isolate the central claim that best pinpoints the given skill (next SKILL TO PINPOINT). Then, state claim and generate three questions that can be used verify the claim, using the following strategy:
Q1 should repeat the claim
no question.
hut contradictory claim, tion. Do not use regations.
Q2 should create
as yes no ques-
Q3 should be open-ended, whose answer is contained in the claim alone.
ANSWER IN THE FORM OF PYTHON DICTIONARY. as shown in the following example.
Example Response Structure:
SKILL TO PINPOINT: identifying the longest bar EXCERPT: Step Identify the Top Companies from the Image
Skill: Perception:
Visual Recognition, Bar chart unalysis, Idemtifying the longest
Evidence: In the chart, the two companies with the longest hars are AG Insurance and AXA.
Conclusion: AG Insurance and AXA are the companies with the highest market shares.
CLAIMAND PROBE QUESTIONS:
"CLAIM": "The companies with the lengest bers in the chart are AG Insurance and AXA.", ",
*Q1*: "Are the two companies with the longest bars the chart AS Insurance and AXA?", *02*: *Are the two companies with the longast bars in the chart Allianz and AG Insurance?",
*03"; *Nhich companias have the longest bars in the chart?"
E.2 CORRELATION OF PROBE INCONSISTENCY AND SKILL-SLICE ACCURACY
In figure plot probe inconsistency rate vs slice accuracy directly and for each model separately, as opposed to the consolidated violin plot we show in figure 7 We observe reasonably high correlations for all three models, especially given that probe inconsistency rate can only be factor of 0.05, as our implementation checks 20 claims per probed skill.
E.3 COMBINING SLICE AND PROBING ANALYSES TOWARDS AUTOMATIC DIAGNOSIS OF SKILL DEFICIENCIES
Lastly, we briefly discuss an end-to-end pipeline for automatically diagnosing skill deficiencies. As shown figure [2] a two-stage pipeline can be engineered that first flags skills with low slice accuracy, and then probes those skills. These two stages are complementary in two ways: first, they admit different kinds of errors. we interpret a deficient skill 'positive', low skill-slice accuracy admit false positives due to the presence of highly co-occurring deficient skill. However, deficient skill will necessarily have low skill-slice accuracy. Probe inconsistency can help remove model will always answer consistently (i.e. correctly) to probe questions
these false positives, the model is proficient at a skill.
The order of these components is also motivated by the fact that probing requires many more LLM calls, and as such is more expensive than skill-slice analysis. Nonetheless, if cost is not a factor, this procedure could potentially uncover skill-level deficiencies scale.
24

---

Published as a conference paper at ICLR 2025
GPT-40 p
Gemini 1.5 Pro
Claude 3.5 Sonnet
0.662,, p
Rate of Inconsistency
over Probe Answers
0.6
0.4
XXV
X peor
0.0
0.1
1.0
1.0
0.4
0.8
Skill slice accuracy
Skill slice accuracy
Skill slice accuracy
lowest accuracy
X
average accuracy
highest accuracy
Figure 11: Rate of inconsistency over probe answers vs. slice accuracy per model per skill, as portrayed as violin plot in Figure IO, For all models, inconsistency negatively correlated with slice accuracy, with an overall Pearson's s r = -0.675.
Stage 1: fast, admits FP
Stage 2: pinpoints problem areas
Relevant localized skills per instance
Per instance, have model generate detailed rationale
Probing Qs per flagged skill
Auro Parsc
Lse LLM
Potential framework for automatic diagnosis of skill deficiencies, leveraging the comple- mentary nature of skill-slice analysis and probe inconsistency.
Figure 12:
F DETAILS ON SKILL-BASED RETRIEVAL
We use the same text encoder See--Emmeediiid - R) for any text features, including embedding skills and attributes. We use CLIP VIT-L/144 Radford al. (2021) for image features. To generate attributes, we prompt GPT-4o to "List attributes describing the content of given question. **Do not list skills needed solve the question** just attributes of the content". We additionally instruct GP to avoid answering the question we wish for it to attribute.
G PROMPT USED TO GENERATE MODEL OUTPUTS
We note that we employ a simple prompt in obtaining model outputs on the benchmarks we analyze. Namely, we ask models "Be concise. Write 'ANSWER: followed by your answer. multiple choices are given, only provide the correct letter." As is well documented, accuracies can be im- proved by engineering stronger prompts, including with model-specife strategies. We opt choose single standard prompt for all models, to avoid giving any single model an advantage due to unequal amounts of time spent on prompt optimization. We leave investigation of these for future work and study models in simple setting for now.
H QUALITATIVE EXAMPLES
We now present numerous qualitative examples so that readers may more easily comprehend the generated rationales and resultant skill annotations that form the basis of our method. Namely, show one example instance for each benchmark from a subset of our evaluation corpus, along with the corresponding generated rationale. For cosmetic reasons, we choose the instances with the shortest question length. Note that even for these shorter questions, rationales can be long. The skill annotations are written as the first bullet under the rationale step they pertain (or are localized) to, with each skill articulated with multiple comma-separated names of cascading granularity.
25

---

Published as a conference paper at ICLR 2025
Example rationale for MMVP instance
Step I: Identtffy the Duck in the Image
Perception: Recognition,
Idenciticttin, Identifynng
Evidenee:
The image shows a bird with recognizable features duck including a green head and distinct body pattern.
Conclusion: bird
indeed duck.
Step Locute the Duck's Beak in the Image
Skill: Perception: Visual Recognition, Feuture Idencification, Locating beak
clearly
the bottom right the image.
Conclusion: The dack's 's's beak is locaued.
Step Visibiiitt
Skill: Pereeption: Visual Assessmenr, Palliness Identification, Checking visibiiiyy of parts
Evidenee: The slape
duck's beak without any by edge image any
Are the buttorfly's wings cleser to being open
closed? Open Cloeed
Conclusion: The entire beak the duck visible in the picture.
ANSW ER:
Example rationale for MMC instance
Step 1: Recognize the type of content the Imaae
Perseptio: Recognition, Memiiiication, Cide Snippet Rocognition
image
appears
print ("Hello, C++!")
Conclusion: The content the image snippet of code.
Identify programming language
Skill: Know Knowledge: ledge: Programmingg Syntax Recognition, Python Syntax
Evidence: The code HHello, C++!"y uses the 'print function with parentheses and string quotes, which matches systax Python.
structure matches Pythoa syntox.
Step 3: Evaluate the print statement in Python
Knowlodge: Programaiing, Python Functions, Statement
Evidence: In Python, 'prit' statements are nsed to display text or vasiables, and the code snippet does exactly that with prans] Hella, Cea!"!:
a python code shown the picture?
Helll""Hlll,
Conclusion:
valid Python
ANSWER:
Example rationale for SEEDBench instance
Step 1: Recognize und Ideatity Relevant Ohjects
Pereeption: Recognition, Object Identification, recognition
Evidence: The image shows various objects within buthroom setting, including a sink. buthub, and items placed nearby.
Conclusion: Towels must be idcatifed within the bathroom
Step Locute Towels in the Image
Skill: Penseption: Recognition, Spatial Reasening, Identifying multiple objects
of therhhe sink, folded On the comer the buthub''s structurs, to the left of the sink, there stack of what lowels.
look like touels.
Conclusion: Towels are located near the sink and the corner of the buchrab structuee.
How the image?
Stop the Towels
One
Two
Skäll: Reasoning: Quanitasive Amalysis, Counting Clbjects
Three
Four
Evidence:
appenrs single composed foldee towels.
Comelusion: There are total of theee towels visible the image.
26

---

Published as a conference paper at ICLR 2025
Example rationale for RealWorld-QA instance
Step I: Recugnize the Type of Objects
Perception: Recognition, Ubject Ideutification,
recognition
Evidence: The objects are electronic devices.
Conclusion: The three objects are elecironic devices placed side by side.
Step Compare Object Heights
Skill: Perception: Visnal Assessmenr, Height Comparison
Evidence: The
appears
height compared
Comclusion: The object the middle is shorzer in height.
Compare Object Widths
Skill: Poreeption: Visual Aseessmast, Width Compurioon
Evidence: The object in the middle appears narrower in wichh compured to the ubjects the left right.
object
widh.
Step 4: Comppre Object Depths
Perception:
Assessment, Dopth Comparison
Which the objects the smallest?
A. The object the right the smallest object.
Evidence: The object in the middle appears to be thinser depth compared the abjects the left and right.
B. The object the left is the smallest object.
C. The object the middle the smallest object.
Coaclusion:
the middle thinner
Step Overall Size Comparison
Skill: Reasoning: Logical Deduccion, Appregation Dimensions
middle shorter height, narrower wilth, thinner deptt compared to the other two objocts.
Evidence: The objoct
Conclusion:
The object the middle is the amallest overal.
Example rationale for MMC instance
Stop Rerngnize the Type Image
Skill: Perception: Visual Recognition, Image Classification, Graph identification
Evidence: The image displays labeled plot with data, legondhitle ing is chart.
Participation rate sports activities (Belgium)
2018
Share of the Flemish citizens
Conelusion: The image is line chart.
07
Step Identify
06.
Skäll: Knowledge: Reading Comprehension, Understanding title and labels
65-
Evidence: The title reals "Purticipttion sports activities Flanders (Belgium) from
Comclusion: The chart about participution máe in sports activities in Flanders over time.
02-
Step Specific
(2014)
01.
Skill: Penception: Visuul understanding, Locuting puints on the graph
0..
Evidence: axis, locate the 2014.
2008 2010
2012
2014
2016
point comesponding
identified graph.
Response
Step 4: Compare the Participation Rate Around 2014
There vas signticant incresse something specific 2014. Please answer whether the description true or false according to the mage.
Perception: Analysis, Comparing values time
Evidence: Otserving the height the line at 2014 compared the previous year (2013)
and the subsequent year (2015).
Conclusion: The line shows rise from 2013 2014.
Step Deternane If There Was Significant Increase
Skall:: Reasoning: upward movement
Deduction, Ideatifying significant changes
along slight
wand/plateau from 2014 2015.
Conclusion: There noticeable increase from 2013 2014.
27

---

Published as a conference paper at ICLR 2025
Example rationale for MM instance
Step I: Underslund the Equation
Knowlodge: Algebm, Esquation understanding
(K+3)
Evidence: The given equation (a 23 4.
-4
need solve for Conclusion: understand
the
Stop 2: Take the Square Root of Bath Sides
Skäll: Reavoning: Algehnii manipulation, Solving
Evidence: To simplity (s 3) = 4, we take the square root of both sides.
Conelusion: v 33
implies a +
Sulve for Both Cases (Positive and Negutive Roots)
Skill: Reusoming: Algebraic munipulation, Solving lineur equations
Evidencr:
What & the equation?
Conclusion:
Sulving each
ANSWER:
Example rationale for VibeEval instance
Step Identify Ingredients
Skill: Pereeption: Visual Recognition, Food Identifictiion
Evidenee: The image
with noadles, tnft, hechs, vegetables (visi- bly distinct green gumishes and yellow accents).
Conclusion: The visible ingredients are noodles, tofa, herts (possibly parsley or cilantro), and some vegerables.
Step Identify
Non- Vegan Ingredients
Skill: Knowledge: Food Compositios, Typical Nao- Nan-Vogut Ingredients Pista Dishes
Evidenee: products sance), and cpgs (ia types noodles).
non-vegan
posta
include
Conclusion: There is no visible dairy, meat, or fish product the dish. The presence tofu suggests plant based protein source, but whether the noodles contain cannot be detemmited by sight.
Step 3: Cousider Potential Hidden Non-Vegan Ingredients
Skäll: Knowledger Vegan Requirements,
might contain non-vegan ingredients oyster sance, sauce, honey, or certain seasonings.
Evidence: Some
Conclusion: While totu and the visible vegetables vegun. the noodles sauce might still non-vegooni ingredients.
this safe vegan to eat?
Step Conclude Based on Visual and Common Knowledge
Reusoning: Logical
Drawing
Evidence: Given visible ingredients vegetables, berbs) absence any
ous animal products, the dish appears vegan. However, without confirming the ingredients the noodles and suuce, it's uncertain.
Conclusion: Visually, this appears he vegan-friandly,
confirm
ingrediests
entirely for vegan
28


Reviews:
[{"review_id": "otSfL4x958", "reviewer": "Reviewer_ubHn", "summary": "This paper uses LLMs to annotate benchmark instances with skills required to solve them. The authors validate their annotation in multiple different (mostly automated) ways. They also analyze how different models' capabilities vary when restricted to so-called skill slices, i.e. sets of benchmark instances that all require a specific skill to solve (according to the automated annotation).", "questions": "- What does figure 8 show? Is the last row's annotation a typo?\n- What happens when combining skill and text/image embeddings in figure 8?\n- What are the pros and cons of the verification approach based on negative samples?\n- Which skills and benchmarks does the routing approach improve most on (let's say compared to an ensemble, as well as \"perfect\" routing that has access to the ground truth)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing is generally quite engaging \n- Decomposing broad benchmarks into more specific ones seems quite useful for things like model selection, whenever models are to be deployed in a specific setting. \n- Despite the caveats listed in the weaknesses, the results on differences in model capabilities are quite interesting. The additional validation experiments, like the one on model routing, make me believe that there is indeed meaningful signal in the results.", "weaknesses": "- All results on comparing pairs of models would strongly benefit from adding error bars to show that the differences are indeed exceeding what would be expected from noise when comparing equally performant models. \n- Additional statistical analysis to show that the most stark model differences cannot be explained by random noise + testing many hypotheses (i.e. comparing models on many different skills) would also strenghten the results. \n- Releasing the annotations would be helpful for the reviewers to get a better intuitive picture of the validity of the proposed methods. \n\n- Nitpicks:\n   - The writing sometimes seems a bit too grandiose:\n       - For example, whether the paper is \"providing a valuable resource for the research community\" seems like something the community will decide on.\n  - The formatting in Appendix E is off (lots of empty space)\n  - Appendix G is incomplete (it ends mid-sentence)\n  - Some kind of face-value evaluation (such as a small scale human evaluation) beyond GPT-4 as judge would be useful for section 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ag0QF62N7O", "reviewer": "Reviewer_9gum", "summary": "This paper proposes a new method to evaluate LLMs by analyzing their skills rather than just relying on overall accuracy scores. It extracts detailed reasoning steps (rationales) from models like GPT-4 to identify the skills used, grouping these into \"skill-slices\". By doing this, the researchers can reveal strengths and weaknesses that are not revealed through traditional evaluations. They also introduce probing questions to validate the identified skills, ensuring that the analysis reflects the model's true abilities. This approach aims to provide a deeper understanding of model capabilities and guide improvements for future models.", "questions": "Given that models may hide internal process and not explicitly demonstrate all the skills they use, how would we evaluate those models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a new insight to evaluate LLMs by evaluating on fine-grained skill analysis rather than traditional scores. This method of extracting \"skill-slices\" from model-generated rationales allows for a deeper understanding of models’ capabilities. The insight is reasonable, and the meaningful for current LLM studies.\n- The design of probing questions allows for a robust validation for the extracted skills, which could reduce the risk of hallucination.\n- The paper provides extensive experimentations across multiple benchmarks and models. It clearly provides solid analysis and findings which might benefit for future studies.\n- The paper provides skill annotations, which would guide future model and dataset developments. \n- The skill-slices design could benefit for the explainability and transparency of large language models.\n\nIn general, it is a good paper.", "weaknesses": "- Real-world tasks often require a combination of multiple skills. The current method of skill extraction assumes that skills can be categorized and analyzed in isolation. This overlooks the fact that the interaction between multiple skills may be non-linear and context-dependent, meaning that combining skills does not always produce the same outcome as using those skills independently.\n- Some LLMs may operate as \"black boxes,\" where the true internal reasoning process is not fully transparent. This means that the skills inferred from the generated explanations may not align with the model's real decision-making processes.\n\nThe paper has already discussed some limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zDz83JXtlN", "reviewer": "Reviewer_VpcF", "summary": "This paper mainly studies the evaluation challenge from testing multiple skills in modern LLM evaluations. Given a specific benchmark dataset, the skill-wise performance is obscured and the possible hidden rich signal is usually underutilized. To alleviate this issue, this paper propose a automatic approach for recovering the underlying skills given any evaluation instance. In specific, this is implemented by inspecting model generated rationales. By examining the relevance of rationale-parsed skills and inferring skills for 46k instances over 12 benchmarks, it is shown that many skills to be common across benchmarks, resulting in the curation of hundreds of skill-slices. These skill slices are finally further shown to deliver novel insights on model trade-offs.", "questions": "- Assume in the industry, you are not allowed to access those strong models (e.g., GPT-4o). Given a specific LLM, the skill discovering ability is actually constrained by the skill ability of this LLM. How could you explain the effectiveness of automatic method, if let's say the model itself is weak in one category of skill while you expect this LLM can identify this skill and generate the corresponding rationales automatically?\n\n- Given the generated rationales will be treated as a golden standard for the future use in other model evaluations, how to guarantee the trustworthiness of such rationales? This is actually the same question listed in weakness part.\n\n\n- I am curious: How is the cost of conducting such kind of research?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- First of all, the LLM evaluation problem is very important, especially under the case where multiple or many skills are evaluated together. This has been shown by Figure 1 as well to clearly clarify the motivation of this work. \n\n- The paper is well written and easy to follow\n\n- Automatic discovering the skills related to any evaluation instance is an important topic.\n\n- Some findings are quite interesting. For instance, Gemini 1.5 Pro to be much stronger in math and science skills, like ‘computing molar mass’, while falling far behind for legal skills like ‘applying constitutional law’.", "weaknesses": "- The automatic verification protocol should be further explained. This is the single most important part of this method pipeline. From my understanding, querying LLM models or APIs with specific prompt engineerings can always get some sort of skills, rationales or chain of thoughts. The more important thing is how can you guarantee these generated content are properly validated. The author claimed that they will \"release all our rationales, skill annotations, and skill-slices, which we term the Skill-Index, to the public.\", but more explanation should be provided. The verified trustworthy accuracy here will be quite important.\n\n- The technique contribution looks a bit limited. All the work done here are PE related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62AGes5Hmv", "reviewer": "Reviewer_AkzK", "summary": "This paper proposes an automatic approach to infer underlying skills for evaluation instances by inspecting model-generated rationales. This helps in achieving a finer-grained understanding of model capabilities from existing benchmarks, increasing the interpretability of evaluating LLMs with benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The way of analyzing the model evaluation benchmark from the aspect of inferred skills is interesting and novel to me.\n\n- This work utilizes a post-hoc verification of skills and validates its validity by comparing the results with human validation.\n\n- The authors conduct experiments to show some interesting findings on skill generation and model evaluation.", "weaknesses": "Overall, the presentation needs improvement such as:\n- A simple prompt example used to generate a detailed rationale shown in Figure 2 would be better.\n- A clear subsection of used benchmarks and LLMs in the main paper.\n\nI am confused about the description of the experimental section such as:\n\n- Figure 8 is not clear enough at first sight. Which method uses skill annotations? And does the higher precision @ k = 20 indicate better performance or worse? Could you please explain the reason behind the evaluation?\n- Figure 9 only depicts the number of skills per instance obtained by direct prompting vs. our rationale parsing. While the larger number cannot show the generation effectiveness. Reliability or relevance to the question are more important. Is there any other ablation study showing the comparisons concerning the reliability of annotated skills?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)