SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation
Authors: Yufei Tian, Jiao Sun, Nanyun Peng, Zizhao Zhang
Accepted at: acl25

SKILLVERSE : Assessing and Enhancing LLMs with Tree Evaluation

Yufei Tian1*1    Jiao Sun2    Nanyun Peng1    Zizhao Zhang3

1University of California, Los Angeles,
2Google DeepMind, 3Google Cloud AI
yufeit@cs.ucla.edu

Abstract
As language models evolve to tackle complex and multifaceted tasks, their evaluation must adapt to capture this intricacy. A granular, skill-specific understanding of model capabilities can empower researchers to make informed model development plans. In this paper, we introduce SKILLVERSE, an unsupervised tree-structured diagnosis framework for understanding model proficiency in specific abilities. With LLM as a judge, SKILLVERSE first critiques the model responses, and then organizes them into a hierarchical structure termed dendrogram. Given proficiency at arbitrary levels of granularity, SKILLVERSE is flexible to produce insights of behaviors of modern large models. We also demonstrate its efficacy in two downstream tasks: 1) improving model in-context learning by 25% using a tree-search algorithm to select more informative few shots, and 2) accurately predicting new model weaknesses with a 55% success rate, 22% higher than the baseline.

1 Introduction
In recent years, leaderboard and benchmark results such as ChatbotArena (Chiang et al., 2024) and MMLU (Hendrycks et al., 2020) have become the dominant practice for evaluating the potency of language models. While these results provide a high-level snapshot of a model’s rank, their limited interpretability makes it difficult to identify subtle behavioral traits and derive actionable insights (Murahari et al., 2024; Moayeri et al., 2024).

The limited interpretability of the current evaluation paradigm makes it hard to compare the relative strengths and weaknesses of different models. For instance, does a higher-ranked model consistently outperform lower-ranked counterparts across the entire benchmark? Do comparable scores transfer to equivalent model performance on all sub-domains? Addressing such questions typically requires manual inspections, which is both time-consuming and costly. These challenges highlight the need for automatic, granular analyses that provide valuable insights to enrich our understanding of model behavior, which paves the road for targeted model improvement of specific capabilities.

Lately, LLM-based evaluations (Li et al., 2023; Zheng et al., 2023), where LLMs critique and judge model responses, have emerged as a scalable approach to approximate human preferences (Wang et al., 2023; Yuan et al., 2024). These methods enable detailed analysis with rich contextual feedback, forming the foundation of our diagnosis framework, SKILLVERSE. Orthogonal to developing more reliable auto-raters, this paper contributes to structuring contextual feedback to generate actionable insights for model model evaluation, comparison, debugging, and improvement.

As shown in Figure 1 and Figure 2, SKILL-

![A diagram showing the SKILLVERSE framework, including nested skills, critiques, dendrograms, and applications for in-context learning and predicting weaknesses.](page_768_320_496_384.png)

Figure 1: Up: The input-output flow of SKILLVERSE. Skill-specific critiques are extracted, structured into a dendrogram, and sliced at varying granularities to reveal nested clusters of skills and model proficiency. Bottom: Versatile applications of SKILLVERSE, from selecting informative few-shot demonstrations to uncovering hidden model weaknesses.
Figure 2: The overall framework of SKILLVERSE. A set of critiques on model responses is parsed into atomic judgments and organized using bottom-up clustering into a dendrogram, which is then unfolded at varying levels of granularity to form nested clusters, allowing for detailed analysis of model proficiency. Thanks to the hierarchical structure, this novel pipeline is highly flexible in interpreting model capabilities.

VERSE generates personalized insights through a tree-structured model assessment, tailored to the level of detail preferred by human scientists. To quantify and extract actionable insights from critiques to diverse real-world data, we introduce atomic judgment: an assessment of an indivisible aspect of model capability. Next, we conduct agglomerative clustering on these atomic judgments based on their semantic distance, resulting in a dendrogram. This tree can be chopped at different levels, resulting into clusters of varying sizes or granularities. Each cluster represents a specific skill, for which we calculate success rates to evaluate model performance (§2).

SKILLVERSE produces insights of current model behaviors, such as Gemini, Claude, and GPT-4 (§3). For instance, on Arena-Hard benchmark (Li et al., 2024) where Claude-3.5-Sonnet (Anthropic, 2024) ranks 2nd and Gemini-1.5-Pro (Google, 2024a) ranks 6th.1 SKILLVERSE finds out Gemini needs improvement in debugging, writing command lines, and business analysis. On the other hand, the higher-ranked Claude, falls short in providing analogical examples, debate, and evaluate arguments. Notably, by comparing different sized models within the same family, we also identify instances of inverse scaling, where larger models underperform smaller ones due to strong parameterized knowledge (McKenzie et al.). Such tasks include handling word inclusion/exclusion, word counts, and adherence to specific formats.

To validate SKILLVERSE’s ability to identify true model errors and its potential values for model improvement, we design a few extended improvement explorations that lead to promising gains. In §4, we show that the dendrogram enhances in-context learning by enabling a tree search algorithm that adaptively selects challenging and relevant examples as contrastive few-shot demonstrations. This approach achieves a 25% relative improvement over the standard contrastive in-context learning method (C-ICL, Yan et al. (2021)). In §5, we demonstrate that a strong reasoner such as GPT-4o can digest the model proficiency report generated from SKILLVERSE to predict weaknesses in unseen scenarios. For example, the model proficiency on ten hypothesized tasks is only 55%, 22% lower than uninformed predictions by the same reasoner.

We showcase that SKILLVERSE can serve as a powerful tool for providing fine-grained interpretation of model behaviors and developing targeted improvement of discovered model deficiencies during inference. Future research could also leverage the actionable feedback derived from SKILLVERSE to a wide range tasks: such as model routing, curating targeted training data for specific subdomains where the current model underperforms, and etc.

2 SKILLVERSE : Diagnosis Framework

2.1 Overview

Figure 2 presents the overall framework of SKILLVERSE. Starting with a large set of user prompts and model responses, we collect critiques that evaluate model responses in detail (§2.2). These critiques are then parsed into atomic judgments, enabling efficient organization and large-scale quantification. Using bottom-up clustering algorithms,
Figure 3: Upper right: A dendrogram produced by SKILLVERSE on a combination of two datasets: ChatbotArena (Chiang et al., 2024) and IF Eval (Zhou et al., 2023). Main figure: Multiple layers of nested clusters that represent model proficiencies from coarse-grained to fine-grained, by horizontally slicing the dendrogram at different levels. For each group, we can then calculate skill-level model proficiency based on the atomic judgments, as demonstrated in the parenthesis. An LLM summarizes all members in the same cluster and generate the skill-level description.

the atomic judgments are structured into a dendrogram. To interpret the results, the dendrogram is unfolded at varying levels of granularity based on the detail preferred by engineers and researchers (§2.3 and Figure 3). Finally, clusters derived independently are anchored to support multi-party analyses (§2.4).

2.2 Collecting Accurate Critiques

A straightforward way to collect critiques is adopting language models off-the-shelf or finetune them on domain specific data—a reliable approach for evaluating content relevance, style, etc. For a comprehensive representation of model capabilities, our critique model takes in both positive and negative samples and is capable of identifying both the weaknesses and strengths of a model’s response.

However, recent studies (Murugadoss et al., 2025; Son et al.; Jing et al., 2024) highlight limitations of LLMs-as-a-Judge in domains like factual verification, format checking, and calculations. Fortunately, aspects like format and calculation are programmatically checkable, eliminating the need to rely solely on language models for feedback.

Checkable Rubrics To enhance accuracy, we first identify the checkable components of a user request and leverage programs to evaluate these metrics. Previous work on instruction following (Zhou et al., 2023) identified and open-sourced 25 types of verifiable instructions for writing tasks (e.g., multiple sections, forbidden words, numbers). We built upon their efforts to create a similar database of verifiers for these checkable subtasks. In practice, we first identify whether any part of a given instruction can be mapped to our database of checkable subtasks. If a match is found, a tuple of “target checkable task, user request, model response” is passed to our program, which produces a verified result. These results are then provided as input to the critique model, alongside the original user prompt and model response, resulting in a more robust evaluation.

2.3 Structuring Diverse Critiques

Converting to Atomic Judgments To efficiently organize thousands of free-form critiques, we introduce the concept of atomic judgments, which serve as act as the building blocks for systematically organizing these critiques. An atomic claim is a statement that addresses a single, non-decomposable aspect of model ability (e.g., “Model A + failed to + identify cities in Easter Europe.” as is illustrated in Figure 2). We enforce all atomic judgments to follow a strict syntax with three components: Subject (i.e., the model name) + Verb (i.e., succeed, partially succeed, or fail) + Object (i.e., a specific task), which provides the necessary certainty and precision to quantify large volumes of critiques and calculate model proficiency.

As a preparation step for clustering, we leverage Google’s Text Embedding API (Google, 2024b) to vectorize these atomic judgments. Since the both the subject and verb are deterministic, we focus exclusively on embedding and clustering the third component, the specific task.
Hierarchical Clustering  We perform agglomerative (i.e., bottom-up) clustering on the atomic judgments based on their semantic distance. The algorithm begins by treating each claim as an individual cluster. It then identifies the two closest clusters and merges them. This unsupervised process is repeated until all the clusters are merged into a single one, resulting in a tree of nested clusters, also known as a dendrogram.

Interpreting the Dendrogram  To analyze model behavior, we horizontally slice the dendrogram at a preferred level. Next, to obtain a description for each resulting cluster, we prompt an LLM to summarize the group members.

Figure 3 illustrates a dendrogram produced in one of our experiments on the ChatbotArena (Chiang et al., 2024) and the IF Eval dataset (Zhou et al., 2023). This hierarchy captures relations among all data points. A horizontal cut at the highest level yields two primary branches: a left, technical branch and a right, non-STEM branch. Further slicing these two branches reveals subclusters: 1) coding, 2) calculating formulas and explaining STEM concepts for the left and 1) format output, 2) providing helpful information, 3) writing assistance and other content creation for the right one. Each cluster is further nested into smaller subclusters (refer to Figure 3 for details). Model proficiency can be computed by calculating the ratio of positive atomic judgments within these clusters.

2.4 Anchoring Clusters

As different models often generate varying responses, the corresponding dendrograms produced by clustering may differ slightly. Additionally, re-running the clustering process with all existing model responses every time a new model is added is inefficient. To support multi-party analyses, it is essential to merge or anchor clusters derived independently and ensure consistency across models.

Our algorithm merge two clusters only if two key conditions are satisfied. First, the centroids of the clusters must be close to each other in the feature space. Let \( \mu_i \) and \( \mu_j \) denote the centroids of clusters \( C_i \) and \( C_j \). The clusters can be merged if their similarity exceeds a threshold \( \tau \):

\[
\operatorname{sim}(\mu_i, \mu_j) = \frac{\mu_i \cdot \mu_j}{\|\mu_i\| \|\mu_j\|} \geq \tau \tag{1}
\]

Second, there must be significant overlap between the clusters. For instance, consider a counterexample where cluster \( C_i \) is a large circle with 1,000 members, and \( C_j \) is a single point at the center, merging would be inappropriate despite centroid proximity due to minimal overlap. Overlap is quantified as the intersection of cluster regions \( \operatorname{Area}(C_i \cap C_j) \) relative to their union \( \operatorname{Area}(C_i \cup C_j) \), satisfying:

\[
\frac{\operatorname{Area}(C_i \cap C_j)}{\operatorname{Area}(C_i \cup C_j)} \geq \epsilon. \tag{2}
\]

Both conditions ensure the merged cluster represents the data accurately without adding ambiguity or excessive variance.

3 Results

3.1 Experimental Setup

Dataset  We conduct our experiments on a combination of two datasets: the Instruction-Following Eval (IFEval) benchmark (Zhou et al., 2023) and ChatbotArena (Chiang et al., 2024). IFEval consists of verifiable instructions—prompts with programmatically checkable criteria, such as "write more than 400 words" or "mention the keyword 'AI' at least three times". In contrast, ChatbotArena offers a broader set of real-world LLM use cases spanning diverse topics including language understanding, creative writing, reasoning, logic, and science. These prompts are less structured and more reflective of open-domain interactions. To balance the dataset sizes, we downsampled ChatbotArena to match the scale of IFEval.

Critique Model  We use off-the-shelf language models as critique models, augmenting them with checkable rubrics to perform pairwise comparisons. Due to cost considerations, we primarily employ Gemini-1.5-Pro as the critique model. However, this choice raises the potential concern of bias toward its own outputs. To address this, we replicate the critique process using GPT-4o on a subset of 1,000 randomly selected response pairs generated by Gemini-1.5-Pro and GPT-4o. The agreement between the two critique models, measured by Pearson correlation, is 0.65—indicating a moderately strong level of reliability.

Hierarchical Clustering  We perform agglomerative clustering using the cosine similarity of text embeddings derived in Section 2.3. To facilitate human interpretation of the resulting tree structure, we select two horizontal levels to slice the dendrogram: a fine-grained level that captures more specific behaviors (e.g., write a riddle), and a higher,
<table>
  <tr>
    <th></th>
    <th>Predicted Neg.</th>
    <th>Predicted Pos.</th>
  </tr>
  <tr>
    <td>Actual Neg.</td>
    <td>TN = 0.883</td>
    <td>FP = 0.084</td>
  </tr>
  <tr>
    <td>Actual Pos.</td>
    <td>FN = 0.117</td>
    <td>TP = 0.916</td>
  </tr>
</table>

Table 1: The performance of our clustering algorithm.

more abstract level that aggregates broader categories (e.g., generate creative text). To identify the optimal clustering level, we employ the elbow method to determine the ideal number of clusters. An example of the resulting output report is shown in Figure 11 and Figure 12 in the appendix.

3.2 Verifying SKILLVERSE Reliability

Assuming the critiques are reliable and given that the success rates are algorithmically calculated, the only potential source of error in our framework arises from the unsupervised clustering process.2

We judged the accuracy of our unsupervised cluster with human evaluation, as reported below:

Accuracy of Clustering We recruit human annotators to evaluate the similarity between pairs of user requests sampled from atomic claims (e.g., <writing lyrics that are less cliché, calculating RAM occupation>), rating similarity on a scale from 1 (completely different) to 5 (highly similar). Each input is rated by 3 annotators, resulting in 1,590 annotations. Detailed guidelines are shown in Figures 13 and 14 in Appendix B.2. Comparing these human annotations with our embedding similarity produces a Pearson correlation of 0.643 (\( p<0.0001 \)), indicating substantial agreement.

Next, we converted the human-provided 5-point scale scores into binary categories, where scores of 4 and 5 indicate the same cluster and 1 and 2 indicate the opposite. Ambiguous pairs with a score of 3 were excluded, leaving 993 cases. These remaining cases were then divided into validation (for optimal slicing threshold) and test sets (for evaluation). Table 1 shows that hierarchical clustering achieved a true positive rate of 0.916 and a true negative rate of 0.88.

Accuracy of Anchoring We evaluated our anchoring procedure using dendrograms from three independently derived model sets (LLama3, Gemini1.5, Claude3). Slicing these dendrograms at the

2SKILLVERSE works with any critiques or rationales whether human-provided or automatically generated. While improving LLM evaluators is beyond the scope of this paper, advancements in neural evaluators will naturally enhance the framework’s reliability.

![Performance comparison between GPT-4-turbo and GPT-4o on coding tasks](page_1012_186_388_234.png)

(a) Performance of GPT-4-turbo and GPT-4o on coding.

![Results on other STEM areas](page_1012_420_388_234.png)

(b) Results on other STEM areas.

Figure 4: Comparing the strengths and weaknesses of two proprietary models that were release consecutively.

same threshold yielded 54, 58, and 55 clusters, respectively. Human annotators reviewed 30 random members from each cluster to decide on merging, establishing a gold standard. These clusters were split into validation and test sets. We apply grid search to optimize thresholds \( \tau \) and \( \epsilon \) on the validation set. Test results show our merging algorithm achieved a precision of 0.926 and a recall of 0.980.

In summary, both confirm the effectiveness of our approach in constructing and merging clusters.

3.3 Insights of Fine-Grained Model Behavior

While our framework is generic and can be applied to any {prompt, response} pairs, we showcase a few exemplars: 1) comparison of two proprietary models that were release consecutively by the same company (i.e., GPT-4o vs GPT-4-turbo), 2) comparisons across different model families, and 3) comparison between large and small models within the same family.

Comparison within the same model family Figure 4 illustrates the performance comparison between GPT-4o and GPT-4-turbo on STEM tasks. Despite GPT-4o being a more recent and ostensibly stronger release, SKILLVERSE reveals that GPT-4-turbo outperforms GPT-4o in specific areas, including writing SQL queries (6.1% improvement), reading and processing files (9.1% improvement), and handling music-related tasks (2% improvement).

Comparison across different families Similarly, we compare the best-performing models (as of
Identified Capabilities that Follow Inverse Scaling
<table>
  <tr><th>Wrap the entire response in double quotes</th></tr>
  <tr><td>Format text using markdown</td></tr>
  <tr><td>Output in JSON format</td></tr>
  <tr><td>End the response with a specific phrase</td></tr>
  <tr><td>Following format of limericks and rhyme scheme</td></tr>
  <tr><td>Include/exclude specific phrases</td></tr>
  <tr><td>Comply with the word count requirement</td></tr>
</table>

Table 2: SKILLVERSE identified capabilities that follow inverse scaling, where increasing model size deteriorates performance.

November 1, 2024) across three families: Claude 3.5-Sonnet, Gemini-1.5-pro, and GPT-4o. SKILLVERSE reveals that Claude excels in coding and analytical tasks such as visualization (e.g., 85.5% vs 76.8%-79.5%), creating or using AI models, handling edge cases, and writing shell commands; Gemini performs best in developing contents for educational purposes, game creation, and text formatting; while GPT-4o is superior at producing mathematical proofs, and it is exceptional at inferring the user’s precise intent from vague instructions (83.7% vs 63.2%). We provide a comprehensive view of the wins and losses of these models in Figures 11 and 12 in the Appendix.

Are larger models always better than its smaller counterparts? Another interesting finding emerges when comparing large and small models within the same family: Gemini-1.5 (Pro vs. Flash), Llama3.1 (405B, 70B, and 8B), and Claude3 (Opus, Sonnet, Haiku). On average, larger models outperform smaller ones across over 95% of identified capabilities, including STEM, problem-solving, and writing tasks. However, there are a few exceptions that demonstrate inverse scaling, where increasing model size deteriorates performance (McKenzie et al.). SKILLVERSE discovers inverse scaling on tasks with fine-grained constraints, such as keyword inclusion/exclusion and strict formatting, as detailed in Table 2.

4 SKILLVERSE Enhances Model Performance at Inference Time

The remainder of this paper explores two extended tasks. In this section, we illustrate how SKILLVERSE serves as a knowledge base of model proficiency and helps improve inference-time performance by providing better few-shot demonstrations that considers both relevance and challenges posed to the target language model. In Section 5, we demonstrate how the the uncovered model proficiency can serve as a foundation to predict model failures in previously unseen scenarios.

A. Inference Prompt
Write me a Python script in OOP for a contextual multi armed bandit sampling from 3 models. My reward distributions have high variability.

B. Individual Skills
Handle the logic of a contextual multi-armed bandit
Design an OOP architecture in Python
Initialize three model classes
Write code to simulate an interaction

C. SkillVerse

D. Demonstration Needed?
<table>
  <tr>
    <th>Yes</th>
    <th>No</th>
    <th>No</th>
    <th>Yes</th>
    <th>Relevance</th>
    <th>Contrast</th>
  </tr>
  <tr>
    <td colspan="5">Briefly explain how to implement a multi-armed bandit...</td>
    <td>0.8 x 0.3</td>
  </tr>
  <tr>
    <td colspan="5">In addition to the epsilon-greedy algorithm, what are the more advanced selection strategies ...</td>
    <td>0.5 x 0.7</td>
  </tr>
</table>

Figure 5: The dendrogram produced by SKILLVERSE helps to selects more informative few-shot demonstrations by considering both relevance and challenges posed to the target model. In contrast, previous methods selected ‘...output a Python script in OOP for a bandit-inspired approach to optimize hyper-parameters across 3 models.’ which is semantically most similar but helped less as an in-context example.

4.1 Approach

Motivation Contrastive in-context learning (C-ICL), which presents an LLM with both correct and incorrect examples as demonstration, have been shown to effectively guide the models in distinguishing between desired and undesired outputs across various tasks such as information extraction (Chao et al., 2024) and reasoning (Chia et al., 2023a; Zhang et al., 2024).

However, a typical method to construct contrastive examples involves synthetically generating negative responses by introducing hand-crafted error types, which may not best reflect a model’s own distribution. Moreover, errors in C-ICL may arise from models “over-reflecting” on simple prompts that LLMs already know how to answer. We hypothesize that SKILLVERSE mitigates the first issue by naturally storing pairs of good and bad responses, thereby facilitating an LLM’s ability to learn from its own mistakes. Additionally, access to detailed model proficiency helps resolve the second issue, as it allows us to dynamically determine whether—and which part of—an inference prompt poses more challenge to the target model.
Method Figure 5 illustrates the three steps to select few-shot examples with SKILLVERSE:

Step 1: Skill Identification. Given an inference prompt, an LLM analyzes and predicts the individual skills required to solve the task

Step 2: Mapping and Pruning. The identified skills are located within an existing dendrogram, where simpler branches (e.g., those with a success rate \( \geq T \)) are pruned.

Step 3: Selecting Few-Shot Demonstrations. The remaining candidate pairs are re-ranked based on two factors: (1) content relevance and (2) the benefit provided by the current contrastive pair. Here, the benefit is defined as \( C(r_1) - C(r_2) \), where \( C(\cdot) \) denotes the scaler score labeled by the critique model, \( r_1 \) refers to the correct response generated by another model and \( r_2 \) refers to incorrect response generated by the target model.

4.2 Experiments

Compared models We compare with principle learning from mistakes (Zhang et al., 2024), which prompts the model to learn from the distilled principles derived from self-made mistakes. We also ran two ablations for selecting few-shot examples: similarity-only that selects semantically similar instances (Mo et al., 2024), with or without incorporating self-generated errors as negative responses.

Data Used in In-Context Learning We evaluated our approach to select few-shot demonstrations on GPT-4o, Gemini-1.5-pro, and Gemini-1.5-flash using two datasets: a well-structured IF-Eval dataset that involves instruction following such as format (Zhou et al., 2023), and the less structured ChatbotArena that involves reasoning tasks (Chiang et al., 2024). We conduct the diagnosis process of SKILLVERSE using 450 prompts from the first dataset and 2,500 prompts from the second, with inference performed on 150 held-out prompts for each dataset. Additional details about the experimental setup can be found in § B.

Results We present the performance under different ICL settings in Figure 5. Interestingly, we find that ‘learning from principles’ works well with smaller models such as Gemini-1.5-flash. One possible explanation is that smaller models have limited capacity for reasoning about correct and incorrect answers in long contexts. Therefore, directly providing high-level principles might be a more effective strategy. Overall, SKILLVERSE consistently outperforms or performs on par with all baseline models. This indicates that it successfully serves as a knowledge base of granular model proficiencies, enabling the selection of more informative in-context examples to guide the target model.

![Improvement of different in-context learning approaches, compared to direct generation on IF Eval (top) and Chatbot Arena (bottom).](page_768_120_627_410.png)

Figure 6: Improvement of different in-context learning approaches, compared to direct generation on IF Eval (top) and Chatbot Arena (bottom). We posit that the performance gain is smaller for GPT-4o because of the strong performance of direct generation.

5 Auto-Discovery: Extrapolating Model Weakness to Unseen Scenarios

5.1 Approach

We explore the feasibility of automatically extrapolating to unseen error types where models may underperform. As is shown in Figure 7, we first provide the target model’s capabilities on existing data and to a reasoning LLM\(^3\) to uncover the underlying connections between areas where models perform well and poorly. Based on this analysis, we ask the reasoning model to hypothesize potential deficiencies of the target model, based on which humans curate prompts to test these hypothesized weaknesses individually.

5.2 Experimental Setup

We conduct experiments under **two settings**: 1) identify new weaknesses in a single model, *e.g.*, Gemini-1.5-flash, and 2) predict inverse scaling,, where a larger model underperforms its smaller counterpart. *e.g.*, Claude-3-Opus underperforming Claude-3-Sonnet on certain skills. We ask the reasoning model to hypothesize 50 tasks for the first setting and 20 for the second. To filter out

\footnotetext{3A different reasoning LLM is deliberately chosen to minimize inherent biases in the target model. Specifically, we use GPT-4o as it had the strongest reasoning capabilities when our experiments were conducted.}
<table>
  <tr>
    <th>ID</th>
    <th>SKILLVERSE-Informed Hypothesis</th>
    <th>Succ.</th>
    <th>Uninformed Hypothesis</th>
    <th>Succ.</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Multilingual Code Switching: Seamlessly alternating between two or more languages.</td>
    <td>0.607</td>
    <td>Opposing Opinions: Present two opposing opinions with equal depth and justification.</td>
    <td>0.982</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Logical Relations: Avoid or include words based on logical rules (e.g., AND, XOR, conditional).</td>
    <td>0.148</td>
    <td>Encode Hidden Information: Insert a hidden message using techniques like acrostics or word placements.</td>
    <td>0.622</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Avoid Specific Phonemes: Write text excluding words with selected phonemes (e.g., "th").</td>
    <td>0.271</td>
    <td>Physical Uncommonsense: Write a story where physical laws are broken (e.g., objects floating, time moving backward).</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Argument Construction: Develop a three-part argument (premise, reasoning, conclusion).</td>
    <td>0.506</td>
    <td>Speech Impediments: Create dialogue with a specific impediment or linguistic quirk.</td>
    <td>0.704</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Dynamic Math Puzzles: Create riddles where each solution depends on the previous one.</td>
    <td>0.460</td>
    <td>Rhyming with Meaning: Write a poem where rhyming words form a meaningful phrase.</td>
    <td>0.966</td>
  </tr>
</table>

Table 3: Comparison of SKILLVERSE-informed and uninformed predicted model weaknesses, with Gemini-1.5-flash success rates. The Kolmogorov-Smirnov test confirms statistically different distributions (p-value = 0.02).

![A diagram showing the process to extrapolate to model deficiencies in unseen scenarios. An LLM acts as a reasoning model to generate the hypothesis, and humans execute experiments to verify these AI-generated hypotheses.](page_184_370_484_180.png)

Figure 7: The process to extrapolate to model deficiencies in unseen scenarios. An LLM acts as a reasoning model to generate the hypothesis, and humans execute experiments to verify these AI-generated hypotheses.

![A histogram showing the performance of Gemini-1.5-flash under different settings. The success rate of SKILLVERSE-informed hypothesized weakness is only 55%, 22% lower than the uninformed hypothesis. We also list exemplar hypothesized weakness and the corresponding model performance in Table 3.](page_184_624_484_180.png)

Figure 8: The performance of Gemini-1.5-flash under different settings. The success rate of SKILLVERSE-informed hypothesized weakness is only 55%, 22% lower than the uninformed hypothesis. We also list exemplar hypothesized weakness and the corresponding model performance in Table 3.

less significant predicted tasks, such as “writing a paragraph in alternating capital and small letters”, we re-ranked them by practical relevance and selected the top half. For each selected task, we then collected 150 user prompts to test model capability solely on this task, gathered model responses, and evaluated model success rates.

As an uninformed baseline, we test the reasoning model’s ability to predict weaknesses without performance data. Specifically, we give it a random subset of skills and prompt it to propose new tasks where the model may fail. Comparing these uninformed predictions to informed ones reveals whether meaningful extrapolations arise from the reasoning model inherently or from SKILLVERSE.

![A bar chart showing ten hypothesized tasks that are likely to follow inverse scaling, and the actual performance gain of scaling up. These hypotheses include formatting a bibliography in APA style, and inserting hyperlinks into a document, as listed in Table 4. Positive values show larger models outperforming smaller ones; negative values show underperformance. We show the average gain from existing data as a horizontal reference line.](page_1012_370_484_180.png)

Figure 9: Ten hypothesized tasks that are likely to follow inverse scaling, and the actual performance gain of scaling up. These hypotheses include formatting a bibliography in APA style, and inserting hyperlinks into a document, as listed in Table 4. Positive values show larger models outperforming smaller ones; negative values show underperformance. We show the average gain from existing data as a horizontal reference line.

5.3 Result

We visualize Gemini-1.5-flash’s success rate on SKILLVERSE-informed predicted weaknesses in Figure 8. These predicted tasks are, on average, 14.2% more challenging than existing tasks and 22% more challenging than uninformed predictions. As shown in Table 3, the reasoning model successfully predicted weaknesses in following logical relations (14.8%) and avoiding specific phonemes (27.1%). In contrast, without the contrastive insights provided by SKILLVERSE, the same reasoner wrongly predicted weaknesses in presenting opposing opinions (98.2%) and contradicting physical realities (100%)—tasks where the model actually excelled.

Moreover, as is shown in Figure 9, the reasoner also succeeds in identifying capabilities where
stronger models may underperform their weaker counterparts. Appendix B lists the predicted tasks for inverse scaling, both with (Table 4) and without (Table 5) the findings by SKILLVERSE. Under the informed setting, the average performance gain of scaling up is merely 0.5%, which is statistically different from the 10.6% gain observed in uninformed predictions. Both results highlight the value of SKILLVERSE in predicting unseen model weaknesses, enabling proactive identification of potential limitations before deployment, rather than merely fixing issues after they arise.

6 Related works

Interpreting Model Behaviors. Shifting focus from aggregated leaderboard metrics, researchers have been striving to interpret model losses more effectively. For instance, LLMSys (Chiang et al., 2024) uses BERTopic to embed prompts, reduce dimensionality, and cluster them into a predefined number of groups. QualEval (Murahari et al., 2024) and a concurrent work, SkillIndex (Moayeri et al., 2024), identify attributes like subtasks and domains from evaluation data and then assign them to individual data points. In contrast, SKILLVERSE derives hierarchical clusters entirely unsupervised. Its tree structure enables efficient tracing of semantically similar prompts for downstream tasks while giving users control over granularity—chopping the tree at lower levels provides finer-grained loss categories for model capabilities.

LLM as Evaluator. Recently, LLM-based evaluation (Li et al., 2023) that requires the LLM to provide critiques to responses across a wide range of domains have emerged a scalable method for approximating human preferences (Zheng et al., 2023; Chang et al., 2024). Vu et al. (2024) and Wang et al. (2023, 2024) have demonstrated that critique models improve agreement with human judgment and reduce bias of the assessments when supervised multi-task fine-tuning is used. As a result, LLM-as-a-judge offers a practical alternative to traditional, labor-intensive methods of human preference collection and reward modeling (Wang et al., 2023; Yuan et al., 2024).

Learning from Mistakes. With discovered losses, how can we further improve the model? Extensive research has explored both training-time correction and inference-time improvement from mistakes or feedback (Pan et al., 2024) (LLM-Refine (Xu et al., 2024), SelfRefine (Madaan et al., 2023), etc.). SKILLVERSE enhances model performance during inference. Among prior works, contrastive chain-of-thought prompting (Chia et al., 2023b) and principle learning from mistakes (Zhang et al., 2024) are most relevant as both leverage model mistakes via in-context learning. However, the few-shot demonstration examples are fixed and predefined in these works, whereas SKILLVERSE adaptively selects examples for in-context learning through its dendrogram, balancing semantic relevance and potential benefit.

7 Conclusion

We developed a hierarchical diagnosis framework that distills a tree of fine-grained model capabilities from unstructured traffic data. Our framework offers the following key benefits: 1) it provides flexible insights into nuanced model abilities that are not captured by existing leaderboards or benchmarks, 2) SKILLVERSE serves as a knowledge base of model proficiency and helps enhance the model at inference-time by providing better few-shot demonstrations, and 3) it can be used to predict unseen error types before deployment.

Limitation

As pointed out by Murahari et al. (2024), fine-grained model analysis does not reject the use of benchmark metrics but uses them as one of the parts of a more actionable evaluation.

One limitation of SKILLVERSE is that we use LLMs as judges to generate critiques of model responses to user prompts, which might introduce errors. Although out-of-scope of this work, developing more robust and accurate automatic critique models can definitely improve the utility of SKILLVERSE. Inspired by the conclusion from prior works that large language models are better at evaluating model capabilities from comparison than evaluating the single model’s response in isolation (Liusie et al., 2024a,b), we always compare responses from a pair of models and generate critiques. However, as recent work suggested, pairwise comparisons can sometimes amplify biases present in LLM evaluators (Kawabata and Sugawara, 2024). Therefore, it is crucial to be aware of potential biases that may arise during the evaluation process. In addition, as a framework designed to systematically assess model capabilities and enhance performance, we emphasize the importance
of preventing the misuse of SKILLVERSE.

Acknowledgements

We would like to thank Le Hou, I-Hung Hsu, other members at Google Cloud AI, and the anonymous reviewers for the helpful discussions.

References

Anthropic. 2024. Introducing claude 3.5 sonnet. Accessed: 2024-12-14.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45.

Liang Chao, Wei Xiang, and Bang Wang. 2024. In-context contrastive learning for event causality identification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 868–881.

Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023a. Contrastive chain-of-thought prompting. arXiv preprint arXiv:2311.09277.

Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023b. Contrastive chain-of-thought prompting. Preprint, arXiv:2311.09277.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In International Conference on Machine Learning, pages 8359–8388. PMLR.

Google. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Google. 2024b. Text embeddings api. https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings. Accessed: 2024-12-14.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Xiaonan Jing, Srinivas Billa, and Danny Godbout. 2024. On a scale from 1 to 5: Quantifying hallucination in faithfulness evaluation. arXiv preprint arXiv:2410.12222.

Akira Kawabata and Saku Sugawara. 2024. Rationale-aware answer verification by pairwise self-evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16178–16196, Miami, Florida, USA. Association for Computational Linguistics.

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.

Adian Liusie, Potsawee Manakul, and Mark Gales. 2024a. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 139–151, St Julian’s, Malta. Association for Computational Linguistics.

Adian Liusie, Vatsal Raina, Yassir Fathullah, and Mark Gales. 2024b. Efficient LLM comparative assessment: A product of experts framework for pairwise comparisons. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6835–6855, Miami, Florida, USA. Association for Computational Linguistics.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegrefe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534–46594. Curran Associates, Inc.

Ian R McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, et al. Inverse scaling: When bigger isn’t better. Transactions on Machine Learning Research.

Ying Mo, Jiahui Liu, Jian Yang, Qifan Wang, Shun Zhang, Jingang Wang, and Zhoujun Li. 2024. Ciicl: contrastive in-context learning for information extraction. arXiv preprint arXiv:2402.11254.

Mazda Moayeri, Vidhisha Balachandran, Varun Chandrasekaran, Safoora Yousefi, Thomas Fel, Soheil Feizi, Besmira Nushi, Neel Joshi, and Vibhav Vineet. 2024. Unearthing skill-level insights for understanding trade-offs of foundation models. arXiv preprint arXiv:2410.13826.
Vishvak Murahari, Ameet Deshpande, Peter Clark, Tanmay Rajpurohit, Ashish Sabharwal, Karthik Narasimhan, and Ashwin Kalyan. 2024. Qual-eval: Qualitative evaluation for model improvement. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2093–2111.

Bhuvanashree Murugadoss, Christian Poelitz, Ian Drosos, Vu Le, Nick McKenna, Carina Suzana Negreanu, Chris Parnin, and Advait Sarkar. 2025. Evaluating the evaluator: Measuring llms’ adherence to task evaluation instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19589–19597.

Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484–506.

Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do.

Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. Foundational autoraters: Taming large language models for better automatic evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17086–17105.

Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024. Self-taught evaluators. arXiv preprint arXiv:2408.02666.

Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04592.

Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2024. LLMRefine: Pinpointing and refining large language models via fine-grained actionable feedback. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1429–1445, Mexico City, Mexico. Association for Computational Linguistics.

Caixia Yan, Xiao Chang, Minnan Luo, Huan Liu, Xiaojin Zhang, and Qinghua Zheng. 2021. Semantics-guided contrastive network for zero-shot object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:1530–1544.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020

Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. 2024. In-context principle learning from mistakes. arXiv preprint arXiv:2402.05403.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
Appendix

A  Full Analysis Result

Figure 10 is an example of the Gemini-1.5-flash’s capability report generated by our framework. It consists of a high-level summary, descriptions of fine-grained capabilities along with the model success rates. Overall, it is good at text formatting, and needs improvement in subdomains such as following length constraints, writing riddles, and assisting STEM tasks.

Figures 11 and 12 present a comprehensive capability report comparing Gemini-1.5-pro, Claude3.5-Sonnet, and GPT-4o on the ChatbotArena benchmark (Chiang et al., 2024). The report includes a high-level summary, detailed descriptions of fine-grained capabilities, and the models’ success rates. SKILLVERSE reveals that Claude excels in coding and analytical tasks, such as visualization (e.g., 85.5% vs. 76.8%-79.5%), creating or using AI models, handling edge cases, and writing shell commands. Gemini performs best in creating content for educational purposes, game development, and text formatting. Meanwhile, GPT-4o stands out in producing mathematical proofs and is exceptional at inferring user intent from vague instructions (83.7% vs. 63.2%).

<table>
  <tr>
    <th>High-Level Summary</th>
    <th>Fine-Grained Description</th>
    <th>Success Rate</th>
  </tr>
  <tr>
    <td rowspan="4">Creative writing in multiple forms.</td>
    <td>write jokes</td>
    <td>87.8%</td>
  </tr>
  <tr>
    <td>write a riddle</td>
    <td style="color:red;">70.0%</td>
  </tr>
  <tr>
    <td>write a song (rap or jazz)</td>
    <td>82.6%</td>
  </tr>
  <tr>
    <td>write a poem (limerick, haiku)</td>
    <td>72.9%</td>
  </tr>
  <tr>
    <td rowspan="2">follow length constraint</td>
    <td>write within word limit</td>
    <td style="color:red;">64.8%</td>
  </tr>
  <tr>
    <td>follow request on sentence count</td>
    <td style="color:red;">69.1%</td>
  </tr>
  <tr>
    <td rowspan="4">Text formatting</td>
    <td>format and use bullet points</td>
    <td>73.7%</td>
  </tr>
  <tr>
    <td>format the title (double angular brackets)</td>
    <td>90.7%</td>
  </tr>
  <tr>
    <td>separate items with six asterisks</td>
    <td>79.4%</td>
  </tr>
  <tr>
    <td>separate paragraphs with special characters</td>
    <td>92.9%</td>
  </tr>
  <tr>
    <td rowspan="3">add a postscript starting with e.g., "PS."</td>
    <td></td>
    <td>78.6%</td>
  </tr>
  <tr>
    <td>format and highlight sections in the text</td>
    <td>85.5%</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">generate visualization</td>
    <td></td>
    <td>78.3%</td>
  </tr>
  <tr>
    <td>write command-line script (for variable replacement, synchronize folders, etc.)</td>
    <td style="color:red;">71.3%</td>
  </tr>
  <tr>
    <td rowspan="2">add correct comments for functions</td>
    <td></td>
    <td>76.7%</td>
  </tr>
  <tr>
    <td>identify and handle errors, troubleshooting</td>
    <td>75.0%</td>
  </tr>
  <tr>
    <td rowspan="2">format output in JSON format</td>
    <td></td>
    <td style="color:red;">64.2%</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">provide technical instructions on a wide range of technologies</td>
    <td></td>
    <td>76.3%</td>
  </tr>
  <tr>
    <td>calculate or estimate (range, confidence interval, ambiguity, CPU usage)</td>
    <td>87.0%</td>
  </tr>
  <tr>
    <td>work with time and date (calculate, provide, explain)</td>
    <td style="color:red;">59.3%</td>
  </tr>
</table>

B  Experimental Details

B.1  Detailed Results of Auto-Discovery

Table 4 and Table 5 lists the predicted inverse scaling tasks with or without the insights produced by SKILLVERSE. A positive value indicates that the larger model outperforms its smaller counterpart, while a negative value indicates underperformance. On average, the larger models outperform their smaller siblings by only 0.5%, compared to 10.6% in the uninformed predictions.

B.2  Human Annotation

We present the complete annotation guideline used in Section 3 to verify the accuracy of our clustering algorithm in Figure 13 and 14 The inter-annotator agreement is 0.88 as measured by Pearson correlation.

Figure 10: An example of the Gemini-1.5-flash’s capability report generated by our framework. It consists of a high-level summary, descriptions of fine-grained capabilities along with the model success rates. We highlight model weaknesses in red.
<table>
  <tr>
    <th>High-Level Summary</th>
    <th>Detailed description</th>
    <th>Gemini-1.5-pro</th>
    <th>Claude3.5-Sonnet</th>
    <th>GPT-4o</th>
  </tr>
  <tr>
    <td rowspan="6">Write functional and specific-purpose code, focusing on implementation and problem-solving.</td>
    <td>write code for visualization purpose (e.g., scattered plot, bar chart)</td>
    <td>79.49%</td>
    <td>85.53%</td>
    <td>76.83%</td>
  </tr>
  <tr>
    <td>implement a feature or write code for game development (e.g., terrain generation, sprite editor)</td>
    <td>71.25%</td>
    <td>69.03%</td>
    <td>74.04%</td>
  </tr>
  <tr>
    <td>create or use AIML model/chatbot</td>
    <td>87.61%</td>
    <td>94.56%</td>
    <td>90.70%</td>
  </tr>
  <tr>
    <td>write SQL queries containing specific techniques like CTEs, joins, window functions</td>
    <td>84.92%</td>
    <td>89.92%</td>
    <td>85.42%</td>
  </tr>
  <tr>
    <td>improve code quality (organization, efficiency, conciseness, functionality)</td>
    <td>62.51%</td>
    <td>58.81%</td>
    <td>70.44%</td>
  </tr>
  <tr>
    <td>comment on code quality (efficiency, readability) translate into first-order logic (FOL) and conjunctive normal form (CNF), write test cases, handling edge cases and generating the expected output</td>
    <td>90.93%</td>
    <td>84.87%</td>
    <td>92.44%</td>
  </tr>
  <tr>
    <td rowspan="6">Compose clear and comprehensive explanations for algorithms</td>
    <td>provide detailed instructions for package installation, configuration, and setup</td>
    <td>96.31%</td>
    <td>89.28%</td>
    <td>91.83%</td>
  </tr>
  <tr>
    <td>write code for games/chess/elo, create card decks, solve rubiks cube</td>
    <td>76.79%</td>
    <td>73.57%</td>
    <td>80.78%</td>
  </tr>
  <tr>
    <td>write code or script for shell commands (file operations, automation, configuration)</td>
    <td>80.40%</td>
    <td>90.11%</td>
    <td>96.43%</td>
  </tr>
  <tr>
    <td>write code or function about network, stack, TCP, IP address</td>
    <td>74.99%</td>
    <td>65.23%</td>
    <td>70.44%</td>
  </tr>
  <tr>
    <td>Fixing bugs or troubleshooting</td>
    <td>86.14%</td>
    <td>77.92%</td>
    <td>94.48%</td>
  </tr>
  <tr>
    <td>write code for data processing (e.g., scrape data, convert/generate files, save files, extract content)</td>
    <td>79.78%</td>
    <td>85.14%</td>
    <td>86.43%</td>
  </tr>
  <tr>
    <td rowspan="6">Provide mathematical analysis, calculations, or proofs.</td>
    <td>provide explanations of algorithms including aspects like steps, examples, details, and target audiences</td>
    <td>96.97%</td>
    <td>90.63%</td>
    <td>89.54%</td>
  </tr>
  <tr>
    <td>use examples (analogy, specific, real-world, illustrative)</td>
    <td>89.18%</td>
    <td>62.06%</td>
    <td>81.98%</td>
  </tr>
  <tr>
    <td>compare (pros/cons of) methods/strategies/options</td>
    <td>93.90%</td>
    <td>82.99%</td>
    <td>89.55%</td>
  </tr>
  <tr>
    <td>provide explanation (sentiment analysis, chatbot, LLM, Langchain, AutoGen)</td>
    <td>88.05%</td>
    <td>88.55%</td>
    <td>91.07%</td>
  </tr>
  <tr>
    <td>complexity analysis for algorithm</td>
    <td>86.93%</td>
    <td>87.89%</td>
    <td>82.83%</td>
  </tr>
  <tr>
    <td>provide helpful, correct and relevant suggestions and advice</td>
    <td>100.00%</td>
    <td>97.96%</td>
    <td>98.15%</td>
  </tr>
  <tr>
    <td rowspan="2">Create and analyze technical content with practical details.</td>
    <td>write documentation including comments and docstrings</td>
    <td>94.87%</td>
    <td>90.70%</td>
    <td>83.86%</td>
  </tr>
  <tr>
    <td>provide or write mathematical proofs</td>
    <td>62.00%</td>
    <td>75.73%</td>
    <td>83.77%</td>
  </tr>
  <tr>
    <td>calculate financial terms (interest rate, future value, present value, profit/loss, etc) or analyze (options, hedging, market behavior, etc)</td>
    <td>79.56%</td>
    <td>75.63%</td>
    <td>82.63%</td>
  </tr>
  <tr>
    <td>solve a (math, physics) problem</td>
    <td>82.35%</td>
    <td>88.25%</td>
    <td>78.46%</td>
  </tr>
  <tr>
    <td>perform calculation or derive formula</td>
    <td>77.42%</td>
    <td>70.25%</td>
    <td>75.46%</td>
  </tr>
  <tr>
    <td rowspan="4">debate or evaluate arguments</td>
    <td>create game related content like character sheet or game session outline</td>
    <td>89.20%</td>
    <td>70.74%</td>
    <td>86.51%</td>
  </tr>
  <tr>
    <td>analyze and advise on security vulnerabilities and best practices</td>
    <td>89.20%</td>
    <td>79.41%</td>
    <td>85.68%</td>
  </tr>
  <tr>
    <td>write music-related items (song, melody, chord progression) using formats like ABC notation and MIDI</td>
    <td>85.73%</td>
    <td>81.59%</td>
    <td>71.26%</td>
  </tr>
  <tr>
    <td>provide detailed business analysis like strategy, model, SWOT, market, and finance design on a topic like software, schema, or engineering process</td>
    <td>92.20%</td>
    <td>96.60%</td>
    <td>91.86%</td>
  </tr>
  <tr>
    <td>debate or evaluate arguments</td>
    <td>90.90%</td>
    <td>69.11%</td>
    <td>74.95%</td>
  </tr>
</table>

Figure 11: The capability report comparing Gemini-1.5-pro, Claude3.5-Sonnet, and GPT-4o on ChatbotArena (Chiang et al., 2024). It consists of a high-level summary, descriptions of fine-grained capabilities along with the model success rates.
<table>
  <tr>
    <th>High-Level Summary</th>
    <th>Detailed description</th>
    <th>Gemini-1.5-pro</th>
    <th>Claude3.5-Sonnet</th>
    <th>GPT-4o</th>
  </tr>
  <tr>
    <td rowspan="6">Develop, evaluate, and refine diverse content (creative, educational)</td>
    <td>provide comprehensive recipe (ingredient, flavor, nutrition, instruction)</td>
    <td>93.29%</td>
    <td>92.97%</td>
    <td>85.72%</td>
  </tr>
  <tr>
    <td>create a structured and well-organized curriculum or outline for a course (covering topics, levels, titles)</td>
    <td>95.22%</td>
    <td>97.31%</td>
    <td>88.28%</td>
  </tr>
  <tr>
    <td>write creative content like story, script, lyrics, article</td>
    <td>79.55%</td>
    <td>81.85%</td>
    <td>70.26%</td>
  </tr>
  <tr>
    <td>translate or analyze (grammar, meaning, usage) of language and write in a specific style (tone, format)</td>
    <td>84.50%</td>
    <td>86.47%</td>
    <td>82.01%</td>
  </tr>
  <tr>
    <td>create a detailed training plan with time estimates and specific workouts</td>
    <td>80.48%</td>
    <td>76.45%</td>
    <td>66.69%</td>
  </tr>
  <tr>
    <td>categorize, extract, and identify entities, relationships, from long context</td>
    <td>78.81%</td>
    <td>82.65%</td>
    <td>75.01%</td>
  </tr>
  <tr>
    <td rowspan="4">Focus on clarity, conciseness, and formatting in writing.</td>
    <td>structure text using headings, (numbered) bullet points, and bolding</td>
    <td>91.65%</td>
    <td>80.00%</td>
    <td>88.72%</td>
  </tr>
  <tr>
    <td>write in a clear, well-organized, easy to understand, and readable format (e.g., formatting, structure, clarity)</td>
    <td>92.37%</td>
    <td>89.20%</td>
    <td>88.72%</td>
  </tr>
  <tr>
    <td>provide relevant explanation</td>
    <td>96.64%</td>
    <td>90.01%</td>
    <td>95.81%</td>
  </tr>
  <tr>
    <td>be concise and to the point (instead of overly verbose, repetitive)</td>
    <td>95.21%</td>
    <td>97.00%</td>
    <td>91.32%</td>
  </tr>
  <tr>
    <td rowspan="2">write concisely, balancing the level of detail (using examples, visual aids)</td>
    <td>84.85%</td>
    <td>72.74%</td>
    <td>91.50%</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="6">Understand and address user intent with clear, structured responses.</td>
    <td>address vague requests and identify the user's actual intent</td>
    <td>63.18%</td>
    <td>63.18%</td>
    <td>83.68%</td>
  </tr>
  <tr>
    <td>understand and fulfill the prompt, addressing requirements, instructions, and questions</td>
    <td>91.53%</td>
    <td>89.50%</td>
    <td>89.99%</td>
  </tr>
  <tr>
    <td>provide help on ethical related issues</td>
    <td>97.81%</td>
    <td>92.74%</td>
    <td>97.32%</td>
  </tr>
  <tr>
    <td>not refuse too many requests due to ethical concerns</td>
    <td>63.34%</td>
    <td>83.77%</td>
    <td>87.26%</td>
  </tr>
  <tr>
    <td>provide resources (links, references)</td>
    <td>72.02%</td>
    <td>69.39%</td>
    <td>78.28%</td>
  </tr>
  <tr>
    <td>write about truthful, relevant, accurate, and factual information</td>
    <td>77.38%</td>
    <td>80.97%</td>
    <td>80.52%</td>
  </tr>
  <tr>
    <td>write about comprehensive details (scope, limitations, potential issues)</td>
    <td>89.29%</td>
    <td>83.78%</td>
    <td>87.42%</td>
  </tr>
</table>

Figure 12: Figure 11 continued.

<table>
  <tr>
    <th>ID</th>
    <th>Description</th>
    <th>Gemini-1.5</th>
    <th>Llama-3</th>
    <th>Claude-3</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Format a bibliography in APA style based on a list of references</td>
    <td>-13.2%</td>
    <td>7.7%</td>
    <td>-13.6%</td>
    <td>-6.4%</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Insert a hyperlink for every occurrence of a specific word</td>
    <td>4.7%</td>
    <td>0.4%</td>
    <td>2.3%</td>
    <td>2.5%</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Write all numbers in words instead of numerals</td>
    <td>-3.4%</td>
    <td>-3.5%</td>
    <td>29.0%</td>
    <td>7.4%</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Write a response where every second sentence starts with the same word</td>
    <td>-8.7%</td>
    <td>6.5%</td>
    <td>7.7%</td>
    <td>1.8%</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Use exactly one comma per sentence, placed in a specified position</td>
    <td>2.9%</td>
    <td>-1.7%</td>
    <td>0.1%</td>
    <td>0.4%</td>
  </tr>
  <tr>
    <td>6</td>
    <td>End each sentence with a specific punctuation mark (e.g., every sentence must end with an exclamation point)</td>
    <td>-20.0%</td>
    <td>14.6%</td>
    <td>-6.3%</td>
    <td>-3.9%</td>
  </tr>
  <tr>
    <td>7</td>
    <td>Format a table using LaTeX syntax</td>
    <td>-9.0%</td>
    <td>5.2%</td>
    <td>5.2%</td>
    <td>0.5%</td>
  </tr>
  <tr>
    <td>8</td>
    <td>Replace every occurrence of the word 'the' with 'a'</td>
    <td>13.7%</td>
    <td>12.6%</td>
    <td>-6.5%</td>
    <td>6.6%</td>
  </tr>
  <tr>
    <td>9</td>
    <td>Enclose the entire response in an HTML <p> and </p> tag</td>
    <td>1.8%</td>
    <td>1.8%</td>
    <td>-8.8%</td>
    <td>-1.8%</td>
  </tr>
  <tr>
    <td>10</td>
    <td>Write a strict-structure sonnet while maintaining a particular rhyme scheme (e.g., iambic pentameter)</td>
    <td>-0.2%</td>
    <td>5.3%</td>
    <td>1.6%</td>
    <td>2.2%</td>
  </tr>
  <tr>
    <td>11</td>
    <td>Wrap conversations and individual sentences in parentheses</td>
    <td>-10.9%</td>
    <td>5.9%</td>
    <td>-7.3%</td>
    <td>-4.1%</td>
  </tr>
</table>

Table 4: Predicted inverse scaling tasks (SKILLVERSE-informed) and the performance gap between larger and smaller models. We evaluated Gemini-1.5 (pro vs. flash), Llama3 (405b vs. 70b), and Claude3 (Opus vs. Sonnet). A positive value indicates that the larger model outperforms its smaller counterpart, while a negative value indicates underperformance. On average, the larger models outperform their smaller siblings by only 0.5%, compared to 10.6% in the uninformed predictions (Table 5 in the Appendix).
<table>
  <tr>
    <th>Index</th>
    <th>Description</th>
    <th>Gemini-1.5</th>
    <th>Llama3</th>
    <th>Claude-3</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>1</td>
    <td>Respond to a question using only questions</td>
    <td>13.2%</td>
    <td>15.8%</td>
    <td>-7.6%</td>
    <td>7.1%</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Recommend a Playlist Based on Mood</td>
    <td>17.8%</td>
    <td>4.0%</td>
    <td>9.0%</td>
    <td>10.3%</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Describe something as if it were a recipe</td>
    <td>16.7%</td>
    <td>15.0%</td>
    <td>-3.0%</td>
    <td>9.6%</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Suggest a Weekly Meal Plan</td>
    <td>28.7%</td>
    <td>2.1%</td>
    <td>10.7%</td>
    <td>13.8%</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Create a riddle for technical concepts</td>
    <td>29.4%</td>
    <td>23.8%</td>
    <td>16.4%</td>
    <td>23.2%</td>
  </tr>
  <tr>
    <td>6</td>
    <td>Describe a process, but only using the future tense</td>
    <td>4.4%</td>
    <td>-3.4%</td>
    <td>1.3%</td>
    <td>0.8%</td>
  </tr>
  <tr>
    <td>7</td>
    <td>Reverse Word Order</td>
    <td>-3.4%</td>
    <td>20.9%</td>
    <td>26.8%</td>
    <td>14.8%</td>
  </tr>
  <tr>
    <td>8</td>
    <td>Generate Advice for Improving Public Speaking Skills</td>
    <td>5.1%</td>
    <td>28.9%</td>
    <td>8.1%</td>
    <td>14.0%</td>
  </tr>
  <tr>
    <td>9</td>
    <td>Write a song chorus and bridge</td>
    <td>7.1%</td>
    <td>14.7%</td>
    <td>5.0%</td>
    <td>8.9%</td>
  </tr>
  <tr>
    <td>10</td>
    <td>Generate Rhyming Words</td>
    <td>-0.6%</td>
    <td>7.9%</td>
    <td>1.2%</td>
    <td>2.8%</td>
  </tr>
  <tr>
    <td>11</td>
    <td>Alliteration Generation</td>
    <td>10.2%</td>
    <td>2.1%</td>
    <td>22.3%</td>
    <td>11.5%</td>
  </tr>
</table>

Table 5: Predicted inverse scaling tasks (uninformed predictions) and the performance gap between larger and smaller models. We evaluated Gemini-1.5 (pro vs. flash), Llama3 (405b vs. 70b), and Claude3 (Opus vs. Sonnet). A positive value indicates that the larger model outperforms its smaller counterpart, while a negative value indicates underperformance. On average, the larger models outperform their smaller siblings by 10.6%, with a distribution that is statistically different from those predicted with the performance data as input.
Introduction of Tasks

Welcome to the phrase similarity annotation task! The objective of this task is to annotate the similarity between two phrases that describe user requests. As a rater, you will be provided with pairs of short phrases sourced from various user requests or feedback. Some requests are extremely similar, while others are not. Your task is to label their similarity on a scale from 1 to 5.

Rater Instructions

To ensure the quality of the annotation, please follow these detailed instructions:

1. Review the given phrases carefully
   a. The phrases you will annotate come from various user requests or feedback to these requests
   b. Ignore the sentiment of the feedback (e.g., positive or negative), and focus solely on the similarity of the user request parts.
   c. For more details, please refer to examples 01 and 03

2. Choose a similarity rating from a 1-5 scale
   • 5-Highly similar (same group) - The phrases are highly similar and should belong to the same group.
   • 4-Relevant (sister groups) - The phrases are relevant and should belong to related groups (sister groups).
   • 3-Neutral - Choose this when the similarity is ambiguous or you are not sure.
   • 2-Weakly related (leaning towards unrelated) - The phrases are leaning towards unrelated, but you can see how they might be distantly related.
   • 1-Totally unrelated (never be sister groups) - The phrases have no similarity and should never be in the same group

Examples:

Please ignore the sentiment of the feedback (e.g., succeed or fail), and only focus on the similarity between the user request parts.

<table>
  <tr>
    <th>ID</th>
    <th>Input Pairs</th>
    <th>Expected Annotation</th>
  </tr>
  <tr>
    <td>01</td>
    <td>A-write a poem about playing video games,<br>B-wrote a haiku about missing class, failing to follow the poem structure</td>
    <td>5-Highly similar (same gro...</td>
  </tr>
  <tr>
    <td>02</td>
    <td>A-used the word "peace" at least 10 times,<br>B-include the keyword "Resume" for 3 times</td>
    <td>5-Highly similar (same gro...</td>
  </tr>
</table>

Comment: Both pairs involve specific instructions on word usage within text. The two phrases should belong to the same group – 01 on avoiding capitalization, while 02 on repeating specific words.

P.S. Following the instruction to ignore the sentiment of the feedback, wrote a haiku about missing classes, failing to follow the poem structure is reduced to wrote a haiku about missing classes

<table>
  <tr>
    <th>ID</th>
    <th>Input Pairs</th>
    <th>Expected Annotation</th>
  </tr>
  <tr>
    <td>03</td>
    <td>A-avoid using commas</td>
    <td>4- Relevant (sister groups)</td>
  </tr>
</table>

Figure 13: Annotation Guideline: Phrase Similarity Annotation Instructions Page 1
<table>
  <tr>
    <th>ID</th>
    <th>Input Pairs</th>
    <th>Expected Annotation</th>
  </tr>
  <tr>
    <td></td>
    <td>B-exclude the words "can" and "ride" in the written response, but "ride" appeared twice</td>
    <td></td>
  </tr>
  <tr>
    <td>04</td>
    <td>A-write 3 to 5 paragraphs<br>B - (not) stay within the 100-word limit, going 122 words over</td>
    <td><a href="#">4- Relevant (sister groups)</a></td>
  </tr>
</table>

Comment: Both pairs involve specific instructions to restrict certain elements in writing — 03 avoiding punctuation or keywords and 04 limiting paragraph or word count. These user requests are highly relevant but not identical, hence "sister groups."

P.S. Following the instruction to ignore the sentiment of the feedback, exclude the words "can" and "ride" in the written response, but "ride" appeared twice in the response is reduced to exclude the words "can" and "ride" in the written response

<table>
  <tr>
    <th>ID</th>
    <th>Input Pairs</th>
    <th>Expected Annotation</th>
  </tr>
  <tr>
    <td>05</td>
    <td>A-wrote the poem without capitalization<br>B-writing lyrics that are less cliche</td>
    <td>3-Neutral</td>
  </tr>
  <tr>
    <td>06</td>
    <td>A-finished writing the email<br>B-wrote the entire response in English</td>
    <td>3-Neutral</td>
  </tr>
</table>

Comment: Both pairs contain instructions related to style or language use, but lack clear connections. The tasks are broadly related, resulting in a "neutral" rating.

<table>
  <tr>
    <th>ID</th>
    <th>Input Pairs</th>
    <th>Expected Annotation</th>
  </tr>
  <tr>
    <td>07</td>
    <td>A-write an academic proposal,<br>B-explain the calorie content of almonds</td>
    <td>2-Weakly related (leaning ...)</td>
  </tr>
  <tr>
    <td>08</td>
    <td>A-focus on making the tone of statements sound more like a formal announcement,<br>B-included more information, such as a product image and name</td>
    <td>2-Weakly related (leaning ...)</td>
  </tr>
</table>

Comment: Both tasks demand distinct content and presentation approaches. However, they are distantly related because ID 07 involves the concept of preparation, and ID 08 focuses on enhancing information delivery.

<table>
  <tr>
    <th>ID</th>
    <th>Input Pairs</th>
    <th>Expected Annotation</th>
  </tr>
  <tr>
    <td>09</td>
    <td>A-exclude commas from the output.<br>B-explain the calorie content of almonds</td>
    <td>1-Totally unrelated (never ...)</td>
  </tr>
  <tr>
    <td>10</td>
    <td>A-writing quality that is repetitive and clunky<br>B-use the proper markdown syntax for italics</td>
    <td>1-Totally unrelated (never ...)</td>
  </tr>
</table>

Comment: Both pairs involve completely different writing elements—one addresses content or style, while the other focuses on format or syntax. These tasks are unrelated in their objectives and techniques.

Figure 14: Annotation Guideline: Phrase Similarity Annotation Instructions Page 2

Reviewer paper summaries: This paper proposes a evaluation framework for fine-grained analysis of LLMs' capabilities, and validate the effectiveness of this evaluation frameworks with two downstream tasks: in-context learning demonstration selection and predicting LLMs' weakness. This paper is organized well and the experimental results demonstrate the effectiveness of the proposed evaluation framework.

**Comments after Author Discussion** thanks for the author's responses, your answers are interesting.

This study propose an unsupervised tree-structured diagnosis framework for understanding model proficiency in specific abilities. SkillVerse provides insights of large language models’ behavior by analyzing the model response and organizing them into hierarchical structure termed dendrogram. Experiments show that the SkillVerse as an external knowledge base of model behavior helps select better in-context examples and thus improve model performance; Besides, it also helps predict models’ new weakness.
Summary of strengths: 1. Clear presentation and a contribution for evaluating LLMs
2. The downstream applications are very interesting.

1. Novel framework.  Organizing the model’s behavior through a hierarchical structure is a great innovation since it is helpful for both analyzing high-level and fine-grained behavior. The tree structure facilitates skill search and gives users sufficient flexibility for the level of granularity and cost budget. 
2. Strong experimental results and clear practical benefits. The paper presents strong metrics for the proposed frameworks at various level and dimensions. It investigates the accuracy of clustering and anchoring, which asserts confidence on the model behavior representation encoded in the tree structure. The study also investigate two practical use case of the propose framework: select targeted in-context examples during inference time, and model weakness discovery. I am joyful to see strong positive results on both study.
Summary of weaknesses: I have a question regarding the design of SKILLVERSE, which is more of an open-ended inquiry. While I appreciate that the authors have included a wide range of skills, I wonder if it would be possible to include a static dendrogram. Since the clustering process primarily captures semantics, it might pose challenges for tasks at the pragmatic level. For example, the tasks **please do not show bias to ladies** and **do not violate the principle of loyalty** both require the skill of ethics, despite their semantic differences.

I don’t see major weakness of this paper, though it could be better to see the cost analysis of the proposed method. For example, the cost of constructing such the mode behavior tree, the cost of selecting in-context samples (since one need to evaluate the relevance & benefits). 

I also expect to see more details the proposed method for the sake of transparency. For example, how is the cluster area defined? how are the relevance and benefits of in-context examples obtained? task details, etc.
Comments, suggestions and typos: Please see the weakness section

1. Does the critique generation only cover the failed samples? If so, what is the motivation of excluding positive examples? Wouldn’t it be a better representation of model skills & behaviors if one includes both positive and negative examples?
2. How do you identify checkable components for non-technical tasks such as writing? Is there any significant difference on accuracy of clustering and anchoring for technical and non-technical tasks? How about the performance gain through selecting in-context examples?
3. Could you also provide the list of tasks used in experiments? (I might miss it through.
Overall assessment scores: 4

4
Metareview score: 4
Metareview: This paper proposed a novel framework, SkillVerse, for understanding model proficiency in specific abilities. SKILL VERSE critiques the model responses and categorize them into a hierarchical tree structure. It improves model in-context learning performance and predicts new model weaknesses.
Summary of reasons to publish: * A novel framework is proposed
* Experiment results show the proposed framework improves against the baselines.