Law of the Weakest Link: Cross Capabilities of Large Language Models
Authors: Ming Zhong
Aston Zhang
Xuewei Wang
Rui Hou
Wenhan Xiong
Chenguang Zhu
Zhengxing Chen
Liang Tan
Chloe Bi
Mike Lewis
Sravya Popuri
Sharan Narang
Melanie Kambadur
Dhruv Mahajan
Sergey Edunov
Jiawei Han
Laurens van der Maaten
Venue: ICLR 2025 Poster (2025)
Keywords: Cross Capability
Law of the Weakest Link
Evaluation
Large Langauge Models
Benchmark
Primary area: datasets and benchmarks
TL;DR: We define and benchmark cross capabilities in LLMs, revealing the "Law of the Weakest Link": collaborative performance is significantly constrained by the weakest individual capability.

Abstract: The development and evaluation of Large Language Models (LLMs) have largely focused on individual capabilities. However, this overlooks the intersection of multiple abilities across different types of expertise that are often required for real-world tasks, which we term **cross capabilities**. To systematically explore this concept, we first define seven core individual capabilities and then pair them to form seven common cross capabilities, each supported by a manually constructed taxonomy. Building on these definitions, we introduce *CrossEval*, a benchmark comprising 1,400 human-annotated prompts, with 100 prompts for each individual and cross capability. To ensure reliable evaluation, we involve expert annotators to assess 4,200 model responses, gathering 8,400 human ratings with detailed explanations to serve as reference examples. Our findings reveal that current LLMs consistently exhibit the ``Law of the Weakest Link,'' where cross-capability performance is significantly constrained by the weakest component. Across 58 cross-capability scores from 17 models, 38 scores are lower than all individual capabilities, while 20 fall between strong and weak, but closer to the weaker ability. These results highlight LLMs' underperformance in cross-capability tasks, emphasizing the need to identify and improve their weakest capabilities as a key research priority. The code, benchmarks, and evaluations are available on our [project website](https://www.llm-cross-capabilities.org).

Published as a conference paper at ICLR 2025
LAW OF THE WEAKEST LINK: CROSS CAPABILITIES OF LARGE LANGUAGE MODELS
Ming Zhong 1.2 Aston Zhang" Xuewei Wang' Rui Hou'
Wenhan Xiong'
Chenguang Zhu'
Zhengxing Chen' Liang Tan- Chloe Bi Mike Lewis' Sravya Popuri"
Sharan Narang'
Melanie Kambadur' Dhruv Mahajan' Sergey Edunov' Jiawei Han
Laurens van der Maaten 1
Llama Team, AI @ Meta University of Illinois Urbana- Champaagn
ABSTRACT
The development and evaluation of Large Language Models (LLMs) have largely focused on individual capabilities. However, this overlooks the intersection of multiple abilities across different types of expertise that are often required for real-world tasks, which we term cross capabilities. To systematically explore this concept, we first define seven core individual capabilities and then pair them form seven common cross capabilities, each supported by manually constructed taxonomy. Building on these definitions, we introduce CROSSEVAL, a benchmark comprising 1 1,400 and cross capability. To ensure reliable evaluation, we involve expert annotators to assess 4,200 model responses, gathering 8,400 human ratings with detailed ex- planations to serve as reference examples. Our findings reveal that current LLMs consistently exhibit the "Law of the Weakest Link," where cross-capability perfor- mance is significantly constrained by the weakest component. Across 58 cross- capability scores from 17 models, 38 scores are lower than all individual capabili- ties, while 20 fall between strong and weak, but closer to the weaker ability. These results highlight LLMs' underperformance in cross-capatee cross-capability tasks, emphasizing the need to identify and improve their weakest capabilities as key research prior- ity. The code, benchmarks, and evaluations are available on our project website.
prompts, with 100 prompts for each individual
INTRODUCTION
The development and evaluation of Large Language Models (LLMs) (OpenAI, 2023; 2024; An- thropic, 2024; Reid et al., 2024) have predominantly centered on individual capabilities. Devel- opers commonly construct specialized datasets tailored to distinct abilities, and then train models by blending these data sources. For instance, Llama 3's post-training incorporates a mix of data, from general English to code and multilingual content, among others, each subset aimed at honing a specific skill (Llama Team, 2024). Evaluation methods follow a similar pattern, with benchmarks typically assessing these abilities in isolation, offering snapshot of how well model can rea- son (Clark et al., 2018; Cobbe et al., 2021; Hendrycks et al., 202 lb). code (Chen et al., 2021; Austin al., 2021), or manage factual knowledge (Hendrycks et al., 2021a).
However, can all real-world tasks be adequately categorized under just one capability, or do they frequently demand the seamless integration of multiple skills, thereby challenging the prevalent approach to evaluating these advanced LLMs? Consider a user prompt asking, "Which direction has the total rainfall in Tokyo, Japan been trending over the past 10 years? Explain step by step." Such a task requires the integration of tool use (web browsing) with analytical reasoning. Similarly, when a developer provides HTMI. and JavaScript for an API-driven application and asks, "Give me basic understanding of what this web app does." " the model must combine long-context comprehension with coding expertise. We define these scenarios as cross capabilities- the intersection of multiple distinct capabilities across different types of expertise necessary to address complex, real-world tasks. This discrepancy between the isolated focus of current LLM evaluation and the multifaceted demands of user interactions raises a critical question:
How does the performance of LLMs on tasks requiring cross capabilities reflect or diverge from their performance in individual capabilities?

---

Published as a conference paper at ICLR 2025
This question opens up various possibilities for portraying the relationship between distinct abilities in LLMs and their collective performance. Insights from multiple fields can shed light on these dy- namics. For example, "Synergy Theory" (Coriing, 1983) suggests that the interaction of different components in a system can produce effects greater than the sum of individual parts, while "Com- pensatory Mechanism" (Adler, 1917), a concept from psychology, introduces that stronger abilities within a system can offset weaker ones. Additionally, "Law of the Weakest Link" (Liebig, 1840) presents that system's 's performance limited by its weakest element, and the idea of "Emergent Properties" (Anderson, 1972) highlights how new behaviors can arise from the interaction of com- ponents, which are not predictable from their individual components alone. Given the substantial investment in enhancing the particular abilities of LLMs, identifying how individual capabilittiss impact performance on tasks requiring cross abilities crucial for guiding future development.
In this paper, we investigate how the interplay of individual capabilities influences collective per- formance, with the goal of providing insights for advancing LLM effectiveness in handling cross- capability tasks. Specifically, our research explores the following key questions:
RQ1: How can we comprehensively define individual and cross capabilities in LLMs? To effectively define all capabilities in LLMs, must systematically categorize tasks that reflect real-world interactions. We identify seven core individual capabilities, including English, Rea- soning, Coding, Image Recognition, Tool Use, Long Context, and Spanish, and pair them to form seven common cross capabilities, such as Coding & Reasoning. For each capability, we provide clear definitions and manually construct detailed taxonomy that links the capability to complex tasks, systematically breaking it down into two hierarchical levels: broad categories at the first level and specific tasks at the second. These taxonomies lay the groundwork for constructing benchmarks that can comprehensively cover and assess a broader range of LLM capabilities.
RQ2: How can we benchmark both individual and cross capabilities in LLMs? To benchmark all capabilities in LLMs, we construct a detailed evaluation framework, CROSsEVAL, based on manually annotated prompts that align with our established taxonomy. Each prompt is categorized by capability and difficulty, ensuring thorough coverage of both individual and cross capabilities. We collect multiple model responses for each prompt and engage expert human annotators to rate and explain these responses. In total, CROSSEVAL comprises 1,400 prompts, 4,200 model responses, and 8,400 expert ratings with detailed explanations. Finally, we introduce LLM-based evaluators to assess responses using these reference examples, achieving strong agreement with human judgments, thereby establishing a reliable benchmark for evaluating I.LM performance across a wide spectrum of open-ended tasks.
RQ3: What patterns exist in the relationship between individual and cross-capability perfor- mance in I.Ms? Through extensive evaluation using CROSSEVAL, we uncover clear patterns in the relationship between individual and cross-capability performance. Most notably, cross- capability performance is typically constrained by the weakest capability, following the "Law of the Weakest Link" effect. This pattern is consistent across different LLMs and evaluators, sug- gesting that deficiencies in an individual capability can significantly limit overall performance in more complex tasks. Specifically, of the 58 cross-capability scores from 17 models, 38 fall below the individual capabilities, while 20 lie between the strong and weak, skewing towards the weaker. These results underscore the need for targeted optimization to strengthen the weakest capabilities, especially in areas like Tool Use, where models struggle the most.
RQ4: How do shifts in individual capabilities impact cruss-capability performance in LLMs? Beyond evaluating the static relationship between individual and cross capabilities, we in- vestigate how altering individual capabilities affects cross-capability performance. Through case studies using a principle-baser based system prompting method, we selectively enhance specific capabil- ities and find that improvements in weaker capabilities lead to substantial gains in cross-capal poss-cappbillty tasks, while changes in stronger capabilities often result in only minor shifts. This finding further corroborates the "Law of the Weakest Link" effect, as an LLM's cross- capsblaapyblltyy performance continues to conform to this phenomenon when its individual capability performance changes.
In summary, this paper highlights the critical oversight of cross capabilities in LLMs, despite being essential for real-world tasks. To systematically explore it, we establish a comprehensive benchmark to model both individual and cross capabilities, revealing the "Law of the Weakest Link" effect. Given that LLMs generally underperform in cross-capablle cross-capability tasks, identifying and enhancing these weak points should be priority for future research and development.
2

---

Published as a conference paper at ICLR 2025
Image & Reasoning
Image
Reasoning
(a) Image Recognition
(b) Reasoning
(c) Image & Reasoning
Figure 1: Taxonomy visualizations for Image Recognition, Reasoning, and the corresponding cross capability. Each node represents specific type of task. The first two taxonomies illustrate tasks that require only individual capabilities for LLMSs to complete. The final taxonomy, however, depicts tasks that lie at the intersection of Image Recognition and Reasoning capabilities, necessi- tating the use of both abilities to accomplish them. For the full taxonomy of all the individual and capabilities and cross capabilities, please see Appendix B.
DEFINING INDIVIDUAL & CROSS CAPABILITIES IN LLMS
Real-world interactions with LLMs encompass tasks that may require either individual capability or the simultaneous engagement of distinet skills. To evaluate LLMs effectively, defining and dif- ferentiating these capabilities is crucial. In this section, identify seven individual and seven cross capabilities that reflect a broad spectrum of user queries and organize them into taxonomies.
As illustrated in Figure the taxonomies follow a hierarchical structure: the represents in- dividual or cross capability, while the next two layers (Level-1 and Level-2) break them into more specific tasks. This framework helps distinguish tasks based on single or multiple capabilities, en- abling comprehensive LLM evaluations. We next detail the selected capabilities.
2.1 INDIVIDUAL CAPABILITIES
We begin by selecting seven core individual capabilities LLMs: English, Reasoning, Coding, Image Recognition, Tool Use, Long Context, and one representative of multilingual capabilities, Spanish. Each of these defined as follows:
English: The ability understand, generate, and manipulate English text for tasks such factual answering, procedural instructions, creative writing, and dialogue.
The ability perform logical deduction, mathematical computation, analytical problem-solving, and domain-specific reasoning, including areas such as scientific, legal, and ethical contexts.
Reasoning:
Coding: The ability to write, synthesize, document, debug, and review code across various pro- gramming languages.
Image Recognition: The ability to perceive, identify, interpret, and describe visual information. Tool Use: The ability to interact with external tools and APIs, including functions such as web browsing, code execution, and file uploads.
Long Context: The ability process, comprehend, and synthesize information generate sponses from extensive textual inputs, ranging from 16K to 128K tokens.
Spanish: The same as the English capability, but for the Spanish language.
The selection of individual capabilities is primarily based on the capability evaluations outlined for post-trained LLMs in Llama 3 (Llama Team, 2024), as these represent key areas interest within the LLM community. Additionally, we include Image Recognition to account for the expanding mul- timodal capabilities of state-of-the-an LLMs, ensuring that our findings are applicable across both language and vision modalities. Appendix B.I provides the full taxonomy of individual capabilities.
3

---

Published as a conference paper at ICLR 2025
2.2 CROSS CAPABILITIES
We explore cross- casaabiiyy scenarios by combining two capabilities. To achieve this, we pair the individual capabilities described earlier and select seven common combinations: Coding & Reason- ing. Image Recognition & Reasoning, Tool Use & Coding, Tool Use & Reasoning, Long Context & Coding, Spanish & Reasoning, and Spanish & Image Recognition. Below is the Level-1 taxonomy:
Coding & Reasoning: Coding Q&A (Text to Text) (5), Code Explanation (2), Programming
Assistant (5), Mathematical Calculation (7).
Image Recognition Reasoning: Diagram Understanding (3), Chart Understanding (3), Text-
Rich Understanding (2), and Visual Math and Science (2).
Tool Use & Coding: Code Execution (3); Code Debugging with Execution (2), Programming
Assistant with Execution (1), and Code Execution with File Uploads (3)
Tool Use & Reasoning: Mathematical Reasoning (2), Scientific Reasoning (15), and Mathemati-
cal Calculation (13).
Long Context & Coding: Repository-Level Code Generation (5), Repository-Leve Code Un- derstanding (2), Repository-Level Code Debugging (1), Log Analysis (3), and API Does Under- standing (2).
To clarify, the number in parentheses represents the Level-2 subcategories within each Level-1 cate- gory. To select cross-capability scenarios, we ensure comprehensive coverage of individual capa- bilities and analyze the frequency of capability pairs using data from public real user queries (Zheng et al., 2024; Lin et al., 2024). This analysis enables us to identify the seven most common and practically relevant pairs. The full taxonomy of cross capabilities is available in Appendix B.2.
CROSSEVAL BENCHMARK CONSTRUCTION
This section outlines the annotation process for building the CROSSEVAL benchmark and the con- figuration of the LLM used its evaluator.
3.1 PROMPT SET ANNOTATION
The prompt set forms the foundation for benchmarking L.Mss, but real-world prompts often contain low-quality inputs, making it hard to distinguish advanced models (Li et al., 2024). Additionally, creating sufficiently difficult prompts is inherently challenging (Padlewski et al., 2024). To address this, we adopt a comprehensive annotation process to ensure both quality and difficulty levels.
Annotation Procedure. In this paper, we focus on single-turn, open-ended prompts. The annotation process follows ''category-fite prompt-second" approach to ensure comprehensive coverage of our taxonomy. Specifically, annotators from the data vendor are instructed to randomly select a leaf node from the taxonomy, which determines the category to be annotated. Once the category selected, annotators create prompts aligned with the provided guidelines. Each capability has clear criteria for three difficulty levels-easy, medium, and hard-detailed in Appendix D.6. For Spanish as an individual capability, all prompts are annotated from scratch, without overlap with the English set. In cross-capability scenarios involving Spanish, the prompt sets are derived by translating the associated English-basse based prompts. For instance, the Spanish & Reasoning set is created by translating the Reasoning prompts into Spanish.
To maintain consistency and high quality, we begin with pilot annotation phase where the authors act as reviewers, providing feedback to identify any issues with the initial annotations and refine the annotation guidelines accordingly. Afterward, the main annotation phase begins, resulting in 100 to 500 prompts for each capability, depending on the size of the annotator pool assigned to it. Reviewers then perform quality checks and apply filtering to produce final set of 100 high- quality prompts per capability. This process ensures the difficulty distribution follows the standards used in Llama 3's human evaluations, with 10% easy, 30% medium, and 60% hard prompts (Llama Team, 2024). Ultimately, the final prompt set consists of 1,400 prompts, with 100 prompts for each capability, covering all 76 Level-1 and 332 Level-2 categories as listed in Section $2.
4

---

Published as a conference paper at ICLR 2025
3.2 MULTIPLE REFERENCES WITH HUMAN ANNOTATIONS
Providing gold reference for each instance, while standard before the rise of LLMs. is not feasible for our prompt set for three reasons: 1) Open-ended queries often lack a single correct answer, and relying on one reference risks introducing evaluation bias. 2) Prompts requiring domain expertise, such as coding or mathematics, are sometimes too difficult even for college-level expert annotators. 3) For tool-use prompts, like "What is the temperature in the Bay Area today?," the correct response is dynamic and changes over time. To address this, we propose using multiple model responses, scored and explained by human annotators, to serve references for evaluation.
Annotator Qualifications. For all annotations in this paper, we use the same data vendor, employ- ing professional experts with domain-specific knowledge, such reasoning. coding, and Spanish. The data vendor selects the appropriate annotator pool based on the capabilities being evaluated. While creating a definitive gold reference is impractical, our annotators are capable of assessing the correctness of model responses and providing well-justified ratings.
Model Response Collection. For each prompt, we aim to gather three distinct model responses representing varying levels of quality: low, medium, and high. These responses are randomly drawn from various models within the Llama and GPT model families, including Llama 3.1 8B/70B/405B and different versions of GPT-4. For capabilitiees involving Reasoning, Image Recognition, and Tool Use, we manually annotate one response if all three collected responses contain noticeable errors.
Annotating Human Ratings with Explanations. For each model response, two independent anno- tators rate it on 1-5 Likert scale, providing paragraph to explain their rating. Multiple reference examples are offered in Appendix D.5. We track inter-rater agreement and find that evaluating model responses can be challenging, even for expert annotators, making consensus difficult to achieve.
To enhance consistency, we initially annotate 30% of the prompt set in pilot phase. During this phase, the inter-rater agreement 33.65% with a Krippendorff's Alpha (K-Alpha) (Krippendorff, 2018) of 0.48, indicating relatively poor agreement. We then conduct the second and third rounds of annotation, allowing new raters from the same pool to review previous annotations, better understand the scoring criteria, and provide their ratings with explanations. After each round, we update the guidelines to improve the annotation process. This iterative procedure proves effective: inter-rater agreement improves from 33.65% 45.79% and finally to 47.38%, while K-Alpha increases from 0.48 to 0.66, and eventually to 0.73. After completing these rounds. we apply the updated guidelines annotate the full dataset using the same trained annotator pool. On the full dataset, the inter-rater agreement rate reaches 54.93%, with K-Alpha of 0.76.
For comparison, in Chatbot Arena (Zheng et al., 2023), the human agreement rate is 81% for binary classification (win/lose) and 63% for 1-3 scale (win/tie/lose). In contrast, we independently score each response on a more granular 1-5 scale, yet still achieve a substantial level of agreement.
CrossEval Benchmark Statisties. The final CROSSEVAL benchmark comprises 1,400 prompts across 14 capabilities. 4,200 reference model responses, and 8,400 human ratings with accompany- ing explanations. Additionally, we provide several examples of the prompt set, along with human ratings and explanations, Appendix D.4 and D.5, respectively.
3.3 BUILDING LLM- -ASSDA EVALUATORS
CROSSEVAL is also the largest benchmark available for measuring the correlation between LLM rating and human judgments. With each prompt containing three responses and six human ratings, it enables the exploration of the most effective in-domain LLM evaluators for this benchmark.
3.3.1 PROMPTING LLMS FOR EVALUATION
While the LLM-as-a-Judge paradigm has gained popularity (Zheng et al., 2023), there is no stan- dardized method for designing prompts for guiding LLMs output evaluation scores. Common practices include generating an answer first, setting evaluation rules manually, and then instructing the model to assign score to the response being evaluated (Zeng et al., 2024).
In practice, we find that self-generatee answers frequently lead to issues. For instance, response length can exceed model limits, preventing the model from generating score. This approach also
5

---

Published as a conference paper at ICLR 2025
Capabilities
GPT-4o mini
Llama 3.1 405B
Clande 3,5 Sonnet
GPT-4o-05-13
English
0.383
0.452
0.516
0.498
Reusoning
0
0.699
0.704
0.731
Coding
0.627
0.568
0.599
Image Recognition
0,576
0.733
0.760
Tool Use
0.587
0.609
0.683
0.629
Long Context
0.405
0.500
0.609
0.594
Spanish
0.552
0.536
0.596
0.594
Coding & Reasoning
0
0.600
0.623
0.664
Image Reasoning
0.775
Tool Use Coding
0.484
0.545
0.588
0,639
Tool Use Reusoning
0.642
0.698
0.665
0,729
Context & Coding
0.524
0.535
0.620
0.593
Spanish Reasoning
0.691
0.734
0,715
0.772
Spanish Image
0.752
0.669
Overall Pearson
0.696
0.697
Overall Spearman (r,)
0.609
0.676
0.679
Kendall
0.550
Table 1: Correlations between LLM ratings and human judgments. The top section shows Pearson correlations across individual and cross capabilities for four LLMs, and the bottom three shaded rows present the overall correlations.
causes the LLMs to overly rely on their own generated answers, overlooking valuable insights from human-annotated references. To solve this, propose the following prompting strategy:
ReweRReeeeeeeaaseeeeee Prompting. We provide the 1-5 Likert scale rubrics in the system prompt (see Table 34 in Appendix D.7), followed by any relevant attachments (e.g., image) and the user prompt. When evaluating LLM-as-a-Judge performance, up to two reference responses with scores and explanations can be included. For example, when evaluating a medium-quality response, we provide low- and high-quality responses with their ratings for context. For new model evaluations, all three model responses are included with human annotations references.
Point Deduction-bassed Prompting. As noted in prior studies (Zheng et al., 2023), LLM-as-a-Judge often favors longer, more structured responses, leading to inflated evaluation scores. To mitigate this, we no longer have LLMs directly generate their own answers and assign scores. Instead, they summarize issues both the reference examples and the evaluated response, specifying point deductions (Zhong et al., 2024). This approach ensures more balanced and systematic evaluations. The complete prompt is detailed in Table 35 in Appendix D.7.
3.3.2 CORRELATIONS WITH HUMAN JUDGEMENTS
To assess the effectiveness of the LLM evalua- tor CVOSSEVAL, conduct experiments with four advanced LLMs: GPT-4o mini, Llama 3.1 405B (Llama Team, 2024), Claude 3.5 Sonnet (An- thropic, 2024), and GPT-4o (OpenAI, 2023). For each prompt, provide two reference examples and ask the LLM to evaluate the third, comparing model-based scores with the average human rating. We evaluate 4,200 samples across 14 capabilities, with correlations shown in Table
Correlations
0.4
Pearson
Spearman
Kendall
Fwo Ref low/
Ref Btw/ 2 Refs
Each LLM demonstrates strengths in different ar- eas. For example, Claude 3.5 Sonnet excels in eval- uating Tool Use, Image Recognition & Reasoning, and Spanish Image Recognition, while GPT-4o performs best in cross capabilities like Coding & Reasoning, Tool Use & Coding, and Spanish & Reasoning. For comparison, the recent BigGen Bench (Kim et al., 2024) reports a Pearson correlation of 0.627 with gold references, while our score approaches 0.7, showing that multiple reference examples still enable reliable evaluations without gold reference. GPT-4o achieves the highest correlations overall, uator, with results from Claude Appendix E.2 for reference. Further details and discussions on evaluator selection are available in Appendix D.8.
Figure Ablation study on the number of reference examples.
select it the final eval-

---

Published as a conference paper at ICLR 2025
Individual Capabilities
Models
English
Reasoning
Coding
Image
Tool Use
Long Context
Spanish
GPT-4o mini
73.64
69.31
71.17
65.23
76.15 18.18
74.51
GPT-4o
76.12
72.84
72.03
73.02
77.17
78.10
80.70
o1 preview
78.59
82.30
79.09
90
79.64
Claude Haiku
63.87
$6.81
61.64
$1.00
69.68
67.95
Claude 3 Sonnet
69.19
62.88
66.09
56.56
72.40
69.43
Clande Opus
66.22
69.68
61.76
74.69
74.01
Clande 3.5 Sonnet
75.00
71.54
74.01
68.57
74.32
76.12
Gemini 1.5 Flash
66.59
63.25
65 60
56.81
73.52
700.05
99.56
Gemini 1.5 Pro Exp
75.87
73.02
71.17
75.37
76.24
Reka Edge
52.23
45.30
39.36
48.89
33..0
$1.48
Flash
Roka, Core
71.54
68.60 69
62.38
56.94
60.90
73.77
Dlama 3.1 AB
64.11
53.97
55 06
42,00'
559.53"
35, 70
Llama 3.1 70B
62.88
65.47 .7.4
47.04 47.90
64-48
Llama 3.1 405B
73.52
69.31
69.19
09.31
72.59
Cross Capabilitics
Models
Coding & Rea.
Image Rea. Long & Coding Spanish & Rea.
Spanish & Image Tool & Coding
Tool Rea.
GPT-4o mini
72.03 73.33
65.60 71.29
65.10
69.56
65,10 74.63
GPT-4o
67.95
73,52
45.80
54.41
o1-mini ol -repevveww
76.12
79.83
79.58
73.39
80.70
Clande Haiku
49.88
58.67
$7.80
$2.85
Clande 3 Sonnet
61.14
$4,71
58.79
60.77
60.52
Clande Opus
63.37
53.84
58.17
67.33
64.11
Sonnet
69.81
"Gemin 1.3 Flash
64.73
31.74
62.13"
66.110
53:10
Gemini 1.5 Pro
69.68
67.95
65.97
69.56
62,26
Gemini 1.5 Pro Exp
65.97
71.54
70.18
Reka Edge
41.34
28.60
20.43
40.97
45.06
Reka. Flash
59.66
55,82
Reka Core
63.62
46.66
68.01
54.71
"llitta 3,1 8B
33,088
"45:06" "
46.42
46:911
$3.882
Llama
50.50
Ilama 3.1 405B
66.96
54.58
64.48
$2.23
51.74
Table 2: Experimental results on the CROSSEVAL benchmark. cross-capability evaluations, define one of the involved individual capabilities stronger and the other as weaker if the absolute score difference between them exceeds A = points. In 58 cross-capability scenarios where this difference is present (indicated by colored background), 38 cases show performance lower than both individual capabilities red background ), and 20 show performance between the two but closer to the weaker capability blue background Notably, no cross-capability score ever comes close to or exceeds the stronger individual capability.
Discussion on Reference Examples. Given the substantial effort invested in collecting and annotat- ing reference examples, ensuring their effectiveness for evaluation is crucial. To this end, we conduct ablation studies with GPT-40 assess how the number of reference examples impacts the correla- tion metrics, as illustrated in Figure 2. A clear trend emerges: as the number of reference examples increases, all three correlation metries improve significantly. For example, the Pearson correlation starts at 0.578 with no reference examples, rises to 0.655 with one reference, and reaches 0.697 with references. Notably, when evaluating new responses on CROSSEVAL, we include three reference examples, potentially achieving even higher correlations for more accurate evaluations.
4 "LAW OF THE WEAKEST LINK" OF CROSS CAPABILITIES IN LLMS
In this section, explore the relationship between individual and cross capabilities in LLMs. We first present the experimental setup. followed by a detailed discussion of the findings.
4.1 FINDINGS ON THE CROSSEVAL BENCHMARK
To better present the results, we linearly map the average scores for each capability from a 1-5 scale to 1-100 scale. The full results of 17 LLMs provided in Table 2. Since LLMs tend prefer self-gent reffgeeerated answers (Zheng et al., 2023). we exclude GPT's results from the comparative analysis and treat them reference point. Our experiments reveal several key findings:

---

Published as a conference paper at ICLR 2025
Informaiee Cxppaliity 325
Muat caas Story Copatilty
3X
Density
0336
022
2.66
045
GPT-as-a-Judge
Claude-as-a-Judge
Figure 3: Density distribution of cross-capability performance compared to the two individual capabilities. The plot illustrates pronounced "Law of the Weakest Link" effect in LLMs, where performance in cross-capability tasks tends to cluster around the weaker individual capability.
CROSSEVAL effectively differentiates advanced models. The CROSSEVAL benchmark success- fully distinguishes between state-of-the-art LLMs. For example, the four Claude model variants achieve progressively higher reasoning scores: 56.81, 62.88, 66.22, and 71.54, reflecting the in- creasing capabilities larger parameter models (Haiku = Sonnet = Opus) or updated versions (Claude 3 = Claude 3.5). Similar trends observed across all model families, demonstrating that CROSSEVAL is capable of capturing subtle performance differences across capabilities.
LLMs exhibit "Law of the Weakest Link" effect in cross capabilities. To better understand how individual and cross capabilities interact, we identify "strong" and "weak" capabilities within cross-capability tasks when the absolute difference between their individual scores exceeds A 3. In all cases where distinct strong and weak capability is present, cross-capability performance either matches slightly underperforms the weaker capability. This suggests that performance on tasks requiring multiple abilities is constrained by the weakest component, a phenomenon closely aligned with the "Law of the Weakest Link" (Liebig, 1840). Much like a barrel's capacity limited by its shortest stave, the weakest capability in LLMs dictates overall performance.
"Law of the Weakest Link" is evaluator-agnostic. strong and weak capability scores to a standardized scale from -1 to I and plot the density of cross- capability performance relative to these scores. score below -1 indicates that the cross-capability performance falls below the weaker individual capability, while 0 represents the average of the two. As shown Figure 3, the "Law of the Weakest Link" holds true regardless of the evaluator used. With GPT, the density peaks slightly below the weaker capability, while Claude peaks slightly above it. However, in both cases, performance clusters closely around the weaker capability. Moreover, explore varying A values E.3, consistently confirming the "Law of the Weakest Link."
further validate this effect, we normalize
Given that many real-world tasks require integrating multiple capabilities, this finding offers key in- sights for future L.M development. The "Law of the Weakest Link" suggests that deficiencies in one capability can substantially limit performance across any tasks involving that capability. CROSSE- VAL provides foundation for identifying LLM weaknesses, but further research needed more comprchensively diagnose and address these deficiencies without compromising other capabilities.
Tool Use is currently the most challenging capability for LLMs. Among the capabilities tested, Tool Use stands out as the most challenging. Our prompt set includes tasks involving web browsing and code interpretation, and Llama 3.1 is the only model family that currently supports both. How- ever, even Ilama 3.1 405B struggled with Tool Use, scoring below 50 on this individual capability and only slightly above on tasks combining Tool Use with Coding or Reasoning. These scores significantly lower than those for other capabilities, indicating critical area for improvement.
LLMs underperform in cross-capability tasks. Despite our efforts to maintain a consistent dif- ficulty level across both individual and cross-capability tasks, LLMs generally perform worse tasks requiring multiple capabilities. For instance, in the Spanish & Reasoning and Spanish & Im- age Recognition tasks, where prompts direct translations from their English counterparts, the models underperform in most cases compared to individual capabilities. Across all models, the average score for individual capabilities is 65.72, compared 58.67 for cross capabilities, reveal-
8

---

Published as a conference paper at ICLR 2025
Individual Capabilities Image Recognition
Cross Capabilities Spanish Rea.
Models
Reasoning
Spanish
Image & Rea.
Spanish & Image
Claude Haiku
56.81
51.00
67.95
49.88
57,80
52.85
Reasoning
50.01
52.11
Image
55.45
$4.71
64.98
54,46
57.55
55.08
Spanish
55.20
53.59
67.21
50.13
56.81
53.72
Gemini 1.5 Flash
63.25
56.81
70.05
31.74
63. 15
53.10
Reasoning
66.71
62.50
71.29
54.46
59.04
Image
59,91
63,00
69.43
$1.61
62.13
61.76
Spanish
Table 3: Investigating the impact of -aaiaiiuualcapaabiiity alterations on cross-capability per- formance. "+ X" indicates the application of principle-based system prompting to enhance the specific capability X. Results demonstrate that enhancing weaker capabilities leads to more substan- tial improvements in cross-capability performance compared to enhancing stronger capabilities.
ing a significant performance gap. This disparity demonstrates that current LLMs remain heavily optimized for individual capabilities, with a limited focus on cross-capabilities performance.
INVESTIGATING INDIVIDUAL-CAPABIIA ALTERATIONS
Beyond evaluating the static capabilities of LLMs, we explore the erucial follow-up question in this section: how altering specific capabilities affects overall cross-capability performance?
5.1 PRINCIPLE-BASHE SYSTEM PROMPTING
To explore the impact of altering individual capabilities without significantly affecting others, we propose a principle- baseed method that iteratively refines the system prompt to selectively boost spe- cific capabilities of LLMs, based on the responses and evaluations on CROSSEVAL. This approach allows controlled investigation into cross-capability performance dynamics.
Our method iteratively refines the prompt through operations such as adding, replacing, revising, or keeping principles. After 100 iterations, GPT-4o generates a tailored principle-based prompt to guide the LLM in prioritizing key performance aspects like format adherence, problem-solving strategies, or error avoidance. Detailed prompts used are available Table 41 in Appendix F.I.
5.2 CASE STUDY FOR INVESTIGATION
Experimentally, we select three tasks with the largest gaps relative to individual performance: Image Recognition & Reasoning, Spanish & Reasoning, and Spanish & Image Recognition. We also focus on two LLMs, Claude Haiku and Gemini 1.5 Flash, which display the most significant discrep- ancies in these tasks. These combinations clearer insights into the impact of selective capability enhancement on overall performance. Table shows the full results, leading to the following key observations:
chosen because the more pronounced gaps provide
Principle-based system prompting is particularly effective in enhancing weaker capabilities. For Reasoning capability, both models show substantial improvements: Claude 3 Haiku increases by 2.85 points, and Gemini 1.5 Flash improves by 3.46 points. The improvements are even more pronounced in Image Recognition, with Claude 3 Haiku improving by 3.71 points and Gemini 1.5 Flash by 6. 19 points. These results indicate that the principles derived from the CROSSEvAL eval- uation process offer effective guidance for enhancing weaker LLM capabilities, even when applied solely as system prompts. However, for stronger capabilities like Spanish, the same method shows limited efficacy, suggesting that refining already-strong capabilities is more challenging.
"Law of the Weakest Link" effect persists after individual-capaaiiiiy alterations. Our case study also confirms that performance shifts in individual capabilities continue to conform to the "Law of the Weakest Link" effect. Specifically, altering the weaker capability in cross-capability scenario has a significant effect on overall performance, while changes to the stronger capability result in only minor adjustments. For example, in the Image Recognition & Reasoning scenario with Claude's 3 Haiku, when we introduce a system prompt focused on reasoning, the stronger ca- pability (Reasoning) improves by 2.85 points, but the weaker capability (Image Recognition) drops

---

Published as a conference paper at ICLR 2025
by 0.99 points, leading to overall performance decrease of 3.46 points. Conversely, when an image-related system prompt is added, the weaker capability improves by 3.71 points, the stronger capability decreases by 1.36 points, but the cross-capability performance increases by 4.58 points.
In 10 out of the 18 cross-capability scores examined across the two models, we observe one indi- vidual capability improving while the other declines. Notably, in 90% of these cases, changes in cross-capability performance closely follow the trends of the weaker capability. This strong align- ment with the "Law of the Weakest Link" underscores the importance of addressing the weakest links in LLM capabilities to drive meaningful improvements in complex, real-world tasks.
Conclusion of the case study. These case studies offer further insights into how LLMs conform to the "Law of the Weakest Link". We show that targeted enhancement of weaker capabilities results in more significant improvements in cross-capability performance than focusing on stronger capabilities. Since LLMs underperform in cross-capability tasks, prioritizing the identification and enhancement of the weakest points should be a key focus for future research and development.
6 RELATED WORK
Evaluation of LLMs. The advancements in LLMs have shifted the focus of evaluation from tra- ditional NLP tasks (Wang et al., 2019b;a) to specific capabilities such as reasoning (Clark et al., 2018; Hendrycks et al., 2021a;b; Rein et al., 2023), coding (Chen et al., 2021; Austin et al., 2021; Cassano et al., 2023; Liu et al., 2023a), multilinguality (Shi et al., 2023), tool use (Srinivasan et al., 2023: Patil et al., 2023; Li et al.. 2023; Yan et al., 2024), long context (Shaham et al., 2023; Kam- radt, 2023; Zhang et al., 2024; An et al.. 2024), image recognition (Yue et al., 2024), instruction following (Zhou et al., 2023), mastering domain-specific knowledge (Hendrycks et al., 2021a), and weakness identification (Chen et al., 2024). Moreover, benchmarks like BiGBench Bench assess range of abilities across multiple tasks but still target individual capabilities in isolation (Lin al., 2024; Kim et al., 2024). Although LLMs are capable of handling increasingly complex tasks, cross- capability evaluation remains underexplored. Our work fills this gap by systematically investigating these crucial yet often overlooked cross capabilities.
Another emerging area is the evaluation of LLM-based agents, which inherently require cross ca- pabilities to function effectively in real-world applications. Unlike standalone LLM evaluation, which focuses on specific skills, agent assessment emphasizes the overall success rate in completing tasks (Yao al., 2022; Zhou et al., 2024; Koh al., 2024; Liu et al., 2024; Xie et al., 2024) or executing particular actions (Deng et al., 2023; Ma et al., 2024). While CROSSEVAL VAL is designed for LLMs rather than agents, it still encompasses key agent-related capabilities such as modalty, multilingualism, and tool use. Furthermore, it provides a comprehensive distinction between indi- vidual and cross capabilities. providing more granular framework for evaluation and analysis.
Evaluation Metrics for Open-Ended Generation. Evaluation metrics have evolved alongside ad- vances in model generation capabilities, moving from n-gram-baaae based measures (Papineni al., 2002; Lin, 2004) to pre-trained language model (PI.M) 2020; Yuan et al., 2021; Zhong et al., 2022) and, more recently, to L-LM-as-a-Judge frameworks (Liu et al., 2023b; Zheng et al., 2023). Given the large set of complex, open-ended prompts in our bench- mark, we employ LLMs as evaluators to assess model outputs. Unlike previous methods that rely on self-generated prompts, we adopt a point deduction- based prompting technique. Each instance is supported by three expert-annotated references to ensure reliability. Furthermore, CROSSEVAL is also the largest benchmark available for measuring the correlation between LLM ratings and human judgments, providing insights into which capabilities different LLMs excel at evaluating.
evaluators (Zhang et al., 2020; Sellam et al.,
CONCLUSION
We systematically investigated the cross capabilities of LLMs by introducing CROSSEVAL, a testbed designed to evaluate both individual and cross capabilities. We also developed an LLM-based judge that showed strong agreement with human judgments. Our experiments revealed that LLMs consis- tently follow the "Law of the Weakest Link," where cross-capability performance is limited by the weakest ability, even after enhancing individual abilities. Our benchmark and findings highlight the importance of focusing cross-capability development and evaluation in future LLM research.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alfred Adler. Study of Organ Inferiority and Its Psychical Compensation: A Contribution to Clinical Medicine. Nervous and Mental Disease Publishing Co., New York, 1917. URL https://
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 14388-14411, Association for Computational Lin- guistics. 2024. URL https: cestps://aclantiaan htt/a/aaaaanttoooooeoooooo org/202///2 aew. 1oneelloon... 776.
Philip W. Anderson. More is different: Broken symmetry and the nature of the hierarchical struc- ture of science. Science, 177(4047::393-396 1972. doi: 10.1126/science. 177 4044.99.. URL.
https:/ / /wwwww scw.nce..ecccccoccc org //t///////// /tw//o..1...0 1126/ /sssicinnce... 177. 4047.393.
Anthropic. Introducing the next generation of claude, 2024. URL https: http:/wwttttt.t...... www.anthropic.
com/news/clauee 3-mamde---y
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Pro- gram synthesis with large language models. CoRR, 0as/2108.07332. 2021. URL https: 07732.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinekney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and polyglot approach to bench- marking neural code generation. IEEE Truns. Software Eng., 49(7):3675-3699 2023, doi: /TE.. 2023 3267444
10. 10.1109/TSE.222333377777 URL https:tth/ocmm
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Horbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https: arxiiv www.vvvoo..oooooo.........oooo.r abs/ 03374.
Yulong Chen, Yang Liu, Jianhao Yan, Xuefeng Bai, Ming Zhong, Yinghao Yang, Ziyi Yang, Chenguang Zhu, and Yue Zhang. See what Ilms cannot answer: A self-challenge framework for uncovering Ilm weaknesses. In First Conference Language Modeling, 2024. URL
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. CoRR, Abs/1883.00557. 2018. URL
www.ariwww....aaaaaaaaroeeeeeeee org abs 1801181330 05457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichir Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168. 2021. URL
https:/ /ttps://arxiv. wwtpiiivorrg/eee argg/abs/////oooooooooooo///o / 2210. 14168.
Peter A. Corning. The Synergism Hypothesis: A Theory of Progressive Evolution. McGraw- Hill, New York, 1983. ISBN 0070131724.
URL https:/ www.aiii...o//eeteaaaeeeeeeeeeee
synergi smhypothe0000comm
11

---

Published as a conference paper at ICLR 2025
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Alice Oh, Tristan Nau- mann, Amir Globerson. Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- fion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers: nipp cc paaper_file//eeeeeeeeepeeeeesest pttees//ppp///2233 Ben.charkks html.
Abstract Datasets_ant and
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- neew.net, 2021a. URL httpp:://oee.revieett net www.oooiddddiii jmI 3GmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021b. URL http:///ttttteeeeeeeeeaaeeeeeeee www.eeeedddngssneuiisssese. nwwrineu.i.... wweurips.cc/ppppr... wttpccppppppp...../ee//pppppp//e hash/be8 htt.3333ccc000707666611111111100 -as--bbsttrtttte rouaatttround
Gregory Kamradt. Limtest.nccdeeimhaassssk,,
2023.
URL https://gi http:/gtttttuttttt/ thub com/ md.
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Choi, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. The biggen bench: A principled benchmark for fine-graing grained evaluation of language models with language models. CoRR, abs/2406.00 05761, 2024. doi: 22550/arxi:1 arwiv. 2406 05761.
URL https://
doi. woii.org/10.. 48550/
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yo Yu Huang, Gra- ham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evalu- ating multimodal agents on realistic visual web tasks. In ACL 2024, 2024. URL https:
wwwpl/lyoyooogg/oo/e acl-long.50
Klaus Krippendorff. Content Analysis: An Introduction Its Methodology. Sage Publications, Thousand Oaks, CA, 4th edition, 2018. URL https::/us..aggpu. comb ette//n-uus/e uus uumnmm/u//m//////// nam/
content- mmalysis/book258-.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tugl-aug 1lms. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Confer- ence Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, De- cember 6-10, 2023, pp. 3102-3116. Association for Computational Linguistics, 2023, doi: 0T1/55//112223.MMMPPPP emn.p-matle
URL https::/t///......t. /ooi wwt. http:/00..665//11222 http::::://ooooooo/.1118863/////
1m 187.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E, Gon- zalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. CoRR, abs/2406. 11939, 2024. doi: 10.4850/ARXXV2246.1111 https:://oi. www.i/orrr....0 http//...855//// arxiv. 24..2
URL
Justus Freiherr von Liebig. Die organische Chemie in ihrer Anwendung auf Agriculur und Phys- iologie. F. Vieweg und Sohn, Braunschweig, 1840. URL battss://book: google. com/ books ?id- 6XKSAAAAYAAJ.
Bill Yuchen Lin, Yuntian Deng, Khyathi Raghavi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms
12

---

Published as a conference paper at ICLR 2025
with challenging tasks from real users in the wild. CoRR, abx/2406. 2024. doi: 10.48550/ ARXIV.2406.04770. URL https: htt///oioooorooo///ooooooooo.oo. 10. 400.0.4///0.......
04770.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, 2004. URL attpps:/aclaatthollee wwwp:ethhhwooo....oo...oooo wwtp////g//000000
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a. URL htttt:://papaee hash/ e 9d64 Seccd3e44o7bbbabbb5ee -tt.aaaeesaaaaa00eeeeeasaa----oa
nips. cc faapper_filesppppppe Abstract ceeerraat-oooffee Coneeeeceeeeeeeeeeee.eoooooooee. html.
Xiao Liu, Hao Yu, Hanchen Zhang. Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating iims as agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httts:///pperest
net/ forum? wetteoormmii=edeeeiitte d==ddddU02AAAAT
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 2511-2522 Association for Computational Linguistics, 2023b. doi: 10. 00.18533V1/20233 F20:5///1222..MMLPMMMMM... EMNLP-MAIN..55. URL https: //doi.org// 10. 18653/v662v2 / emn.p-mminnnniee
www.. 153.
Llama Team. The llama 3 herd of models. CoRR, abs/2407.21788, 2024. doi: 10.48550/ARXIV. 2407.21783. URL https: otg/:/toogggg../// 485404 wwtp::xi.......22.....
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn
LLM agents. CoRR, abs/2401.13117, 2024. doi: I048550ARK1W.240.1177.. URL https:
//doi. wwwpoo.g.......... 48550 http://1..855//aa/aaa. /1w./////....22..111......
OpenAl. GPT-4 technical report. CoRR, (bBBRBR2ab0033307777/// 2023, doi: 10.48500AARX1V2333.00777 URL httt:pps//ollooog1..185///////// /ttp://000550//a11..33333..... 200.
OpenAl. Openai ol system card, September 2024.
URL https://
71511382220075111197778822222221 09-18.
Accessed: 2024-
Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models. CoRR, abs/2404. 12387, 2024. doi: 10. 48550/ARXIV.2404, 12387. URL https: //t////ooooo org/ 48550/8550000 arxiv. 2404 12387.
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d' Autume, Dani Yo- gatama, Mikel Artetxe, and Yi Tay. Vibe-eval: A hard evaluation suite for measuring progress of 2024. doi: 10.4850/ARX1.2240000222 multimodal language models. CoRR, www.v.iiiii..... 2405....0 02287.
URL https: /oi. wwwo/o./.......... 48540/
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, 2002. doi: 1033155/073083.07733.... URL www.oog..oo..ooooo.
13

---

Published as a conference paper at ICLR 2025
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. CoRR, abs/2305. 15334, 2023, doi: 10448500ARRX1.235..553 URL httpt:h/tdooiii org 48550/ w.t.aaa.ivvv............ 2330.153333
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ay- oub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Za- heer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530. 2024. doi: 10.48550AARXI..4400.0555 https://dol org/10.............o...o........ 48550/ arxiv. 2403. 05530.
URL
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/2311. 2023. doi: 10:4350AARIV.2311.2222 //doi. www.orgg111.. 48550 (a/.5////aiii... 2311 12022.
URL https:
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault (eds.). Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7881-7892. Association for Computational Linguistics, 2020. doi: 10.18653/V1/2020..444 ACL-MAA.AL URL https: //doi. wtt..o....1/./////... 2020. acl main. 704.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero- shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singa- pore, December 6-10, 2023, pp. 7977-7989, Association for Computational Linguistics, 2023. URL https://dii.onoo wtt.://i....1111.1111111.11
doi: 10. FISSS//IIS22.. v1 w1/2023. www..disennnnngeel....... 536.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das. and Jason Wei. Lan- guage models are multilingual chain-of-thouge reasoners. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https::////........ wwtpoopeerrvveeeeeeee ew. net/ forum? wetorrmmmmdfffaaae (=ffffGCC- TXp.
Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Hanzi Mao, Damon Mosk- Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang. Nexusraven: a commerriilly-permisivee language model for function calling. In NewrIPS 2023 Workshop on Instruction Tuming and In- struction Following, 2023. URL https: operrevieeeevveeeeeeeeee wwwevvieeeeeeee hww.neetffrrutt forum? d
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 3261-3275, 2019a. URL https: httpt:://roceeeeeeeeee http://poooee...neeee neurips.... hcc/paper//019//// 20//0//hhhhh///// Abstract. html.
Neural Information Pro-
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019b. URL https:/ /ttp://////peeeieew/ net forummidd di r54km225t7.
14

---

Published as a conference paper at ICLR 2025
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. CoRR, abx/2404.07972. 2024. doi: 10. 48550/ARXIV.2404.00
URL https::/p//......... //oi http://toooor//////////o htrg/10.48550111 485444555//// wttp//a................ .0000.007922
Fanjia Yan. Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, lon Sto- ica, and Joseph E. Gonzalez. Berkeley function calling leaderboard, 2024. URL
https:://oriin cs. berkeler.e.... wwereeeey.edu/blleee wewpb/llg////eeeeee/eeo/ooogooot Bttr///beeeeee__ueeeeeeee calling_leadeboaare html.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scal- able real-world web interaction with grounded language agents. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35; Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, hhahh//
2022, 2022. URL http::/papers.teeppp/// nips. cw/paper Cew-rr/----bbttaa---Coeeeeee
www.:icc.psp....e.eeeee.epppp www.ffiieeeeeeeeee html.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- mual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 14, 2021, virtual, pp. 27263-27277 2021. URL https: http:///ooeedeeecccccoooo ngs wwwrips.u...m ca/ paper 2022 /hw.h/hahh/////aa de 3ew.aaeeeeeecceeeeeeeeeee3eee
Abboddabbbsrratt.. html.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive musti-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567. 2024. URL https:://penaaee http:///wwwaiiiiiiiiipiieeeeeeee
thecvf..e...cc com/ wwwm///ooeet////oooooooo/oeettet www.onttee/c//22222/1/ Mttt:////ttm//i__MM__..... MMMU_A htwpreddaannndeeetnannnnnnn oorprsteeennnnnaanneeeeessssnn_. Benchmark__or_cvvR, Cvw.oor________e_____ 2024 paper. html.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representutions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL https: / wwwpeeeeeeeeeeeeeeepepe.ee net wttp://ww..ee//oommiiiieee.d
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Eval- uating text generation with BERT. 8th International Conference on Learning Representa- tions, ICLR 2020. Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: httepp://cpeeneeeee...... wwt/ffoummt htt://ooumiiiiiiieceeeese id=SketuCVEE..
Xinrong Zhang. Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. xobench: Extending long context doi: 10.48550/ARXIV22402.
evaluation beyond 100k tokens. CoRR, abs/2402.13711 2024.
13718. URL attpps://dos wwt.///.//..../..////
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: http:/aaaaarrseeeeeeeeeeee nips ec cc/ htcp/appr__fill files feaerrfflls/pppppp /eeeee/ppp///2///// hash/91 f1 8 2 Benchmarks html.
Liammin Zheng, Wei-Lin Chiang. Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E, Gonzalez, Ion Stoica, and Hao Zhang.
15

---

Published as a conference paper at ICLR 2025
Lmsys-chat-Ime A large-scale real-world LLM conversation dataset. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
net owtp://www....eeeoooaaaeeaaooaa. httpe//oommmoccccccEFFFF
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimensional evaluator for text generation. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2023-2038. Association for Computational Linguisties, 2022. doi: URL https::/p///....... /dw.i//ooioo/
10. 11166///1/2222.MMMMPPP22. emnip mwin. emr.ppmmiin.133 131.
Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang. Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. CoRR, abs/2402. 16843, 2024. doi: 10445550ARRXIW2402.116333 URL ptttps://do..rrg/// 48550/arXiv 2402.16843
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instraction-followinn evaluation for large language models. CoRR, abs/231 1.09911, 2023. doi: 10.48550/ARXIV2331..00 1.79111. URL https : /doi.ddi..///////// waciooorrggoooo /10/ 48550/arxiv. 2311. 07911.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A real- istic web environment for building autonomous agents. In ICLR 2024, 2024. URL https: / //openreveew net/ forum? det.rmmmoddoxtteyttyxxx
16

---

Published as a conference paper at ICLR 2025
Appendix Table of Contents
A Limitations
18
A.1 Reflections on Multilingualism
18
A.2 Reflections Multimodality
18
A.3 Reflections on Capability Relationships
18
A.4 Reflections Capability Comparability
19
Complete Taxonomy
20
B.1 Taxonomy of Individual Capabilities
20
B.2 Taxonomy Cross Capabilities
25
B.3 Details for Taxonomy Construction
28
C More Details on Capabilities
29
C.1 Principles for Defining Capabilities.
C.2 Distinction Between Capability and Skill
29
CROSSEVAL Benchmark
30
D.I Why Constructing the Benchmark From Scratch
30
D.2 Selection Process and Criteria for Prompts
30
D.3 Statistics of the Prompt Set
30
D.4 Prompt Set Examples
32
D.5 Reference Examples
44
D.6 Guidelines for Difficulty Levels
49
D.7 Prompts for Evaluation
51
D.8 More Discussions on LLM Evaluator
53
D.9 Case Study for LLM-as-a-Judge on CROSSEVAL
54
"Law of the Weakest Link" of Cross Capabilities in LLMS
56
E.1
Experimental Setup
56
E,2 Results for Claude-as-a-Judge
E.3 Discussion on Distinguishing "Weak" and "Strong" Capabilities
58
E,4
Results for Different Difficulty Levels
60
F
Investigating Individual-Capaabliy Alterations
61
F.1 Prompt to Generate Principle .....
F22 Case Study for Principle-based System Prompts
62
17

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
A.1 REFLECTIONS ON MULTILINGUALLIS
Our work contributes to the ongoing scholarly conversation about how to meaningfully evaluate multilingual capabilities in LLMs. In designing our framework, we face important methodological choices regarding the treatment of languages:
For English capabillities, we follow established conventions from prior work (Llama Team, 2024; Kim et al., 2024), treating English as foundational capability that encompasses knowledge-based QA and instruction following. This reflects the current reality of language model development, where English often serves as the primary training language and evaluation benchmark.
The field currently lacks consensus on how to conceptualize non-English languages in evaluation frameworks. Specific approaches (Llama Team, 2024; Kim et al., 2024) aggregate all -on-nnlishh languages single capability, while others, like Chathot Arena (Zheng et al., 2023), evaluate each language independently.
In our work, we select Spanish as a representative language for studying cross-capability interac- tions, following the approach used in SEAL LLM Leaderboards', This choice allows us to explore rich interactions between language and other capabilities (e.g.. Spanish Reasoning, Spanish & Image Recognition).
We acknowledge that this design choice introduces an important limitation: our findings on cross- capability interactions specifically pertain to Spanish and may not generalize to other languages with different linguistic properties, cultural contexts, or representation training data. Future work expanding this analysis diverse language families can provide more comprehensive understand- ing of how linguistic diversity interacts with other model capabilities. This represents an exciting opportunity develop more nuanced evaluation frameworks that better reflect the complexity of multilingual understanding.
A.2 REFLECTIONS ON MULTIMODALITY
This paper primarily focuses on pure-text scenarios, with 6 out of individual capabilities and out of 7 cross-capabilities being text-only. This focus is reflected in our title and conclusions, which specifically address LLMs rather than vision-language models. We include image recognition as representative multimodal capability explore whether the "Law of the Weakest Link" extends beyond text-only interactions, providing an initial reference point as statc-of-the-as LLMs increas- ingly support multimodal inputs.
Our findings regarding multimodal interactions represent preliminary explorations rather than com- prehensive evaluations of dedicated vision-language models. As the field continues to advance to- ward more sophisticated multimodal architectures, future work can expand this analysis to develop more nuanced understandings of cross-capability interactions across diverse modalities.
A.3 REFLECTIONS ON CAPABILITY RELATIONSHIIPS
Our exploration of capability interactions reveals an intriguing aspect of language model behavior that invites deeper consideration: the varying relationships between different capabilities. While we did not explicitly quantify these relationships, our work acknowledges their importance in under- standing cross-capability performance:
1) For individual capabilities: Our selection embraces a spectrum of relatedness that enriches analysis.
Some capabilities naturally share conceptual foundations (c.g., languages with similar linguistic structures), offering insights into how related skills influence one another.
Other capabilities represent fundamentally different cognitive domains (e.g., coding versus image recognition), allowing us observe how models bridge disparate knowledge systems.
'https://ee Pitps::/sccle com/1 htm//eacae Teaderbor rbooad
18

---

Published as a conference paper at ICLR 2025
2) For cross capabilities: Our choice of cross-capability scenarios is guided by two main criteria:
Coverage: Ensuring all individual capabilities are covered in capabiiapa scenarios.
Real-world relevance: Prioritizing combinations that frequently appear in actual user queries and applications.
Perhaps most compelling is how the "Law of the Weakest Link" emerges consistently across this varied landscape of capability relationships. This consistency suggests fundamental principle of current language models rather than artifact of particular selections. Future research might de- velop formal metrics for capability distance, potentially revealing how the strength of this effect varies with the conceptual proximity of different capabilities, direction that could significantly advance our understanding of emergent behaviors increasingly capable models.
A.4 REFLECTIONS ON CAPABILITY COMPARABILITY
In this work, we explore methods address the challenge of making different capabilities meaning- fully comparable, an ongoing question in LLM evaluation that requires careful consideration:
1) Connecting individual and cross capabilities: We design our evaluation so cross capabilities in- corporate their component individual capabilities, creating natural relationships for comparison. For instance, "Coding Reasoning" includes coding tasks, establishing connections that help illuminate how capabilities transform when combined.
2) Consistency in difficulty and style: We strietly control the difficulty and style distribution of prompts across all capabilities. By standardizing these factors, we ensure that comparisons valid.
3) No comparisons of unrelated capabilities: We do not compare unrelated capabilities with no overlap. as we agree that such comparisons would lack significance. Our evaluations focus on scenarios where meaningful relationships exist, such cross capabilities that naturally combine individual capabilities used in tandem for real-world tasks.
Consistent patterns: Across all evaluated scenarios, involving 14 capabilities and 17 models, our results consistently demonstrate the "Law of the Weakest Link." This robust and recurring observation reinforces the significance of our comparisons, revealing systematic trends in model performance and interactions between individual and cross capabilities.
We acknowledge, reviewers insightfully noted, that establishing perfect comparability between different capabilities requires more in-depth research. Human evaluation could provide valuable ev- idence for the validity of cross-capability comparisons by demonstrating consistent judgment stan- dards. Future work might incorporate domain experts to provide human labels that validate score comparability across different capabilities, perhaps through calibration studies that establish com- mon scales. These ad advancements would build upon our exploratory work to develop increasingly robust foundations understanding how capabilities interact within language models, potentially revealing even more nuanced patterns than we have observed in this initial investigation.
19

---

Published as a conference paper at ICLR 2025
B COMPLETE TAXONOMY
To ensure the comprehensiveness of the prompt sets in our evaluations, we build taxonomy with Level-1 (LI) and Level-2 (L2) categories. More concretely, Tables - and Tables 10 12, present the taxonomy for individual capabilities (English and Multilingual, Reasoning, Coding, Im- age Recognition, Tool Use, and Long Context) and cross capabilities (Coding & Reasoning, Image Recognition & Reasoning, Tool Use & Coding, Tool Use & Reasoning, Long Context & Coding), respectively.
B. TAXONOMY OF INDIVIDUAL CAPABILITIES
L1 Categories
1,2 Categories
Historical events & figures
Scientific concepts and explanations
Factual Questions about Recent and Current Things
Geographical information
Cultural & social topics
Technical information
Historical events & figures Scientific concepts and explanations
Very Accurate Questions (Beyond Expected Model Knowledge)
Geographical information
Cultural & social topies
Technical information
Cooking & food preparation
Home & DIY projects
Technology & devices
Procedural Questions about Recent, Current, Local Things
Arts & crafts
Travel & transportation
Work & productivity
Health fitness
Dining food suggestions
Entertainment suggestions
Recommendations Brainstorming about Local and Current Things
Travel & destinations suggestions
Product & service recommendations
Content Summarization
Tasks with File Uploads
Question Answering
Table 4: Taxonomy of the tool use capability.
20

---

Published as a conference paper at ICLR 2025
L1 Categories
L2 Categories
Historical events & figures
Scientific concepts and explanations
Factual Questions
Geographical information
Cultural & social topics
Technical information
Cooking food preparation
Home & DIY projects
Technology & devices
Arts crafts
Procedural Questions
Travel & transportation
Finance & budgeting
Work & productivity
Health & fitness
Language Assistance
Grammar, spelling, & vocabulary
Analysis
Creative writing: Fiction
Creative writing: Poetry Songwriting
Creative writing: Social media posts
Writing & Content Creation
Creative writing: Nonfiction
Business writing
Legal writing
Classification
Summarization & editing
Identity Personas
Chit-Chat
Advice
Dialogue
Games: Choose-youre adutett
Games: Word language
Games: Social party
Dining food suggestions
Entertainment suggestions
Recommendations/ / Brainstorming
Travel & destinations suggestions Product & service recommendations
Build confidence and self-esteem
Emotional support
Goal setting
Motivation
Personal Growth and Development
Physical health support
Professional and career support
Relationship support
Tutoring and learning support
Debate and opinions
Discuss shared interests Humor and jokes
Social Interaction and Communication
Socialize with friends (group chat)
Table Taxonomy of the English and multilingual capabilities.
21

---

Published as a conference paper at ICLR 2025
L1 Categories
L2 Categories
Arithmetic & basic math
Algebra & equations
Geometry & trigonometry
Mathematical Calculation
Calculus & advanced math
Probability & statistics
Discrete math logic
Ordinary and partial differential equations
Math word problem solving
Math question answering
Mathematical Reasoning
Theorem proving (c.g. proofs)
Mathematical model building
Physical reasoning
Commonsense Reasoning
Temporal reasoning
Spatial reasoning
Identifying root causes & issues Evaluating evidence & reasoning Identifying pros & cons Inductive reasoning Deductive reasoning
Logic/ / Problem Solving
Empathy and perspective taking
Social norm understanding
Humor understanding Negotiation
Social and Emotional Reasoning
Emotion recognition sentiment analysis
Consequence evaluation
Moral and Ethical Reasoning
Applying moral and ethical principles
Resolving moral ethical dilemmas (conflict of principles)
Hypothesis formation and testing
Causal reasoming
Scientific Reasoning
Scientific evidence evaluation
Model-based reasoning
Casc-Based Reasoning
Statutory Interpretation
Contract Interpretation
Legal Reasoning
Administrative Regulation Interpretation
Legal Evidence Evaluation
Table 6: Taxonomy of the reasoning capability.
22

---

Published as a conference paper at ICLR 2025
1. Categories
1.2 Categories
Code generation (Text to Code)
Code completion
Code Summarization Compression
Code Generation Synthesis
Code Code (same language)
CLI
Coding Ecosystem
Code to Code (different languages)
Comment generation
Commit text generation
Code Documentation
Document this function
Create example usages of this function
Create API documentation
Debugging & troubleshooting
Code Debugging
Testing
Code review
Security Review
Code Review Best Practices
Quality Assurance
Log Analysis (Text to Text)
Table 7: Taxonomy of the coding capability.
L1 Categories
L.2 Categories
Single Object Recognition
Object Recognition
Multiple Object Recognition
Fine-Grained Object Recognition
Indoor Scene Understanding
Outdoor Scene Understanding
Scene Understanding
Cultural Scene Understanding
Complex Scene Understanding
Descriptive Captioning
Image Captioning
Abstract Captioning
Object Attribute Identification
Attribute and Relationship Identification
Spatial Relationship Identification Semantic Relationship Identification
Visual How to Memes
Dialogue
Vague or unrelated question Blurry image
Graceful Refusals
Unsupported capabilities
Table 8: Taxonomy of the image recognition capability.
23

---

Published as a conference paper at ICLR 2025
L1 Categories
1,2 Categories
Scientific Documents
Financial Documents
Books
Factoid or Complex Question Answering
Legal Documents
Podcast transcripts
Video/Movie transcripts
Scientific Documents
Financial Documents
Books
Summarization
Legal Documents
Podcast transcripts
Video/Movie transcripts
Home & personal Work & business
Multi-Document Understanding (Q&A)
Table 9: Taxonomy of the long context capability.
24

---

Published as a conference paper at ICLR 2025
B.2 TAXONOMY OF CROSS CAPABILITIES
Categories
1,2 Categories
Programming concepts guidance Software Architecture
Coding Q&A (Text to Text)
features
Code summarization
Frameworks & tools
Code walkthroughs Algorithm explanations
Code Explanation
Code Understanding
Problem decomposition
Programming Assistant
Algorithmic reasoning
Debugging reasoning
Code optimization
Arithmetic & basic math
Algebra & equations
Geometry trigonometry
Mathematical Calculation
Calculus & advanced math
Probability & statistics
Discrete math & logic
Ordinary and partial differential equations
Table 10: Taxonomy of the coding & reasoning capability.
L1 Categories
L2 Categories
Scientific Diagram Undorstanding
Diagram Understanding
Flowchart Understanding
Graph Understanding
Basic Chart Understanding (Localization)
Chart Understanding
Basic Chart Descriptions
Chart reasoning
Document understanding Others
Text-Rich Understanding
Visual Math and Science Formula understanding
Figure understanding
Table 11: Taxonomy of the image recognition & reasoning capability.
25

---

Published as a conference paper at ICLR 2025
L1 Categories
1,2 Categories
Code generation (Text to Code)
Code completion
Repository-Level Code Generation
Code Summarization
Code Code (different languages)
Code modification
Code Q&A summarization
Repository-Leve Code Understanding
Code walkthroughs
Repository-Lovel Code Debugging
Debugging & troubleshooting
Parsing logs into structured templates
Log Analysis
Finding anomalies from raw logs
Detecting errors and debugging suggestions
Q&A on Code generation with API
API Docs Understanding
Table 12: Taxonomy of the long context & coding capability.
L1 Categories
L2 Categories
Code generation and execution (Text to Code)
Code Execution
Code to Code (Same language)
Create example usages of this function
Debugging, troubleshooting, and optimizing code Testing
Code Debugging with Execution
Programming Assistant with Execution
Code Understanding
Data Analysis
Code Execution with File Uploads
Data Visualization
Code Review Explanation Debugging
Table 13: Taxonomy of the tool use & coding capability.
26

---

Published as a conference paper at ICLR 2025
L1 Categories
L2 Categories
Math word problem solving
Mathematical Reasoning
Math question answering
Physics
Chemistry
Units and Measures
Computational Sciences
Earth Sciences
Materials
Space and Astronomy
Scientific Reasoning
Life Sciences
Technological World
Weather and Meteorology
Food Science
Transportation
Health and Medicine
Physical Geography Engineering
Arithmetic & basic math
Algebra & equations
Geometry trigonometry
Calculus & advanced math
Probability & statistics
Discrete math & logic
Mathematical Calculation
Number Theory
Linear Algebra Plotting
Complex Analysis
Continued Fractions
Trigonometry
Ordinary and partial differential equations
Table 14: Taxonomy of the tool use & reasoning capability.
27

---

Published as a conference paper at ICLR 2025
B.3 DETAILS FOR TAXONOMY CONSTRUCTION
The manual laxonomy construction follows a multi-stage annotation process, detailed as follows:
Annotation Team. The annotation team consists of contributors with experience LLM searchers or product managers, including some of the authors of this paper. The team's diverse expertise ensures that the taxonomy captures wide range of real-world use cases.
Ll Annotation. Each capability's 's Ll categories are derived from a large set of real user queries:
Three team members independently annotate the LI categories by identifying common tasks in the queries.
After the independent annotations, the team convenes merge similar categories, address any omissions, and finalize the LI taxonomy collaboratively.
L2 Annotation. Building on the agreed-upor LI categories, the annotators independently propose L2 subcategories:
These finer-gran tner-grained tasks are designed provide additional specificity while maintaining consis- tency with the L1 framework.
A second round of discussions and revisions is conducted to refine the L2 categories and reach consensus.
Team Review, Once the LI and L2 categories are finalized, all taxonomies undergo comprehensive review by the full team:
The team fine-tunes the taxonomy to ensure that all breakdowns adhere to consistent granu- larity across capabilities and align with the MECE principle (mutually exclusive and collectively exhaustive).
Annotation Effort. The annotation effort involves more than ten main contributors and spans ap- proximately one week.
28

---

Published as a conference paper at ICLR 2025
C MORE DETAILS ON CAPABILITIES
C.I PRINCIPLES FOR DEFINING CAPABILITIES
In our framework, capability defined by the following principles:
Distinctiveness: The capability should have unique characteristics that distinguish it from others.
Mcasurability: The capability should be independently evaluatable through specific tasks.
Consisteney:
The capability should maintain its fundamental characteristics across different con- texts and domains.
Using these principles, we define the seven individual capabilities as described in Section $2.1,
C.2 DISTINCTION BETWEEN CAPABILITY AND SKILL
The concept of "skill" is relatively vague and can manifest in different ways:
can be subset of a capability, fitting within our taxonomy (e.g.. "basic math formulas" and "analytical reasoning" as part of Reasoning capability).
It may span multiple capabilities. For instance, "research" can be a skill that may require various capabilities: Tool Use (to gather information via web browsing), Long Context (to process and synthesize extensive documents), Reasoning (to evaluate and draw conclusions from the findings), and potentially Image Recognition (to interpret graphs and figures in academic papers).
Given this variability, we follow the capability-based evaluation approach commonly used in pre- vious studies and utilize our manually created taxonomy to distinguish the boundaries of different capabilities.
29

---

Published as a conference paper at ICLR 2025
D CROSSEVAL BENCHMARK
D.1 WHY CONSTRUCTING THE BENCHMARK FROM SCRATCH
While existing benchmarks for specific core capabilities exist, we find it necessary to construct prompts for the following reasons:
Existing benchmarks often lack precise categorization for indi- vidual prompts. For example, benchmarks evaluating coding tasks may include prompts that, ac- cording to our taxonomy, fall under either the individual capability "Coding" or the cross-capability "Coding" & "Reasoning." Without this categorization, it is impossible to conduct the in-depth anal- yses needed to understand the relationships between individual and cross capabilities.
1) Lack of Clear Categorization:
2) Coverage: Existing benchmarks do not cover all 76 Level-1 and 332 Level-2 categories in taxonomy. To ensure thorough evaluation, we construct prompts to fill these gaps and provide more complete coverage.
3) Difficulty Distribution: Existing benchmarks often lack consistent difficulty distribution or use varying distributions across tasks. These inconsistencies hinder systematic comparisons of indi- vidual and cross capabilities. Our benchmark standardizes difficulty levels across all capabilities, supporting meaningful analysis.
Existing benchmarks vary significantly in prompt such as QA formats, exam-like questions, real-world scenarios. These variations hinder consistency in evaluation. To address this, we standardize all prompts in our benchmark to reflect open-ended, real-world user queries, ensuring uniformity across capabilities.
4) Style Distribution:
D.2 SELECTION PROCESS AND CRITERIA FOR PROMPTS
Selection of 100 prompts. Our reviewers follow strict quality screening process to ensure all prompts align with the task categories in our taxonomy and meet the criteria outlined in the guide- lines. If issues such non-compliance, incomplete task coverage, misalignment with the diffi- culty distribution are identified by our reviewers, the data vendor reannotates the problematic parts. After review and reannotation, the number of annotated prompts for each capability exceeds 100, with a unified quality control standard ensuring high-quality prompts across capabilities.
Why limit to 100 prompts. We limit 100 prompts for each capability for the following reasons:
Standardizing 100 prompts per capability ensures fair comparisons, preventing any capability from being over- or underrepresented due to variations dataset size.
Consistency:
Resource Efficiency: Each prompt requires multiple model responses and expert annotations, which are essurce-intensive Limiting to 100 prompts maintains quality while managing costs and reduces the overall expense of evaluation on our benchmark.
D.3 STATISTICS OF THE PROMPT SET
The prompt set annotated in CrossEval covers all 76 Level-1 and 332 Level-2 categories listed in the previously mentioned taxonomy, with specific statistics provided in Table 15.
30

---

Published as a conference paper at ICLR 2025
Capabilities
# Prompts
# L1 Categories
# L2 Categories
English
100
8
45
Reasoning
100
8
36
Coding
100
4
18
Individual
Image Recognition
100
Tool Use
100
23
Long Context Spanish
100
3
14
100
8
45
Coding & Reasoning
100
4
19
Image Recognition & Reasoning
100
4
10
Tool Use & Coding
100
9
Cross
Tool Use Reasoning
100
30
Long Context Coding
100
5
13
Spanish & Reasoning
100
8
36
Spanish & Image Recognition
100
6
17
Table 15: Statistics of the prompt sets in the CROSSEVAL benchmark.
31

---

Published as a conference paper at ICLR 2025
D.4 PROMPT SET EXAMPLES
To provide an intuitive sense of the types and difficulty of the prompt set in our benchmark CROS- SEVAL, we present examples for each capability, including the difficulty level, L1 and L2 categories, and the prompts. Tables 16 - 22 correspond to individual capabilities, while Tables 23 27 pertain cross capabilities.
Difficully
Ll Calegory
1.2 Calegury
Prompl
Logic
Deductive reasoning
bachelors never married. John man who has married.
Easy
prublem solving
Is John buchelor?
Commonsense
If you a building from the cast side, walking west, and then take two righis and a lefl down corridors, which direction are you lacing?
Easy
Spatial reasoning
Reasoning
Jane won the lottery and decided to spend some of the money. She spent the first day.
Mathematical Calculation
Discrete math Ingic
secund day. She spent $4.50 the third day. She kept spending her winnings in the same pattern and then on the day, spent her remaining $300. How much did lottery?
spent
Medium
the
Social and Emotioonl Reasoning
Empathy and perspective taking
ing ahout firing him. heard from another collengue that he may be going through divorce. taking huge productivity bit. What should I do? Explain
on my
not pulling weight and think-
Medium
should allow affect work. Our team
Solve the initial value peoblem:
Ordinary and partial differential equations
1 Uas My - u(x,
Mathematical Calculation
Hard
You will find that the solution blows up finite time. Explain this terms the characteristies for this equation and explain your reasoming slep by step.
Two chemists are sitting at bur. The first chemist tells the bartender, "I'll also
Social and Emotiionl Reasoning
Hard
Humar understanding
some H2O." second some water". The first chemist tells the second chemist, "darn my murder plot failed". Please explain this joke to me
hartender,
One interesting and complex problem that can be addressed through mathe- matical modcling fisherics biology the effectiveness fish stocking increase angling opportuniries.
Problem Statement: Optimize amount of angling opportunities
Hard
Mathematical Reasoning
Mathematical model huilding
introducing the correct number bass fish to a given lake.
Variables to Considor:
of given body of water.
v: volume of vegetation growing in body of water.
y: amount forage fish per acre sampling data. number of hours angling pressure given body of water per month.
Table 16: Examples of the prompt set for reasoning capability.
32

---

Published as a conference paper at ICLR 2025
Difficulty 1.1 Category
1.2 Category
Prompl
I'm having trouble setting up my router. I want to change the stock passwerd create guest network with a separate pass- word for my friends and family, Can you help me change password, and create separate guest actwork with new password??
Easy
Procedural
Technology & devices
network,
Questions
My wife really into her arts and crafts. She loves painting in her spare time. Can you please recommend something could get her present? Tell me 6 candidates.
Recommendations Brainstorming
Product & service
Easy
recommendations
You Christopher Walken. sentence openers about our movies that can respond to. try and make them serious I can try and make you laugh.
Sylvester Stallone. Create series of
Medium
Dialogue
Identity/ Personus
In Sama- Wiwa's book about traveling to Nigeria "Transwonderland" how does her outlook on Nigeria change as her jourcey through the coun- try progresses? Include descriptions her first impressions when traveling from region/city of Nigeria to another and describe her feelings some- one who hus Nigerian heritage but moved to England at a very young age. vivid language your response.
Writing& & content creation
Medium
Analysis
My question relates to 3d printing. I'd like to understand more about the chemical differences between two printing materials PLA and TPU, Start by explaining the chemical differences. Then talk about the physical prop- erties both materials, listing three five me an insight into practical considerations when printing with these mate- rials. I'm particularly interosted recommended extruder and printing bed settings.
for each. Finally, give
Technical information
Hard
Factual Questions
I'mm looking for detailed response written clear simple English.
I'm novice, so be to stuiiizze any technical terms and provide
definition in parentheses.
Please keep your response 600 words, give take 10 percent.
-Separatc sections and sub-scctions with HI and H2 headers.
You are game developer, focused on choose- menu--en-ddveture style text- based games. You are creating a main character for the story and need to finalize aspects of the character's 'ss personality. The main character is young. male, roughly 14 years of age. He is a wizard with science and technology subjects, and personality should reflect this, He that lead the character down a positive route, with a general increase in his knowledge and skills.
chesen routes
Games:
unlucky in focuses of the namaaive. What sort of creative personality would the character have that allows the player to connect with and better feel themsellvs the role they chouse the paths they going take? List live less common ones. Also, please provide a varicty of routes in this game that might change personality. Make sure you provide at around 150 wards. At least one mute should have negative consequences for character's personality,
romance
major
Hard
Dialogue
Choose-your-own-
adveature
routes
each being
be decisively positive. Provide a name for this character that reflects the personality you choose for them.
one
need to write short stand-up up comedy routine for friend's dinner party. It should take no longor than minutes to perform. The audience will consist of baker, doctor, do good impression of Homer Simpson, so please write it in his style, The tone should be silly and playful. but be sure not to make fun of the audience.
Social interaction and commanication
Hard
Humor and jokes
florist, so try and make jokes relevant to them. can
Table 17: Examples of the prompt set for English capability.
33

---

Published as a conference paper at ICLR 2025
Difficulty
L.1 Category
1.2 Category
Prompt
Find any with following code and suggests any polential lixes. Our goal with code is keep drawing lottery tickets (where is a lottery game where match 6 numbers is winning ticket. We want to print out the time takes and the number of draws until ning ticket along with the theoretical expected number of draws until success.
of 49 numbers) until there
Code review & practices
Basy
Quality Assurance
winning ticket,
peint
{attached code)
command used add package project! you "emor[ED0433]: failed to resolver could not find "quote" in the crate rost" "erro[[Fn432]: unresolved import "quate"
Fasy
Code generation / ssytthesis
Coding Roosystem
{attuched code)
Create an API documentation for this Python code that utilizes Flask for rostes.
Code tion
documenta-
Medium
mentation
{attuched code)
want to update the data I have determined in the database by pulling it with post. However,
updated. The codes view py also added. It may werk for the application. Also ensure to write unit lests for function.
Mediumn
Code debugging
Testing
fattached code)
Create a readme document that breaks down what the code does, and the the techniques
priniout example usage program, displaying the result/solution to the example data in code. Also include section breaking down time/space complexity and how the efficiency this program compares other methods without any uptimsiatiion//ecnnneei level.
Code documentation
Comment generation
program.
Hard
The document should read at an undergrad
{attached code)
Please create React custom hook that can store, retrieve, and syne data from the browser's local storage the React compoment that the hook. Use the IncalStorage API. Hook should called usel oealStomge. When the component mounts, should read and update local storage when the compenent's 's state changes, and listen for changes in storage update the component state accordingly. The book's API should look like useState.
Hard
/ synthesis
Code generation
Coding Ecosystem
storage value
initial value,
I've becn working implementing written the Harris corner detection for finding good features. want some linding algorithms:: FAST those functions? Also, want to implement Lucas Kanade optical flow. have rough idea for me?
computer vision algorithms. I've
features. Can you finish
Code generation synthesis
Hard
Code completion
function take can you also finish this
{atmuched code)
Table 18: Examples of the prompt set for coding capability.
34

---

Published as a conference paper at ICLR 2025
Difficulty
1.1 Category
1.2 Category
Image
Prompt
Easy
Image Captioning
Descriptive Captioning
Describe every creature in this image.
Fasy
Griceuul refuralls
Bluryy image
What does the license plate say?
Altribute und Relationship Identification
Object Attribute Identification
What the color and shape the "10" ball the photo?
Medium
How many these dogs have floppy ears? How many of the floppy-cared dogs have
Multiple Object Recognition
Medium
Object Recognition
Comprehensive use case
Memes
Explain meme
What type of person would you imaginc lives
Soene Understanding
Indoor Scene Understanding
Hard
in this room? Pick items to justity your swer.
Comprehensive use case
bike ridable. What does ieed be lixed and how do lix it?
Hard
Visual How to
Table 19: Examples of the prompt set for image recognition capability.
35

---

Published as a conference paper at ICLR 2025
Difficulty 1.1 Category
1.2 Category
Prompt
Recommeendations
Fasy
Brainstorming about Local and Current Things
Entertainment
are some fun things Illinois today?
on local channels Henry County,
suggestions
Summarize the below content in the attached Gile in 150 words or less, focusing mode of action, possible adverse effect's and effectiveness.
Eusy
Tasks
File Uplonds
Content Summarization
{attached file)
Procedural Questions about Recent, Current, Things
What are the step by step instructions of installing Black And Decker BD05MWT6 Window Air Conditioner, 5000 BTUs, unit? it into window with the dimensions of 24 inches wide and inches height?
Home
Medium
DIY projects
Factual Questions
Cultural & social topies
Show me the current top 10 hockey players in the USA, what the game statis- are for each person, how many awards they have won.
Medium
about Recent
and Current Things
Very accurate questions (beyond expected model knowledge)
Please list all recent coronal mass ejections from the past 365 days and what their magnitude was. Were there any correlating changes in the schumann resonance? Does science recognize effects of coronal ejections from the sun on the schumann resomunce?
Scientific concepts and explanations
Hard
The text mentions several individuals and their contributions or reputations Choose
discues how they are represented their signifi. cance in the broader historical or cultural context of that year. Also, what role does satire and public opinion play in the texi, particularly the portrayal of political figures and events? Provide specific examples from the given file.
Hard
Tasks
Question Answering
File Uplonds
fattuched file)
want re-cover pair of wingback chairs. love idea of painting or printing my own faarii. Can you give me list of supplies for this? I also need step-by-slep walkthrough of easy ways to puint or transfer scenes to fabric.-a selection of two three comprehensive youtuhe videos would really helpful. I'm also open to other ideas like stamping or stenciling. Glve methods
Procedural Questions about Revent Current, or I.ocal Things
Hard
Arts & cralls
Finally. need step hy step instructions for re-covering them well. Please list any fabric and/or craft stores near me, their hours of operation. List them order of most least likely to have the supplies I need.
break
Make table of countries which least 25% tion speaks French. The table should include the name of the country, their current population, the percentage
popula-
population who speak Frrench, and the name of the cureett head of state. Below the table, summarize trends in the percentuge French language speakers globally over the past four years. This summary should be amples of French-language radio stations, French-langume newspapers, and French-language television stations currently operating countrics which French not the predominant national language. Provide links websites for wherever possible.
Factual Questions about Recent Current Things
Geographical information
Hard
than 150 words. Finally, list
Table 20: Examples of the prompt set for tool use capability.
36

---

Published as a conference paper at ICLR 2025
Difficulty
L1 Category
L2 Category
Prompt
Given this podcast transcript, please summarize Daniel Dennett's opánion Gould's idea Not-Overlappin Magisteria. and explain bow Dennett believes science ought to li into society. Give specific quote to support your answer each question.
Factoid or Complex Question Answering
Podcust transcripts
{attuched text)
What are some of the most common pieces of advice for maintaining a vegetable garden. there any information controversial?
Multi Document Understanding
Easy
Home personal
{attached text)
Based on the financial documents of JR-HIFI 2021, summarise the Notes financial statements and the financial statement where each relates to.
Financial Documents
Medium
Summarrzatiion
{attuched sear
What caused Jane be locked the "red-room"? How did the "red- room' affect her adult life in both positive and negative ways? Did she experience any Ilasbbucks regarding
Factoid or Complex Question Answering
Medium
her adult life!
(attached text)
Summarize every four chapters this book in three paragraphs. The first paragraph should summarize the first and second chapters, the second paragraph should summarize the third and fourth chapters, and the third paragraph should in 1-3 words that encapsulates the events of these chapters. Repeat this structure for each subsequent set of live chapters!
chapter.
Hard
Summarization
Books
(attached sexr)
Given the segment of the NBA CBA which outlines finamcial rules, how might strategy differ terms team/roster construction salaries contending team differ from lottery ream? Justify you response with quotes from the text, and give examples with fictional roste with salaries demonstrate the salary repencussonn//ttaaegg being proposed.
Factoid or Complex Question Answering
Hard
Legal Documents
fattached sext
Given latest annual financial reports from AMD (Advanced Microde- vices) and NVIDIA, evaluate each company' strategy and preparedness for addressing worklouds over the mext 5 years include their comparative headwinds and tailwinds. Assess which company currently best positioned to guin the most murket share in Al and why.
Hard
Dolli-Decumet Understurding
Work & business
{attuched tear).
read this book, but forgot take notes. So understand the philosophy, but I don't remember the exact stops that should follow; Extract all the information that the successful so I can print these routines, rules, and truths and have them in sight. I 20. 20.20.22 routine, etc.
Factoid
sharod
his two friends,
Hard
Complex Question Answering
Books
talking the 5.3.1
{attached sexr)
Table 21: Examples of the prompt set for long context capability.
37

---

Published as a conference paper at ICLR 2025
Difficully
L1 Category
L2 Calegory
Prompl
Digamos que shorita Tosla y que responder fucras cl.j me dirias pregunto acerca de tus mejures inventos?
Easy
Dialogue
Identity / Personas
Escrihe una sintesis accren de las características principales del culto iamo en América Latina, con énlasis especial en los paises más grandes y religiosnc: México y Brusil.
Cultural social lopics
Easy
Factual Questions
Escribe una historia de animales similar pelicula de Madagascar pero ambientada on la cueva Hang Son Doong de Victnam. Los tres personajes principales deben ser un murciélago. un pájaro un escorpión. Las estalac- titas estalagmitas deben poder hablar. villamo de historia debe gusano bioluminiscente.
Writing & content creation
Creative writing: Fiction
Medium
Estay teniendo problemas con mi luplop. Es una Dell Latitue 3440 con Windows Cada tanto tiempo, entre minatos y hom, se desconecta el Wi-Fi y debo volverlo concctar manualmente. Dame posibles solu- ciones, on viñetas. Las soluciones deben ser totalmente detalladas, como para alguien que
Procedural Questions
Technology devices
Medium
conocimiento de computación.
ficción. emburgo. año quicro empezar leer otros tipos de libms. Me puedes compartir unn lista de tus libros favoritos que no scan ficcióa? eafócate en autores lati- noumericanos asiáticos y estoy abierta cualquier tipo escritura (poe- mas, biografias, etc.). dime idioma original la macionalidad de cada autor tiene traducción espatiol caso de que necesario
encanta especialmente
Social interaction and communication
Discuss shared interests
Hard
gastos misceláneos y entretenimienta digi- tal. Establece un presupiesto mensual para cocinar la mayoría de las comi- das en casa. Realiza las recomenlaciones lemiendo en mente una personu que vive el suroeste Florida, de seguir comida saludable y barata estén basados en la disponibilidad de dicha región. gasto mixime mensual será $400. y debes especificar como distribuirla. En cuanto los gastos de entretenimiento digital, he identifi- cado que mayoría de streaming. Necesito quedarme solo con dos de estas plataformas. Mis in- tereses principales son los deportes y cine clásico. Recomiéntame las que puedan satisfacer demanda dentro de presupuesto limitado, Lo máximo que pucdo gastar en entretenimicnto digital son $21.
Necesito aborrar
que los mercados donde
Procedural Questions
Finance budgeting
Hard
están vinculades suscripciones a plataformas
Soy una aficionada del trekking de acampor en la naturaleza. Lamentable- meute, a voees me resulta dificil salir caminar, sobre todo a lugares frios perque no cuento pecializada se hu vuelto may cara, asi como las botas de trekking. las bolsas de dormir, tiendas campaña,, etc Por favor, ayídame daciones un equipo básico, que seu de buena calidad, pero con precios accesibles. Necesito atuendos completne trekking, para temperaturas O 15 grados contígrados. Parn zapatillas, estoy dispuesta gastar más un par que me duce menos años. Que sean impermeables, cómodas y calientitas. Finulmente, necesito invertir que sea ligera!
suticiente equipo calidad. ropa de trekking es-
recomen.
Recommenndatinns Brainstorming
Product & service recommendattons
Hard
bolsa dormir. iPero
dime un chiste con más de 300 pulabras que incluya las palabras galo, trueno árhol. que del punto de perspectiva del gato incluya frase final chusca con moraleja.
Social interaction and communication
Hard
Humar jokes
Table 22: Examples of the prompt set for Spanish capability.
38

---

Published as a conference paper at ICLR 2025
Difticulty
L1 Category
L2 Category
Prompt
inherited python function a previous Scientist in my team who has left. noed understand what the function is doing there are no comments and I don't have time to debug this function.
Programming
Code Understanding
Easy
Assistant
(attuched code)
I'm plamning to implesment different shupe classes in Java. They have similar fields, the parameters
example, circle has a radios field, but square has length field. I want to upply the factory design patterns to my program; is a good practice? If not, which design patterns my requirements more?
Codiag Q&&A (Text Text)
Software Architecture
Eusy
Design most efficient algorithm to given list that sum to a target value. Ensure that each pair unique (no pair should repeat oven if integers appear multiple times in the list). Deseribe the algorithm'' 's logic and implementation details. Explain you chose implement in way you did in onder to achieve the best time complexity memory storage possible.
unique pairs integers in
Programming Assistant
Algorithmic reasoning
Medium
Mathematical Calculation
Ordinary and partial differential equations
Suppose that dy uses Faler's it
1, and ((0) = 1. Write me Python code which estimate y(1).
Medium
I'm working on a simple command line game, and want to break it sure bow
reusable functions, but suggest any functions that would make code less repetitive? Also, help me write 5 more useful functions in the code.
you
Programming Assistant
Problem dccomposition
Hard
(attuched code)
Can you explain "tail recursion" and why it's key for hoosting efficiency in recursive functions? It's known using less memory than standard recur- sion. Could you demonstrate this with Python example, showing both tal-recursive lunction and a regular recursive function? Also, since Python doesn't I automatically optimize tail recursion, how does this limilation im- pact its use in larger projects? BTW, how do other programming langunges handle tail recursion differently, and advantages they offer?
Programming concepts & guidance
Coding Q&A (Text to Text)
Hard
There are 8 people in a room, and two boxes. One box has 8 hats, 2 pink, red, blue, and black. The other box has 12 puirs of gloves, 5 green, 4 black, orange, and yellow. The 8 people in the room are invited to each grab and put shake hands with different people at random.
hat and
of gloves. Then, each person will
Mathematical Calculation
Algebra equations
Hard
Create an algehmic equation find the probability of someone wear- ing a pink hat and orange gloves shaking hands with someone wearing a black hat and green gloves least once?
Use python script solve the oquation find the probability, given in percenlage.
Can you please explain what the following code does and bow the outpu represented? Do we really need to represent the output way? Also, provide with way to implement the following algorithm based other librarics.
Code Explanation
Code walkthroughs
(atsached cade)
Table 23: Examples of the prompt set for coding & reasoning capability.
39

---

Published as a conference paper at ICLR 2025
Difficulty
Category
Category
Image
Prompt
Basic Chart
Chart
From name least manufac- turer more popular than Toyota
Easy
Understanding
Underslanding
(Localization)
Visual Math and Seience
Figure understanding
Easy
What this illusion called?
Basic Chart Understanding
Chart
What is the max trading volume in the bur chart?
Medium
Understanding
(Localization)
What the chemical formula that comes after word "Methanol"? Provide for formula, and what the formula is represent- ing/used for.
Visual Math and Science
Formula understanding
Formaldehyde
Medium
"he Memerdd
Diagram Understanding
Flowchart Understanding
Based the diagram, how many steps to do before Xsemble Design?
Microsoft Intervew Question
Text-rich Understanding
So what is the best strategy for the interview question the image!! Explain step by stop.
Hard
Others
How many nodes do I have to remove from this graph so that is possible to visit ev- cry vertes without repeating edge, starting from one at the bottom-leff.
Hard
Diagram Understanndnng
Graph Understanding
Table 24: Examples of the prompt set for image recognition & reasoning capability.
40

---

Published as a conference paper at ICLR 2025
Difficulty
L1 Category
L2 Category
Prompt
In the 'RingIntercom class', there is functionality to unlock door and process notifications. Extend this by adding method that logs unlock actions with timestamp and the user ID who performed the unlock. This method should the log
Repository-Leve Code Generation
Code completion
fike named Jog.
{attached code)
Parsing logs structured templates
Can you parse logs from this log file provided into tree. stricture??
Log Analysis
(atsaohed log)
Referencing the Notily API documentation, you develop sample command line application in Java that checks the balance of your Notify.kk account and prints out? Assume you have the necessary API credentials.
API Does Understanding
Code generation with API
Medium
{attached code)
Optimize application's performance by implomenting lazy loading for the React components usoociated with roates 'Appjs'. Use 'React and 'Suspense" from React lazily load the components for "/profilefid" 'leditor', and 'playes'idd. Also, provide fallback leading component that displays lunding spinner while the components are being loaded. Provide the code modifications 'App-js' implement lazy loading
Repository-Leve Code Generation
Medium
Code modilication
{attuched code)
I'm working on an automatic grader using Python for my intro to OOP Currently
download my studeat's py submissions munually and then uploudlenier each grade manually. Do you think it would be possible use the Canvas LMS API to automate these processes? do I download the submissions an assignment for student using python? here is some documentation for the canvas AP1
API Docs
Hard
Understunding
Q&A API
given student seVupdale the grade
fattached code)
scems there bug the that easures that start point ray casting algoritim is outside the polygon. This 3D items being able to be moved after I update the layout (floorplan) of the room. Locate function that needs fixed suggest fix.
Repository-Level Code Debugging
Debogging & troubleshooting
Hard
(attached code)
Given repository, createe system to display the top eight high scures for the game session. When the game starts up, high scores should [80, 70, 60, 50, 40, 30, 20, 10] all with a name label 'Anonymous" dies, if their final score
when the they should he able enter their name and add their highscore to the list in the appropriate place (i... all lower scores should be moved down the list and the lowest removed altogether). When the game is AppSlute "Game", : The state should aliemate between intro screen and highscores
high sccres list.
Reposilory-Leve Code Generation
Code generation
Hard
(Text to Code)
that displays the high sersen should cousist of text containing the list of high scores displayed over the background
list. The high
the Game and IntroScreen states, text font should be consistent with the other text in the game and should be white unleas it new highscore, which ease it that entry should green.
(attuched code)
Table 25: Examples of the prompt set for long context & coding capability.
41

---

Published as a conference paper at ICLR 2025
Difticulty
Ll Category
L2 Category
Prompt
Group this sales data by month using pandas and then har chart of the sales month. Run the code and save the results.
Code Execution Uploads
Data Visualization
{attuched file)
Can you genemate set of unit for each method? Please have different and valid test cases for each mothod in the given code. Make sure edge cases. Then
Code debugging with execuison
Easy
Testing
and me the outputs.
{ataached code)
wrote the following function which finds all of composite numbers from 'n'. It's bit slow though because it's brute force method. Can you rewrite it in a way that reduces the time complexity? Use a few examples show real differences running time by executing the code.
Code to Code (Same language)
Medium
Code Execution
{attuched code)
Create a Python decorator named measure_time that logs the duration a fune- tion execute. Apply this decorator sorting function that list of random numbers. Test the sorting function with lists of increasing sizes (o.g.. 100, 1,000, 10,000, and 100,000 elements) and use the dccorator out the sorting times for each list size. Remember and give me the results. Make sure your code is cleam and easy to follow!
Code gencration and execution (Text to Code)
Medium
Code Execution
code
Can you please write a python file utilizing the library Streamlit for frontend visualizatio,
case ansunn
and display in chart like diagram? Please sum "Amount", "Cash" and "Check" each customer set the value as "Gross Purchase", attached Winery sales excel spreadsheet file prompt for you follow. Please run your program and show me the visunlization. Save the results.
Code Execution with File Uploads
Hard
Data Analysis
Jattached file)
have a code that simulates game Blackjack, but I'm not sure it's accurate or eflicient. The code takes into account the player's hand, the dealer's upeard, and the player's current bet. returns the recommended action (hit, stand, double down, or split) based on a complex set of rules. However, I'm not sure the you to review the code and suggest imprivments. Help me review the code, identify any issues, and suggest improvements. Then provide some cases and run the improved code demonstrate accuracy.
Programming Assistant with Execution
Codee Understanding
correctly implemented.
like
Hard
(attached code)
game implemeented u Python function. am not able to understand the code. Can you please explain to me how the game works by providing some example usages of this Function. you execute the program providing show me the outputs obtained and explain to me the reason for the ourputs.
provided
Create example usages of this function
Hard
Code Execution
valid inputs. Please
fattached code
Cryptarithons are like puzzles where replace digits in math equation with letters. Eacch letter stands for unique digit. For example. in "SEND MORE MONEY", each letter is unique digit. Plense help write Python function that can crack any cryptarithm with addition, subtraction, multiplication,
Code generation and execution (Texi to Code)
Hard
Code Execution
lunction should spii dictionury where each letter maps to the digit it represents. Please also provide some complex cases and execute the programs you wrote to get the answers these
Table 26: Examples of the prompt set for tool use & coding capability.
42

---

Published as a conference paper at ICLR 2025
Difticully
Culegory
1.2 Category
Prompl
Given the reactants sodium chloride and silver nitrate, what are the products that will be formed?? Give the balanced chemical reaction. don't cure if the answer is generated or you get it with any tool, but please be as detailed as possible.
Scientific
Easy
Chemistry
Reasoning
Mathematical Calculation
Easy
Complex Analysis
Show me the local minimum of function (()
4n
spacecratt moving gravity-fice space along straight path when pilot decides to accelerate forward. He turns on the thrusters, and burned ejected constant
Scientific Reasoning
kg/s the rocket) of 2.5 102 m/s. The initial mass of the spacecraft and its unburned fael 2.0 x kg and the thrusters are on 30
speed
Medium
Engineering
What is the thrust on the spaccorft and what is the acceloration?
Calculate the total calories humed by person weighing 70 kg who walks spead km/h for hour on a surface. Assume walking metabolic equivalent (MET) value of 3.8. Use the formula for caloric expenditure:
Calories hamed MET weight kg duration hours.
Medium
Scientific Reasoning
Health and Medicine
Additionall, factor in the effect of air resistance, assuming wind speed of 10 km/h against the direction of walking, and adjust the MET value accord. ingly using the formula:
MET (1+
wind speed in km/h
Adjusted
walking speed
Provide the detailed calculation steps and final resalt.
Solve the first-order ordinary differential equation (ODE) using the Fuler method:
Mathematical Calculation
Ordinary and partial differential equatiion
dy
de
2y. with initial condition y(U)
Calculate Implement the Euler method to solve this ODE and print the value of each step. Provide Python function to represent the ODE and another perform the Euler method. Give the accurate results.
a step
Mathematical Calculation
and shade in the region equality |x| y² 10. Show me the image, and save it format.
coordinate plane
Hard
Plotting
Scientific Reasoning
day meal plan from McDonald's with meals that are roughly 400, 800, and 1200 calories cach? Please include a drink with each meal and koep
Could you suggest
Hard
Food Science
within calories these targets.
(s logical statement 2) (t - u) trae, the statement false, determine the troth value of the statement
Mathematical Calculation
maih & logic
Hard
(-(r v v)] + [(r v A (s u) -n]. Any tools are allowed. Explain your answer step by step.
Table 27: Examples of the prompt set for tool use & reasoning capability.
43

---

Published as a conference paper at ICLR 2025
D.5 REFERENCE EXAMPLES
To offer more intuitive understanding of the reference examples in our benchmark, we randomly select an instance from both English and Reasoning capabilities and present their reference examples in Tables 28 - 33..
Prompt
LI Calegury: Recommenniations / Brainstorning
1.2 Category: Fntertainment suggestions Difficulty: Hard
Prompt: I'm looking for suggestions of 1970's movics. The genre should be horvor, and it shoold take place in the woods or be summer camp bused. not include movies that also incorporate humne, movies chosen need be strictly bortor. Include titles und directors. Also, include the names of any actors who have gone to be successful in Hollywood after they were in these movies.
Model Response
Here are some 1970s horror movies that place in the woods summer camp, without humor:
1. The Evil Dead (1981) Director: Sam Raimi
This film takes place in
cabin the woods and follows group of friends who temorized by demons. The movie has
since become a cult classic.
Successful actors: Bruce Campbell,
Sandweiss, Richard DeManincor
2. The Burning (1981)- Director: Tony Maylam
TTiis summer camp slasher film tells the story of a group of campers who are stalked and killed by former camper who was horribyy burned prank gune wrong.
Successful actors: Brian Matthews, Lou David, and Holly Hunter
The Town That Dreaded Sundown (1976) Diroctor: Charles B. Pierce
This horror film is based on truc story and takes place in a small town in Arkansas where a scrial killer is terrorizing locals and
tourists alike.
Successful actors: Dawn Wells, Jimmy Lydon, and Andrew Prine
Just Before Dawn (1981) Director: Jeff Lieberman
This backwoods borror film follows group friends who hiking the Pacific Northwest when they stalked and
killed by mysterious figure.
Successful actors: George Kennedy, Chris Lemmon, and Mike Randolph
5. Madman (1977)- Director: Joe Giamznnreaa
sumuee camp
film
stury group campers are stalked and killed by legendary killer said
haunt the woods.
Successful actors: Paul Ehlers, Dehorah Luce, and Thomas Canr
The Prey (1977) Director: Edwin Brown
This wildemess survival horror film follows a group of friends who are stranded in the woods and hunted by a group of mountain mon.
Successful actors: Keith Carradine, Tuisa McReynolds, and Lew Saunders
movies are considered horvor classics are still popular among fans the genre today.
Human Annolations
Rating 1: 2/5
Explanation This response makes several errors. movie Madmen was directed by Giannone, also released in 1981. I. not 1977. The movic The Pray was released in 1983, not 1977. While the other movic dates correct, most of them are not from the 1970s. which was a specific component of the user's 'ss request. The only movic the suggested list that was actually The Town That Dreaded Sundown, which does not have the setling the user requested. summer cump woods. ................................
Giamandree..
releused the 1970s
Rating 2: 2/8 ................................ ................................
Explanation Half movie recommenddaiins are the 1980s. only 1970s movies were requested leads to a wasted recommendations.
First reference example of English capability. Each instance includes prompt, 3 model responses, and 2 human scores with explanations. This table presents the first model re- sponse alongside its human annotations. Together, the model response and annotations serve the reference example for the subsequent evaluation.
Table 28:
44

---

Published as a conference paper at ICLR 2025
Prompt
1,I Category: Recommensdations / Brainstorming L.2 Category: Entertaioment suggestions Difficulty; Hard
Prompt: I'm looking for suggestion of 1970's movies. The genre should be herror, and it should take place in the woods or be summer camp hased. include movies directors. Also, include the of any actors who have gone successful in Hollywood afler they were in these movies.
incorporate humor, movies chosen he strictly horror Include
Model Response
great roquest! Here are some classic horror movies from
that your criteria:
The Last House on the Left (1972) Directed by Wes Craven
Starring: Sandra Peabody, Lucy Grantham, David Hess
A group of friends are stalked and murdered by pair of escuped convicts in the woods.
The Texas Chain Saw Massacre (1974) Directed by Tobe Hooper Starring: Gunnar Hansen, Teri McMinn, Edwin Neal
group of friends encounter a family canmibals in rural Texas.
The Hills Have Eyes (1977) Directed Wes Craven
Starring: Susan Lanier. Robert Houston, Martin Speer
family's trip
desert into nightmase when they eucounter family mutants.
4. Friday the 13th (1980, but lilmed in 1979) Directed by Sean S. Cunninghan
Starring: Betsy Palmer, Adrienac King. Kevin Bacon
group of
counselors stalked and murdered by Jason Voorhees
camp.
The Town That Dreaded Sundown (1976). Directed by Charles B. Pierce Starring: Ben Johnson, Andrew Prine, Down Wells
A small town is terrorized by a scrial killer in this loosely bascd-on-true-oveens horror film.
Madman (1978) Directed by Joe Giannandrea Sturring: Harriet Bass, Jimmy Steele, Seth Jones
group of summer camp counselors are stalked and murdered by a legendary killer known as Madman Marz
Notable actees who went on to achieve success in Hollywood after appearing in these films include:
Kevin Bacun (Friday the 13th) has had saccessful
film and TV, with roles Footloose, Apollo 13, and American Crime
Story.
appear numerous horror films and write several books, including memoir about his time on the set of The Texas Chain Saw Massacre. Left, The Have Eyes) went Nightmare on Elm Street and Scream franchises.
Gonnar Hansen (The Texas Chain Massacre)-
legendary director, creating
Note: While horror films.
films may have been ro-released remade addod comedic elements, original versions strictly
Human Annotatious
Rating 1: 3/5
Explanation 1: The list of movics is related to what the prompt is scarching for but lot of the details are off like years the movic was made and names
1970s genre borror, but 1980 movie the 13th would still be a useful result. The inaccuracies are a hit distracting and provide misinformation. Rating 2: 3/5
The prompt
summer camp like Friday
job of highlighting the horror list were released in the 1970's, as requested by the user, and the list notes that one movie relcused in 1980 was filmed in the 1970's. However, response states
Explanation 2: This response does
who
greater fame. Most of the movies the
directed Joe Gianmudrea and released note that this director's is name is Joe Giannone and the movie was released in 1981. Additionally The Town That Dreaded Sundown was not summer camp, or the woods, as requested the prompt.
Madman
but both
Wikipedia
Table 29: Second reference example of English capability. Each instance includes a prompt, 3 model responses, and 2 human scores with explanations. This table presents the second model response alongside its human annotations. Together, the model response and annotations serve as the reference example for the subsequent evaluation.
45

---

Published as a conference paper at ICLR 2025
Prompt
LI Category: Rocommcudations/ / Brainstorming L2 Category: Entertainment suggestions Difficulty: Hard
Prompt: lonking suggestions of 1970's movies. genre should horror, and it should take place woods summer camp based. Do not include movies that also incorporate humor, the movics chosen need be strictly bortor. Include the titles and the directors, Also, include the names of any actors who have gone on to be succeasful in wolly after they were in these movies.
Model Response
Sure, here are some horror movies from the 1970s that take place in the woods or involve summer camps, and they stick strictly to the horror genre:
1. Title:
The Texas Chain Saw Massacre (1974)
Director:
Tobe Hooger
Notable Actors: While the cast was relatively unknown at the time and didn't 't go on become major Hollywood stars, the film itself'
has become a cornerstone of the horror genre.
Title: Deliverance (1972)
DDirector John Boorman
NNttable Actors: Burt Reynolds, Jon Voight
Jon Voight went on to huve successful career in Hollywood with films like Coming Home (for which he wen an Academy Award)
Mission: Impossible.
Title: The Last House on the Left (1972)
Director: Wes Craven
Notable Actors: While actors from this film went major Hollywood fame, the movie notable for being Wes Craven's
directorial debut, who later bocame logendary borror director.
Title: The Hills
(1977)
Director: Wes Craven
Notable Actors: Dec Wallace
Doe Wallace went to star films like E.T. the CettraTTerestraal and Cujo.
While these films may not huve large ensembles widely recognized stars, they are significant hurror genre feature actins who have heen successfull in Hollywood.
Human Annolations
Rating 1: 4/5
Explamation 1: These are all 1970s horror movies that wouldn't 't be considered comedy. The movie year release dates, directors and actor information accurate. response could be improved with summary plot. Rating 2: 4/5
Explanation 2: This respouse provides specific details on the fame acquired by cortain actors in these horror movics. The response only are the woods
notes four movies, which seems small this genre and decade, but they are from 1970's summer camp. The Hills have Eyes is set in the desert, but it is 'wilderness' setting.
Table 30: Third reference example of English capability. Each instance includes prompt, 3 model responses, and 2 human scores with explanations. This table presents the third model re- sponse alongside its human annotations. Together, the model response and annotations serve as the reference example for the subsequent evaluation.
Prompt
L1 Category:
Mathematical Calculation Discrete math logic
L.2 Category:
Difficulty: Modium
Prompt: Jane won the lottery and decided spend some of the money. speat on the lirst day. She spent $3 second day. She spent 54.50 on the third day. She kept spending her winnings in the same pattern and then on the last day, she spent her remuining $300. How much she win in lottery?
Model Response
Jane spent $1.50 on the lirst day, $3 on the second day, and $4.50 on the third day. So that is 1.50 + 3 4.50 = 9 dollars in the first three days. She then spent $300 the last day. spent 309 dollars in total. The answer
Human Annotations
Rating 1: 1/5
Explanation response not understand that user asking for pattern be identified, and fails understand there were probably several days in berween the third day and the final day. The response does not come up with a suitable formula and fails to correctly calculate the result, it instead just adds up the amounts specified the prompt.
Rating 2: 2/5
Explanation 2: The response was incorrect bocause it failed to identify the pattern in the given logic. The prompt indicates that Jane spent money according a puttern observed
three days, but response only calculated the total money spent lirst three days and the last day, ignoring the expenditures on the middle days.
Table 31: First reference example of reasoning capability. Each instance includes a prompt, 3 model responses, and 2 human scores with explanations. This table presents the first model re- sponse alongside its human annotations. Together, the model response and annotations serve as the reference example for the subsequent evaluation.
46

---

Published as a conference paper at ICLR 2025
Prompt
LI Category: Mathematical Calculation
1.2 Category:
Discrete math & logic
Difficulty: Modium
Prompt: Jane won the lottery and decided spend some of the money. She spent $1.50 1.550 on the first day. She spent $3 the second day. She speni $4.50 on the third day. She kepl spending her winnings in the same pullern and then on the last day, she spent her remmining $300. How much did she win in the lottery?
Mudel Response
On first day, Jane speat 1.50 dollars. On second day, she spent dollars. On third day, she speat 4.50 dollars. The pattern continues and the money she spends doubles each day. So the 4th day, she spends dellars, the Sth day she spends 18 dollars, and so on. we add up the money spent on each day, we will get a sequence of numbers that looks like this:
1,50, 4.50, 9, 18, 36, 144, 288,
Adding these mumbers up, we get 576. Now we know that on the last day, she spent the rest of her money, which was 300 dollars, So the total money spent
answer 876.
Human Amnotations
Rating 1: 2/5
Explanation 1: The identified pattern incorrect. The correct pattern add 1.5 each subsequent of spending, observed in the first three days. The pattern in the response incorrectly suggested doubling the spending each day, which only applics correctlyy to the second day, the ................................ RRaling 2: 1/3
third days.
Explamation 2: The respome knows needs to lind a putlern, bui is unable find the correct one. Because the first to values in the series ($1.50 53) represent doubling, decides that this pattern, even though the number series $4.50. fact that the puttern it identifies doesn't It include $300 should indicate that the formul may not be correct, but instead the response just adds this faulty formula.
calculation
Table 32: Second reference example of reasoning capability. Each instance includes prompt, 3 model responses, and 2 human scores with explanations. This table presents the second model response alongside human annotations. Together, the model response and annotations serve as the reference example for the subsequent evaluation.
47

---

Published as a conference paper at ICLR 2025
Prompt
LI Category:
Mathematical Calculation Discrete math & logic
L2 Category:
Difficulty: Medium
Prompt: Jane won the lottery and decided spead some of the money. She spent S1, 50 on the first day. She $3 the second day. She spent $4.50 on the third day. kept spending her winnings in the same pattern and then on the last day, she speat her remaining $300. How much did she in the lollery?
Model Response
Let's analyze situation:
Day 1: June spent $1.50
Day spent
Day 3: Jane spent $4,50
We that the amount she spent each increasing by $1.50. To find total she spent before the last day, need find the cumber of days she spent money.
Let's assume she spent money lor before the last day. Then, the total amount she spent before the last is:
5(1.5n)
This is an arithmetic series with first term $1.50, common difference $1.50, and terms. The sum the series is:
(1 11.5mX=+Iy?
the
the series $1.5n. Since spent $300 the day, set up the equation:
(1.5nXn+I V2 + $300 Total arnuun won
We also know that the last term of the series is $1.5n $300 - $1.50 (since she spent $1.50 on the first day). Solving for n, we get:
$1.5n $298.50 = 199
Now we find the total amount she won:
(1.5(199)X (298.5 (200)2 $300 = Total amount won
$29850
amount won
$30150 Total amount
$30150.
Human Annotations
Rating 1: 3/5
Explanation response identilies and describes corectt pattern as providing correct answer ($30150). But explanation of the formula provides is very confusing, particularly with the way that sepurates the $300 on the last day. is not user
ahle
formula provided,
they would know
pattern is, reading
this response.
Rating 2: 3//3
pattern adding $1.5 subsequent day spending. correctly calculated the total amount spent before the last day using the uppropriate formula and provided the accurate result of spending from the day last day.
Explanation 2: response correct
Table 33: Third reference example reasoning capability. Each instance includes prompt, 3 model responses, and 2 human scores with explanations. This table presents the third model response alongside its human annotations. Together, the model response and annotations serve as the reference example for the subsequent evaluation.
48

---

Published as a conference paper at ICLR 2025
D.6 GUIDELINES FOR DIFFICULTY LEVELS
To illustrate the difficulty of prompts in our evaluations, below is a summary of the definitions, accompanied by examples, for easy, medium, and hard levels for the English (multilingual) and Image Recognition capabilities. We provide at least examples to illustrate the distinctions and establish clear expectations for the different difficulty levels. They serve as instructional tools to help professional annotators from our data vendor better understand our difficulty level criteria, enabling them to annotate prompts of varying difficulty accurately.
D.6.1 ENGLISH AND MULTILINGUAL
Easy
Definition
Prompt a single askdreguiremeen/oonssraant for the model presented single state- ment OR prompt is single statement without ask/requiremen/coosstrinns AND would not require subject matter expertise to understand.
Examples
Illustrate and explain the proper use of semi-colon.
How uninvite my brother to my wedding?
I've been having trouble sticking to my healthy diet lately. Give some motivational words or tips to help me make better food choices and achieve my health goals.
Medium
Definition
Prompt includes 2-4 asse/regniieeeentsccossaaaann for the model AND would not quire subject matter expertise to produce response.
Examples
My neighbors blast loud music all night, and can't sleep. I've tried talking to them directly, as well as calling 311 but nothing has changed. What else do you think I can try?
How do I ask my boss for a raise? think I'm underpaid but my boss never has time for me.
Pretend you're re Bugs Bunny, I'm Elmer Fudd. How would you greet me?
Write me funny haiku about dogs.
Hard
Prompt contains or more axksfregiremennsscnsttrinss for the model OR requires subject matter expertise above and beyond "common knowledge" in order to respond.
Definition
Examples
Write a poem to say sorry to my dog because I didn't spend enough time with it. The poem should have 26 lines where each line begins with Z, Y, X, ..., A, respectively, and always ends with h. The poem cannot contain any animal words.
Sort the following words alphabetically, and in the result remove the first and the fourth words while capitalizing the rest: sioux fortescue purloin percept helmsman friend friends. Append a new lower-case word that is an animal living Antarctica. Output the result with numbered bullets.
Handling long-sequence inputs presents significant challenge to the KV-cache of Trans- formers. Can address this challenge better by training Transformers with more GPUs?
I'm hosting a dinner party next week. I have a kosher friend coming, but also a vegan friend. Also, I am allergic to nuts. My husband likes spicy food. There might be few picky eaters who are coming too. They may come with kids who attend preschools. What do you think should make for dinner? And what about drinks?
49

---

Published as a conference paper at ICLR 2025
D.6.2
IMAGE RECOGNITION
Easy
Definition Requires to stick to the image focus (instead of the background and details) AND requires NO external knowledge to answer the question (e.g. historical details, specific skills) AND requires NO fine-grained object recognition (e.g. plant species, aircraft models) AND requires NO complicated language format constraint (e.g. multi-level bullets, a specific order listing, creative writing).
Examples
How many pieces of chess are there? Please answer with one English word.
What is the title of the presentation slides?
What color is the watch belt?
Medium
Definition
Neither easy nor hard.
Examples
Who are the cartoon characters in this image? What are they holding? (what they hold is not image focus)
How do fix this home? (requires some knowledge)
What is the species of this cute cat? (fine-grained recognition)
Hard
Requires identification of five or more entities, each possessing distinct characteristics components, amid conditions of visual complexity. This includes scenarios where unrelated vi- sual elements may interfere (visual distraction), or where relevant parts the entities are partially obscured from view (visual occlusion), thus complicating the recognition process OR requires com- plicated format for language generation (e.g. multi-level bullets, a specific order of listing, creative writing) OR requires visual-related professional knowledge.
Definition
Examples
Can you count the balloons of each color? (The image includes more than five balloons along with distractions like children, hats, and other objects.)
It is said that there is a human face in this image. Can you explain how that can be? (comprehensive image understanding related to illusion)
When was this photo taken? Can you tell me more about the related event? (visual recog- nition related to history knowledge)
The distinction between Medium and Hard difficulty hinges on whether the prompt necessitates visual-related professional knowledge. It's important to note that prompt demanding professional knowledge does not automatically qualify as Hard unless that knowledge is specifically related to visual interpretation. For instance, if prompt can be deconstructed into part that solely concerns visual identification and another part that solely concerns factual knowledge, then the knowledge required is not considered visual-related.
50

---

Published as a conference paper at ICLR 2025
D.7 PROMPTS FOR EVALUATION
We provide the complete version of the system and evaluation prompts we adopt for LLM-as-a- Judge in Tables 34 and 35, respectively.
You are an expert Al evaluator tasked with assessing model responses. Rate the response using 1-5 Likert scale according to the following rubrics:
### Rubrics:
5/5 Amazing: The response is flawless and could hardly be improved.
-4/5 Pretty Good: The response quite good, but has room for minor improvements.
-3/5 Okay: They are middle-of-the-roee responses that could be improved in several ways.
22/ Pretty Bad: The response has major problems in helpfulness, truthfulness, or safety.
1/5 Horrible: They are terrible responses and you would caution others against using
models that generate responses like this.
Note: User prompts model responses may include attachments. To ensure thorough evalua- tion, you may need write and execute code.
Table 34: System prompt for LLM-as-a-Judge in our CROSSEVAL benchmark.
51

---

Published as a conference paper at ICLR 2025
[Attached]:
(attached text]
[User Prompt]:
(user prompt):
To calibrate your evaluation, consider these reference examples:
[Reference Example I]:
Model Response: (rexponse 7}
Rating I: fratnng 1 Explanation 1: (epplanation 1}
Rating 2: frating 2/5 Explanation 2: eexplanation 2}
[Reference Example 2]:
Model Response: {response 2}
Rating 1: [rating 1 Explanation 1: (explanation 1}
Rating 2:
frating 2)5 Explanation 2: (explanation 2}
[Reference Example 3]:
Model Response: fresponse 3}
Rating 1: [rating 1 Explanation 1: (explanation 1)
Rating (rating 2)/5 Explanation 2: (explanation 2)
Use these examples as benchmarks for your evaluation scale and scoring consistency. Here the model response for evaluation:
[Model Response to be Evaluated]: {model response)
Please provide your evaluation in the following format:
##### User Prompt Analysis
Identify key requirements and objectives from user prompt.
#*## Reference Examples Insights
Summarize scoring patterns and typical point deductions.
Include how many points should be deducted for each issue.
#### Model Response Evaluation
Pros: List strengths and positive aspects.
Cons: Identify weaknesses, specifying point deductions for each.
##### Holistic Assessment
CConsider if major strengths outweigh minor issues. Combine similar deductions to avoid double penalization. Balance deductions and positive aspects, and then explain your scoring decision.
#### Evaluation Score Score: [X|5
Ensure your evaluation is thorough, fair, and aligned with the reference examples. Your expertise is crucial in providing accurate and insightful assessment.
Table 35: Evaluation prompt for L.LM-as-a-Judge in our CROSSEVAL benchmark.
52

---

Published as a conference paper at ICLR 2025
D.8 MORE DISCUSSIONS ON LLM EVALUATOR
Discussion on Tool Use. In the benchmark, prompts related to tool use involve functionalities such web browsing and code interpretation. However, the LLM APIs we experiment with do support web browsing, and only the GPT-4 API supports code interpreters.
Fortunately, when specify the date of the reference examples and indicate that the answers may be dynamic, LLMs without web browsing features can still serve as effective evaluators, achieving Pearson correlations above 0.6 across all tool use-related capabilities. Additionally, enabling GPT's code interpreter results in similar correlation scores but incurs higher costs. This may be because the reference examples already provide sufficient context for evaluation, eliminating the need for the model to execute code. As a result, we disable the code interpreter in subsequent evaluations.
Final Evaluator Selection. Table shows that different LLMS excel at different capabilities. This naturally leads to the idea of using a mixture of LLMs as evaluators. For example, we could use Claude 3.5 to evaluate Spanish & Image Recognition and GPT-4o to evaluate Spanish & Reasoning, aiming for higher overall correlation. However, this approach proves impractical due to significant differences in scoring distributions across models: Claude 3.5 tends to give higher scores, while GPT-4o is more stringent. While this discrepancy is not issue when presenting single score for the benchmark, it poses issues when analyzing the relationship between individual and cross- capability performance. The varying scoring distributions could make conclusions unreliable. As result, we select GPT-4o as our final evaluator, while providing the results using Claude 3.5 in the Appendix E.2 for reference.
53

---

Published as a conference paper at ICLR 2025
D.9 CASE STUDY FOR LLM-AS-A-JUDGE ON CROSSEVAL
We randomly select one instance from the CROSSEVAL benchmark to demonstrate the format and reliability of LLM-as-a-Judge in Tables 36 and 37, using GPT-4o and Claude 3.5 Sonnet evalua- tors, respectively.
Prompt
The distance hetween cities and is 240 miles. train leaves from city X for city and another train leaves from city Y to City X simaltancously. known train traveling 15 miles per hour faster than the other. After they find they 90 miles upart. Can we calculate the speeds of both trains?
Model Response from ol-preview
The detailed caleulation process omitted for breuity.
Answer: Yes: their speeds are 17.5 mph and 32.5 mph-we can caleulate both train speeds from the given data.
GPT-as-a-Iudge
BNUN User Prompt Analysis
user prompt provides a scenario involving two trains traveling towards from different cities, one train traveling faster than the other. The key requirements determine the speeds of both trains given the distance between the citics, the time traveled, and the distance upart alter hours.
BAUH Reference Examples Insights
Reference Example Deducted points incorrect calculations and assumptions.
Reference Example Deducted points for incorrect assumptions about city Z and addressing potential confusion. Reference Example Deducted points for not solving the problem but correctly identifying the need more information.
Typical point deductions:
Incorrect calculations: points.
Incorrect assumptions failure address potential confusion: 1-2 points.
Failure to solve the problem when solvable: 2-3 points.
#### Model Response Evaluation
Pros:
1. Logical Setup: The response correclly up the problem by assuming the trains moving lowards each other along the line
hetween cities X and Y.
Correct Calculations: response accurately calculates the combined distance covered and the combined speed.
3. Clear Explanation: The steps are clearly explained, making
to follow the logic and calculations.
Cons:
1. Assumption Clarification: The response assumes that city 7 is on the same line as cities and Y without explicitly addresssng the potential confusion prompt. This could lead minor deduction for clarifying the assumption point).
##U# Holistic Assesment
The response is strong logical setup, clear explanation, and correct culculations. The minor the lack of explicit clorification about the assumption regarding city Z. However, this does not significantly detract from the overall quality of the response.
#### Evaluation Score Score:
The response is quite good, with only a minor area for improvement in clarifying assumptions. The major strengths outweigh this minor leading high score.
Table 36: Case study of GPT-as-a-Judge.
54

---

Published as a conference paper at ICLR 2025
Prompt
Cupahility: Reasoning
LI Category: Mathematical Reasoning
L2 Category: Math word problem solving
Difficulty; Hard
Prompt: The distance between citics X and Y is 240 miles. train leaves from city for city Z and another train leaves from city Y to traveling 15 miles per faster other. hours, they they 90 miles apart. Can we calculate the speeds of both trains?
City X simultancously.
Model Response from ol-preview
To determine the speeds of both trains, let's reconsider the setup carefully. The distance between citics X and 240 miles. train leaves from city X for city Z, and another train leaves from city Y towards city X simultanzously. Since they are on the same rail line trains moving Lowards other along line.
X and it's logical assume
The detailed calculation process omitted for brevity.
Answer: Yes; their speeds are 17.5 mph and 32.5 mph we can calenlate both train speeds from the given data.
Claude-as-a- Judge
#### User Prompt Analysis
The user prompi requires calculating the speeds of two trains given:
Distance between citics X and Y (240 miles)
One train leaves X for Z. another leaves for X
One train is mph faster than the other miles apart
Affter hours, they
Key objective: Deternine it's possible calculate train speeds given information.
#### Reference Examples Insights
Major point dedluctions points) failing recugnize umbiguity city
Minor deductions (1 point) for calculation erross misinterpretation of given data
Full points for rocognizing problem cannot be solved without additional information
Partial credit for correct calculations assuming linear path between X and
#АUN Model Response Evaluation
Pros:
Conrectly identifies ambiguity in the original prompt regarding city
Assumes linear path between and Y. which is reasenable interpretation
Calculations are mathematically correct given this assumption
Clearlly explains step-by-stop process
Cons:
Fails explicilly state that the original problem unsolvable without information about (-1 point)
Docsn't It acknowledge that the assumption of lincar path is an interpretation, not given fact (-0.5 points)
#### Holistic Assessment
The model response demonstrates good understanding of the problem and provides a mathematically sound solution based on reason- able interpretation. While doesn't explicilly address the ambiguity in the original prompt, it does implicitly recugnize by making assumption about train paths. The calculations and reasoning are clear and correct.
The main weakness the lack of explicit scknowledgment the original problem's ambiguity. However, the strength of the mathematical reasoning and the clarity of the explanation partially compensate for this oversight.
#804 Evalnation Score Score: 3,5/5
This score reflects the model's strong mathematical approach and clear explanation, balanced against the lock of explicir discussion about original problem's ambiguity. solid response could improved by directly addressing limitations of the given information.
Table 37: Case study of Claude-as-a-Judge.
55

---

Published as a conference paper at ICLR 2025
E
"LAW OF THE WEAKEST LINK" OF CROSS CAPABILITIES IN LLMS
E.1 EXPERIMENTAL SETUP
To ensure comprehensive coverage of LLM performance across capabilities, we select 17 models from five major model families: GPT (OpenAI, 2023), Claude (Anthropic, 2024), Gemini (Reid al., 2024), Llama (Llama Team, 2024), and Reka (Ormazabal et al., 2024). Each model supports at least five cross-capability scenarios in our experiments (except ol models). For consistency, use the GPT-40-05-16 model as the evaluator, with temperature set and seed set to 42 to ensure deterministic scoring. Each model's ' responses to achieve optimal performance. For the Llama 3.1 405B model, we specifically use the FP8 version. A complete list of model versions is provided Table 38. In addition, while the Gemini API supports code interpreter functionality, it does not yet handle non-text outputs (e.g., data plots), so we exclude its results on rellateee prompts in benchmark.
generated using their default decoding parameters
LLM Name
Model Version
GPT-4o-mini
gpI-4o-minii22224-07-BB
GPT-4o
gpt-4o-2024-05-13
ol-mini
o1-mini-2024-09-12
ol-preview
ol-preview-22224-9912
Claude 3 Haiku
cladd=3-haikk--20240007
Claude 3 Sonnet
chaub-3-sommee-222240299
Claude 3 Opus
claud-33-opus-22240229
Claude 3.5 Sonnet
cand--3-5-somee222244600
Gemini 1.5 Flash
gemini-1..-flash-000
Gemini 1.5 Pro
semini-1.5-proo-1
Gemini 1.5 Pro Exp
gemini-l.5-pro-exp00000
Reka Edge
reka-edgg-202240228
Reka Flash
reka reka-flash-202200722
Reka Core
reka-core-20240722
Llama 8B
Llama 3.1 8B
Ilama 70B
Llama 3.1 70B
Llama 405B
Llama 3.1 40SB FP8
Table 38: Exact LLMs versions used in experiment.
E.2 RESULTS FOR CLAUDE-AS-A-JUDOE
To avoid potential bias from using single evaluator, present all results with Claude 3.5 Sonnet as the evaluator in Table 39. Notably, for the Coding Reasoning task, the performance of five models falls between the weak and strong capabilities but tends be closer to the strong one, as highlighted in purple in the Table. This may be due to the fact that Coding and Reasoning are key capabilities in current LLM development, with potentially many cross-capability prompts included in the training data, boosting LLM performance for this specific task. While this pattern does not appear in the GPT-as-a-Judge results, Claude-as-a- auddeee still generally demonstrates clear "Law of the Weakest Link" effect, as illustrated by the distribution in Figure 3.
56

---

Published as a conference paper at ICLR 2025
Individual Capabilities
Models
English
Reasoning
Coding
Image
Tool
Long Context
Spanish
Clande 3 Haiku
64.92
58.17
67. 08.0
56.60
70.80
66.03
Clande Sonnet
72.48
64.36
73.12
62.75
72.03
69.99
Claude Opus
73.17
69.37
74.74
65.42
73.77
75.86
Clando 3.5 Sonnct
78.22
76.52
77.37
77.70
76.60
76.03
GPT-4o mini
76.13
68.74
75.81
68.51
78.20
76.18
GPT-4o
78.60
74.69
76.43
77.35
83.48
80.63
el-mini
77.28
84.28
87.01
83.39
83.80
ol-preview
82,63
88.85
86.49
86.70
86.24
1.5
70.62
73.79
56.56
72.28
Gemini 1.5
75,93
75.19
73.86
79.32
Gemini 1.5 Pro Exp
77.42
75.61
75.62
76.67
80.11
80.87
Reka Edge
$1.86
44.07
43.87
53.41
35.46
$3.31
Reka Flash
63.29
62.36
64.37
61.14
53.22
70.60
Reka Core
73.77
72.44
70.14
60.21
62.69
74.24
Llama 3.1 8B
67.11
55.26
67. 02
47.22
65.29
60.15
Llama
75.51
50.38
Cruss Capubilities
Models
Coding & Rea.
Image & Rea. Long & Coding Spanish & Rea. Spanish & Image Tool & Coding Tool & Rea.
Clande Haiku
66.03
56.38
65.85
59.29
58.73
Clande Sonnet
70.19
60.52
67.27
61.81
65.85
Clande Opus
70.32
59.94
68.65
71.67
65.61
Claude 3.5 Sonnct
78.60
77.92
74.31
76.12
79.25
GPT-4o mini
74.82
71.31
69.42
67.83
GPT-4o
75.19
78.79
73.64
76.28
78.08
48.27
58.21
ol-mini
85.89
83.48
83.29
el o1-preview
87,38
83.74
86.25
Gemini 1.5 Flash
71,54
54.71
69.31
68.82
53.66
Gemini 1.5 Pro
76.13
73.25
74.01
72.60
67.24
Gemini 1.5 Pro
73.15
76.29
72.79
74.51
73.01
Reka Edge
47.63
30.64
23.03
39.18
43.88
Reka Flash
64.67
47.47
45.25
62.01
57.23
Reka Core
69.83
50.87
50.00
68.99
56.56
Llama 3.1 8B
61.39
$5.26
47.65
$1.49
50.17
Llama 3.1 70B
71.65
60.09
58.61
$1.49
51.99
Llama 3.1 405B
71.85
64.42
67.40
56.13
57.03
Table 39: Experimental results for individual and cross capabilities CROSSEVAL using Claude as the evaluator. To avoid potential evaluator bias, we present Claude's results solely as reference point and bold the best non-Claude results. In cross-capability evaluations, define one of the involved individual capabilities as stronger and the other as weaker if the absolute score dif- ference between them exceeds A points. In 48 cross-capability scenarios where this difference present, cases show performance lower than both individual capabilities red background 18 show performance between the two but closer to the weaker capability (buue background ), and 6 show performance closer to the stronger capability ( purple background) ), Notably, no cross- capability score ever exceeds the stronger individual capability.
57

---

Published as a conference paper at ICLR 2025
E.3 DISCUSSION DISTINGUISHING "WEAK" AND "STRONG" CAPABILITIES
In the experiments presented in the main text, we identify "strong" and "weak" capabilities within cross-capability tasks when the absolute difference between their individual scores exceeds A 3. To illustrate the effect of on "Law of the Weakest Link," we adjust its value from to 6 and plot the density distribution using GPT-4o and Claude 3.5 Sonnet as evaluators, as shown in Figures and 5, Notably, regardless of the chosen A value, crpabicapa performance consistently clusters around the weaker performance, clearly demonstrating the "Law of the Weakest Link."
feformncc Lapaality
Sinn Furfammnce Lapaality
112
Density
Performance
Cross Performance
(a) A=1
A=2
Sigg Feformence Weot
Sign Fertrmance
anwe Cauiliiiyy
Dunalty
Perfomaanc
Cross Performance
(c) A=3
(d) A=4
Wast Capsalitt
West Capaciity
-22
Danalty
Penaity
Cross Perfomance
Cross Performance
(e) A=5
( A=6
Figure 4: Effect of A on the density distribution of cross-capability performance evaluated by GPT-4o.
58

---

Published as a conference paper at ICLR 2025
Sing Fartrmance Stowe Caxililly
Crasuity
2014
Dunalty
-306
Performance
Cross Performance
(a) A=1
(b) A=2
Sigle Nertrranaa
West Lopaciity
sieng Expasiiiy
6228
628
Denatty
6328
Cross Parformance
Cross Performance
cc) A=
(d) A=4
Sege Pnthrnance Capatilly
Capacility
5.5
-XX
Dunsity
-015
300
(e) A=S
A=6
Figure 5: Effect of A on the density distribution of cross-capability performance evaluated by
Claude 3,5 Sonnet.
59

---

Published as a conference paper at ICLR 2025
E.4 RESULTS FOR DIFFERENT DIFFICULTY LEVELS
In Table 40, we present the scores of 17 models across prompt sets with varying levels of difficulty. It's important note that these scores are not directly comparable across different model families, as they support varying capabilities. For instance, while Llama does not support Image Recognition, it covers all capabilities related Tool Use.
Nevertheless, as shown in Table 40, 12 out of the 17 models perform better on the Easy prompt compared the Medium set, and similarly, they score higher on the Medium set than the Hard set. This pattern suggests that the difficulty levels we manually defined align well with model performance. An exception to this trend the Claude model family, where all four Claude models scored slightly higher on the Hard prompt set than on the Medium set.
Models
Easy
Medium
Hard
GPT-4o mini
75.14
69.51
69.00
GPT-4o
76.87
69.82
68.97
ol-mini
82.52
78.63
77.75
ol ol-preview
84.07
79.25
78.07
Claude Haiku
64.23
57.85
58.58
Claude Sonnet
68.16
61.41
62.86
Claude Opus
72.21
64.41
65.17
Claude 3.5 Sonnet
76.94
70.86
70.91
Gemini 1.5 Flash
71.65
62.28
61,86
Gemini 1.5 Pro
76.60
68.26
69.44
Gemini 1.5 Pro Exp
79.86
71.20
70.08
Reka Edge
45.89
40.71
40.43
Reka Flash
60.74
55.86
55.57
Reka Core
60.69
59.32
Llama 8B
55.00
52.34
50.69
Llama
64.00
61.06
57.94
Llama 3.1 405B
70.41
62.63
61.77
Table 40: Results of different difficulty levels evaluated by GPT-4o.
60

---

Published as a conference paper at ICLR 2025
F INVESTIGATING INDIVIDUAL-CAPABILII ALTERATIONS
F.1 PROMPT TO GENERATE PRINCIPLE
The complete prompt used for automatically generating principles provided in Table 41.
You Al expert tasked with analyzing common mistakes in model responses and creating compre- hensive set principles to improve the (capability) guideline. Specifically, for each iteration, will provide you with one instance, and you need to update the current principles accordingly. There are 100 instances in total, and the principles should be completed after reviewing all instances.
model. We will work step-by-step build this
For each instance, you have the following information:
User Prompt
Model Response
Evaluation of the Model Response
Current Principles
### Instance {index}
currrntt instance, including the user prompt, model response, and evaluation using LLM-as-a-judge)
For each itcration, choose ONE the following actions:
1. ADD
Introduce new principle that isn't currently listed.
2. REPLACE
Replace less significant principle with one.
Clearly specify which principle being replaced.
3. REVISE
Enhance the principles hy making them more detailed and specific.
4. KEEP
the current instance already covered by existing principles, leave the guideline unchanged.
Current Principles:
{current principles}:
Output Format:
## Summary
Summarize major issues with the present response.
Provide specific, actionable steps to prevent these errors, if any.
Based on your summary and the current principles, decide which action (ADD, REPLACE, REVISE, or KEEP) should be taken for the current instance.
## Principles for Prompts related {capability) ### Principle 1: Title [Use the title specify the context in which this principle should be applied, such as "For Legal Reasoning" or "For Mathematical Reasoning"]
IInclude up three key points.
Each point should be directly applicable the modcl's generation process without requiring additional training or resources.
Each point must be extremely specific allow for direct execution.
For example, instead of saying "Use structured markdown format," clearly define the exact format
each step, including structure the beginning, middle, and end.
Instead advising to "avoid vague terms." provide a specific list terms be avoided. Rather than generally suggesting "avoid errors in math calculations" or "double-check." outline concrete steps to prevent such emror..
[END of Principles]
Requirements:
Follow the output format exactly, including "[END of Principles]" at the end with no remarks after it.
Include up to 10 distinct principles in the report. If there are already 10 principles, "ADD" is not allowed.
You may reorder the principles necessary: Place important, typical, and representative principles at the front, while less important ones can be moved toward the back. Ensure that each suggestion in the principles is detailed and actionable, rather than being general description.
Table 41: Prompt for generating principles based on responses from CROSSEVAL.
61

---

Published as a conference paper at ICLR 2025
F.2 CASE STUDY FOR PRINCIPLE-BASED SYSTEM PROMPTS
Using Gemini 1.5 Flash as an example, we present the automatically generated system prompts for the Reasoning capability in Tables 42 44. The "Note" in Table 44 is added manually.
## Principles for Prompts related Reasoning
##* Principle 1: Mathematical Reasoning
Verify All Mathematical Steps and Properties Thoroughly:
1. Validate each step in mathematical derivation meticulously, focusing on crucial values, properties, and boundary conditions.
Ensure consisteney in use of all variables constants across the steps.
3. Verify accuracy of factorization and simplification steps.
Detail Intermediate Calculations and Logical Steps Clearly:
1. Show all steps in complex calculations for transparency and clarity.
2. Justify intermediate steps thoroughly. explicitly stating relevant formulas and boundary terms.
Identify and Correct Misleading Statements and Errors:
1. Scrutinize claims about variable independence, solution behavior, or mathematical properties for accuracy.
2. Correct misinterpretations about connectivity or group properties for given spaces.
Complete Execution Methods:
Ensure final steps methods like finding steady-state vectors are executed.
2. Provide exact values and solutions without leaving the explanation incomplete.
Balance Thoroughness and Conciseness:
1. Ensure comprehensive coverage of essential steps without overeexplaining redundancy. Avoid unnecessary repetition and condense explanations suitably while maintaining clarity. Leverage Degree Mismatches:
1. Use polynomial degree comparisons to simplify analysis, especially when identifying potential func- tion inverses.
2. Note that two polynomials of different degrees cannot he inverses of each other.
### Principle 2: For Proving Statements Involving the Pigeonhole Principle
Explicitly State the Contradiction:
1. Clearly state why having all elements unique causes contradiction in the problem's context.
Provide specific examples necessary to illustrate the contradiction.
Outline Logical Assumptions:
1. Clearly state every logical assumption at the start of the proof.
2. Reiterate these assumptions when reaching a conclusion to reinforce the logic.
Detail the Application of Principles:
Clearly detail how and why the pigeonhole principle is applied, linking each step back the problem's s context.
### Principle 3: For Logical Sequeneing and Step-by-Step Explanations Detail Logical Steps Clearly:
1. Ensure each step is explained detail, showing how one leads to the next.
2. Break down complex proofs or problems into components, revealing the underlying reasoning.
Explicitly Address and Evaluate Assumptions:
1. Clearly state assumptions made and justify their relevance to the problem.
Evaluate each assumption for feasibility and update reasoning if new information revealed.
3. Explain how these assumptions influence conclusions drawn.
Incorporate Relevant Legal and Logical Principles:
I. Include specific legal principles doctrines if applicable.
2. Explain these principles in context and link the problem's 's scenario.
Ensure Accurate Initial Dependency Analysis:
1. Validate initial dependencies comprchensively before analyzing post-ohservation changes.
Maintain Logical Cohesion:
1. Ensure explanation maintains logical flow from start finish.
Avoid ambiguities, ensuring each point connects clearly the next.
Comprehensive Coverage: Address all potential dependencies and independencies to ensure logical completeness.
Table 42: System prompt (Principles 1-3) automatically generated to enhance the reasoning capability of Gemini 1.5 Flash.
62

---

Published as a conference paper at ICLR 2025
## Principles for Prompts related Reasoning
### Principle 4: Addressing Ambiguities and Considering Multiple Possibilities
Identify and Resolve Ambiguities:
1. Point out any ambiguous terms or conditions within the problem statement.
2. Clearly state how these ambiguitics resolved.
Make Assumptions Clear:
1. Articulate any assumptions made to proceed with the solution.
2. Justify why these assumptions are reasonable and how they influence results.
Evaluate All Possible Correct Answers:
1. Ensure that all potential correct
considered and evaluated.
multiple answers are possible, acknowledge them explicitly and explain why each is plausible.
Re-Evaluate Intermediate Assumptions:
1. Consistently check interim assumptions for feasibility as the solution progresses.
2. Correct initial assumptions if they fail align with further logical deductions.
### Principle 5: Financial Analysis and Reasoning
Step-by-Step Financial Calculations:
1. Break down financial calculations into detailed, transparent steps.
2. Show intermediate steps clearly, not just the final results.
Compare Different Scenarios:
1. Provide comparisons of diffcrent financial scenarios when applicable.
Highlight Key Conclusions:
1. Summarize key financial implications explicitly.
Tailor Negotiation Strategies:
Provide negotiation tactics specific to each buyer's unique offer. Include concrete phrases or tactics the user can use.
3. Justify financial recommendations clearly within the user's context.
#*# Principle 6: For Hypothesis Development in Scientifie Contexts
Ensure Comprehensive Factor Coverage:
1. Verify all specific factors mentioned in the prompt are addressed.
Avoid Redundancy:
1. Consolidate related points prevent repetition.
Provide Clear Hypotheses:
1. Present hypotheses clcarly and in testable terms.
### Principle 7: For Scientific Reasoning and Empirical Analysis
Verify the Existence of Citations:
Confirm citations are based on actual research papers, cross-referencing with recognized academic databases.
2. Avoid inventing or hallucinating studies: confirm publication details before citing.
Summarize Study Findings Accurately:
Provide specific results and data points from studies to back claims.
Include relevant figures or outcomes from cited studies for greater reliability.
Incorporate Empirical Evidence:
1. Support scientific claims with relevant empirical evidence and citations.
2. Avoid overgenerallzationns; specific examples case studies.
Table 43: System prompt (Principles 4-7) automatically generated to enhance the reasoning
capability of Gemini 1.5 Flash.
63

---

Published as a conference paper at ICLR 2025
## Principles for Prompts related to Reasoning
### Principle 8: For Designing Scientific Experiments
Detail the Measurement Methods:
1. Specify tools and procedures for measuring each variable.
2. Include details like frequency of measurements and exact techniques used.
Clarify Statistical Analysis:
Explain how statistical tests will analyze collected data.
2. Provide details on data preparation and results interpretation.
Verify Citations:
1. Ensure all literature references verifiable and credible.
Cross-reference cited studies with recognized academic datahases.
### Principle 9: For Summarizing and Analyzing Policies Incorporate Procedural Details:
1. Include specific procedural elements like roles and responsibilities.
Enhance comparisons with explicit distinctions and summarize key differences and similaritics.
3. Incorporate references to relevant cases or statutes.
Avoid Redundancy:
1. Consolidate related information to avoid repetition.
Include Interpretative Analysis:
Interpret how regulations impact the environment they govern.
2. Clarify the rationale or feasibility of suggested legal arguments.
Address All Policy Elements:
1. Summarize all major sections, including scope, administration, restrictions, and enforcement.
### Principle 10: For Ethical Reasoning
Avoid Redundancy:
1. Consolidate related ethical advice.
Incorporate Ethical Theories:
Explicitly mention ethical theories like consequentialism, deontology, and virtue ethics.
Focus on Specific Actionable Steps:
1. Provide detailed steps for addressing ethical issues.
Note:
Apply the principles above to generate better responses for user prompts that require reasoning.
For prompts that do not require reasoning, disregard these principles.
Avoid quoting or referencing these principles, as the user is not aware of its existence.
[ND of Reasoning Principles]
Table 44: System prompt (Principles 8-10) automatically generated to enhance the reasoning capability of Gemini 1.5 Flash.
64


Reviews:
[{"review_id": "0iEz53ZlCn", "reviewer": "Reviewer_7cFP", "summary": "The goal of the paper is to investigate how LLMs perform at the intersection of various capabilities. The paper starts off by remarking that most of the LLM evaluations tend to focus on single capabilities like long context reasoning and factual knowledge retrieval. In practice, however, LLMs may be used to perform tasks that leverage several such capabilities simultaneously. The main contribution of the paper is a meticulously contracted, human-annotated dataset. The dataset consists of prompts that aim to test one or more capabilities. In addition to the input prompts, the paper also asks humans to annotate responses by various SoTA LLMs. Noting the relatively low degree of inter-annotator agreement, the paper improves the annotation guidelines to increase the agreement. Next, the paper design effective prompting strategies for using LLMs as evaluators. The results of the experiments are quite interesting: the performance of the LLMs on compound capabilities correlates with the performance on the individual capabilities.", "questions": "Please see the questions under the \"Weaknesses heading\".", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is quite correct in pointing out that benchmarks have not kept up with the actual usage of LLMs in the wild. With that in mind, a systematic approach for studying the cross-capability performance of the models will be very much appreciated by the LLM community.\n\n2. The paper is well-written and quite easy to follow. The paper provides illustrative examples at pretty much every step.\n\n3. The human surveys are quite thorough. The authors spent significant effort in understanding the results and improving the annotation performance.\n\n4. The model-based evaluation protocol in Section 3.3 is very meticulously designed. The protocol exposes many interesting choices (though some are from existing papers) which will be enhance readers' knowledge on conducting effective model based evals.", "weaknesses": "While the paper is a pleasure to read, and generally well-executed, I feel there are two main issues that still stand in the way of acceptance, namely, lack of precise definitions and lack of important details.\n\n## Lack of precise definitions\n\nIt is very difficult to understand what counts as a capability, and the paper doesn't not discuss it in sufficient detail. How should a capability be theoretically defined? Are different MMLU categories, e.g., math, medicine, individual capabilities? Is sitting in the SAT exam a single capability? It does require a bunch of skills like recalling meaning of words and basic math formulas, analytical reasoning and equation solving. Having a precise definition of a capability seems quite important in understanding how generalizable the insights of the paper are.\n\nThe reader is left asking why certain skills are designated as different capabilities and where really the boundaries are. For instance, why are English and Spanish two different capabilities? Will all languages be different capabilities? What if the languages are closely related to each other, e.g., Hindi & Urdu, German & Dutch?\n\nOn a related note, do we assume that the capabilities are equally different from each other? For instance, coding and Image Recognition (different input modalities) seems much more different than English vs. Spanish (same modality, even some overlap of words and grammar). Is there a notion of capability Gerrymandering (akin to [fairness Gerrymandering](https://arxiv.org/abs/1711.05144)) where one could form cross capabilities in various ways to show completely different results?\n\n## Missing details\n\nThe paper seems to be missing some critical details which are important to assessing the generalizability of the results.\n\n1. How were the capabilities selected? What was left out? For instance, why not consider Long Context and Language?\n\n2. Also, how are the capabilities broken down into Levels 1 and 2? How exhaustive is this breakdown?\n\n3. How were the examples for annotators generated (Appendix B.4)? Could these example prime the annotators? If these examples are not from the benchmark (as the text notes), what is their propose?\n\n4. Line 201: \"selecting a leaf node from our taxonomy to categorize each prompt by capability\". Where do these prompts come from?\n\n5. How much coverage do these prompts provide of each capability?\n\n6. Line 211: There are 100 to 500 annotated prompts for each capability, but then why are the reviewers asked to select a precise 100 examples out of it? Is it possible that some capabilities have better quality annotations than others because different sets of prompts were forced to be downsampled to a round number of 100? Why not set a fixed quality criteria and let all the prompts through even if the final number if higher / lower than 100?\n\n\n### Small nits\n\n3. Line 156: Why is “Brainstorming about Local and Current Things” a part of the Tool Use? Could you provide examples of tools that perform brainstorming?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2jMlvZpcOp", "reviewer": "Reviewer_A1B6", "summary": "This paper constructs a benchmark for the core and cross capabilities. It identifies seven core capabilities and proposes seven cross-capabilities based on them. The author proposes CROSSEVAL to evaluate each capability. Throughout the extensive experiment, the author claims the Law of the Weakrdy link, meaning that the cross capability is heavily influenced by the weakest core capability associated with it.", "questions": "1. What are the criteria for selecting core capabilities and cross capabilities? It appears the these capabilities are somehow randomly selected without proper justification. For instance, why are *English* and *Spanish* selected as core capabilities like *long-context* instead of creating a different dimension named *lingual*?\n2. Won't this paper be more suitable for a benchmark track instead of a research paper?\n3. How is the image encoded in LLM? Does the LLM include other modalities?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The topic of this paper appears to be interesting. A proper taxonomy of the ability of LLM is necessary.\n2. The paper writing is good. The whole paper flow is easy to follow.", "weaknesses": "1. The reviewer fails to understand why constructing prompts for core capabilities. Don't there exist many benchmarks for specific core capabilities?\n2. The reviewer doesn't understand why each capability is comparable to the others, considering the difference in prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DHwPlSM04I", "reviewer": "Reviewer_Gjoy", "summary": "This paper investigates cross-capabilities - the intersection of multiple distinct abilities across different types of expertise needed to solve complex, real-world tasks. The authors comprehensively define individual and cross-capabilities in LLMs and carefully annotate prompts to evaluate these cross-capabilities. Their findings reveal that current LLMs follow the \"Law of the Weakest Link,\" where cross-capability performance is significantly limited by the weakest component capability. Additionally, they discovered that improving the weakest capabilities leads to the greatest improvements in LLMs' general capabilities.", "questions": "Could you provide more explanation about:\n- The rationale for incorporating reference responses in Section 3.3.2\n- How this relates to correlation between capabilities?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The investigation of cross-capabilities is both interesting and important.\n- The prompt collection and annotation methodology is comprehensive and reliable.\n- The insights about the \"Law of the Weakest Link\" and the finding that \"improving the weakest capabilities leads to the greatest improvements\" are particularly valuable.", "weaknesses": "- This work focuses more on benchmarking rather than algorithm design. It would be more appropriately categorized as a data/benchmark paper.\n- The study's scale is somewhat limited, with a dataset of only 1,400 prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4oDfPh8fvx", "reviewer": "Reviewer_zpqw", "summary": "This paper introduces the concept of cross capabilities in LLMs. It defines seven individual and seven cross capabilities, creating a taxonomy to evaluate LLM performance. The authors introduce CROSSEVAL, a benchmark with 1,400 prompts to test both individual and cross capabilities. Results reveal that LLMs often follow the Law of the Weakest Link, where the weakest capability limits performance in cross-capability tasks.", "questions": "Can the authors clarify the selection criteria for annotators and their domain expertise? How did the authors address potential biases in human ratings and ensure the reliability of the annotations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) The paper provides a detailed taxonomy of individual and cross capabilities in LLMs.\n\n(2) The CROSSEVAL benchmark is a good contribution. It consists of 1,400 human-annotated prompts that rigorously test LLMs on individual and cross capabilities.\n\n(3) The paper identifies a critical pattern in LLM performance -- the Law of the Weakest Link. This result and finding are insightful.\n\n(4) The paper is well-written. And code and data are shared.", "weaknesses": "(1) Although the paper includes a multilingual aspect (with Spanish), the evaluation of cross capabilities involving languages other than English is limited. \n\n(2) The study concentrates on evaluating capabilities in pairs, but it does not investigate scenarios requiring the integration of more than two capabilities simultaneously.\n\n(3) Need a detailed analysis of specific errors LLMs make in cross-capability tasks. A breakdown of common failures for particular capability combinations could provide more insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)