Law of the Weakest Link: Cross Capabilities of Large Language Models
Authors: name: Ming Zhong; name: Aston Zhang; name: Xuewei Wang; name: Rui Hou; name: Wenhan Xiong; name: Chenguang Zhu; name: Zhengxing Chen; name: Liang Tan; name: Chloe Bi; name: Mike Lewis; name: Sravya Popuri; name: Sharan Narang; name: Melanie Kambadur; name: Dhruv Mahajan; name: Sergey Edunov; name: Jiawei Han; name: Laurens van der Maaten
Published: 2024-09-30 05:12:01

Abstract: The development and evaluation of Large Language Models (LLMs) have largely
focused on individual capabilities. However, this overlooks the intersection of
multiple abilities across different types of expertise that are often required
for real-world tasks, which we term cross capabilities. To systematically
explore this concept, we first define seven core individual capabilities and
then pair them to form seven common cross capabilities, each supported by a
manually constructed taxonomy. Building on these definitions, we introduce
CrossEval, a benchmark comprising 1,400 human-annotated prompts, with 100
prompts for each individual and cross capability. To ensure reliable
evaluation, we involve expert annotators to assess 4,200 model responses,
gathering 8,400 human ratings with detailed explanations to serve as reference
examples. Our findings reveal that, in both static evaluations and attempts to
enhance specific abilities, current LLMs consistently exhibit the "Law of the
Weakest Link," where cross-capability performance is significantly constrained
by the weakest component. Specifically, across 58 cross-capability scores from
17 models, 38 scores are lower than all individual capabilities, while 20 fall
between strong and weak, but closer to the weaker ability. These results
highlight the under-performance of LLMs in cross-capability tasks, making the
identification and improvement of the weakest capabilities a critical priority
for future research to optimize performance in complex, multi-dimensional
scenarios.