MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Authors: name: Guoli Yin; name: Haoping Bai; name: Shuang Ma; name: Feng Nan; name: Yanchao Sun; name: Zhaoyang Xu; name: Shen Ma; name: Jiarui Lu; name: Xiang Kong; name: Aonan Zhang; name: Dian Ang Yap; name: Yizhe zhang; name: Karsten Ahnert; name: Vik Kamath; name: Mathias Berglund; name: Dominic Walsh; name: Tobias Gindele; name: Juergen Wiest; name: Zhengfeng Lai; name: Xiaoming Wang; name: Jiulong Shan; name: Meng Cao; name: Ruoming Pang; name: Zirui Wang
Published: 2024-07-18 00:58:41

Abstract: Recent advances in large language models (LLMs) have increased the demand for
comprehensive benchmarks to evaluate their capabilities as human-like agents.
Existing benchmarks, while useful, often focus on specific application
scenarios, emphasizing task completion but failing to dissect the underlying
skills that drive these outcomes. This lack of granularity makes it difficult
to deeply discern where failures stem from. Additionally, setting up these
environments requires considerable effort, and issues of unreliability and
reproducibility sometimes arise, especially in interactive tasks. To address
these limitations, we introduce the Massive Multitask Agent Understanding
(MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need
for complex environment setups. It evaluates models across five domains,
including teal{Tool-use}, teal{Directed Acyclic Graph
(DAG) QA}, teal{Data Science and Machine Learning coding},
teal{Contest-level programming} and teal{Mathematics},
and covers five essential capabilities: orange{Understanding},
orange{Reasoning}, orange{Planning},
orange{Problem-solving}, and orange{Self-correction}.
With a total of 20 meticulously designed tasks encompassing over 3K distinct
prompts, MMAU provides a comprehensive framework for evaluating the strengths
and limitations of LLM agents. By testing 18 representative models on MMAU, we
provide deep and insightful analyses. Ultimately, MMAU not only sheds light on
the capabilities and limitations of LLM agents but also enhances the
interpretability of their performance. Datasets and evaluation scripts of MMAU
are released at https://github.com/apple/axlearn/docs/research/mmau.