MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
Authors: name: S Sakshi; name: Utkarsh Tyagi; name: Sonal Kumar; name: Ashish Seth; name: Ramaneswaran Selvakumar; name: Oriol Nieto; name: Ramani Duraiswami; name: Sreyan Ghosh; name: Dinesh Manocha
Published: 2024-10-24 21:20:10

Abstract: The ability to comprehend audio--which includes speech, non-speech sounds,
and music--is crucial for AI agents to interact effectively with the world. We
present MMAU, a novel benchmark designed to evaluate multimodal audio
understanding models on tasks requiring expert-level knowledge and complex
reasoning. MMAU comprises 10k carefully curated audio clips paired with
human-annotated natural language questions and answers spanning speech,
environmental sounds, and music. It includes information extraction and
reasoning questions, requiring models to demonstrate 27 distinct skills across
unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes
advanced perception and reasoning with domain-specific knowledge, challenging
models to tackle tasks akin to those faced by experts. We assess 18 open-source
and proprietary (Large) Audio-Language Models, demonstrating the significant
challenges posed by MMAU. Notably, even the most advanced Gemini Pro v1.5
achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio
achieves only 52.50%, highlighting considerable room for improvement. We
believe MMAU will drive the audio and multimodal research community to develop
more advanced audio understanding models capable of solving complex audio
tasks.