MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
Authors: S Sakshi
Utkarsh Tyagi
Sonal Kumar
Ashish Seth
Ramaneswaran Selvakumar
Oriol Nieto
Ramani Duraiswami
Sreyan Ghosh
Dinesh Manocha
Venue: ICLR 2025 Spotlight (2025)
Keywords: Benchmark
Audio Language Models
Complex Reasoning
Primary area: datasets and benchmarks
TL;DR: We introduce MMAU, a comprehensive benchmark evaluating multimodal audio understanding across speech, sound, and music, challenging models with 27 distinct skills requiring advanced perception, reasoning, and domain-specific knowledge.

Abstract: The ability to comprehend audio—which includes speech, non-speech sounds, and music—is crucial for AI agents to interact effectively with the world. We present MMAU, a novel benchmark designed to evaluate multimodal audio understanding models on tasks requiring expert-level knowledge and complex reasoning. MMAU comprises 10k carefully curated audio clips paired with human-annotated natural language questions and answers spanning speech, environmental sounds, and music. It includes information extraction and reasoning questions, requiring models to demonstrate 27 distinct skills across unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes advanced perception and reasoning with domain-specific knowledge, challenging models to tackle tasks akin to those faced by experts. We assess 18 open-source and proprietary (Large) Audio-Language Models, demonstrating the significant challenges
posed by MMAU. Notably, even the most advanced Gemini 2.0 Flash achieves only 59.93% accuracy, and the state-of-the-art open-source Qwen2-Audio achieves only 52.50%, highlighting considerable room for improvement. We believe MMAU will drive the audio and multimodal research community to develop more advanced audio understanding models capable of solving complex audio tasks.

Published as a conference paper at ICLR 2025
MMAU: A MASSIVE MULTI-TASK AUDIO UNDER- STANDING AND REASONING BENCHMARK
S Sakshi Utkarsh Tyagi " Sonal Kumar Ashish Seth Ramannewaaan Selvakumar"
Oriol * Ramani Duraiswami* Sreyan Ghosh Dinesh Manocha
University of Maryland, College Park, USA * Adobb,, USA
* Equal Contribution Equal Advising (ssaksti.sonakkuu.sreaall) Sonteyaann@@uuuuueee @umd.edu
Project: https://sakst. githhhtthhb........ io/ mmau homepage/
Comprehensive Skill
Extensive Domain Coverage
Diverse Task Types
Speech
Music
Speaker Role Mapping
Tongue twisters
Understanding
World Knowledge
Sound
iibh Scene Understanding
Socio-cultural Muxic Understanding
Reasoning
Figure 1: Overview of the MMAU Benchmark. MMAU provides comprehensive coverage across three key domains: speech, sounds, and music, featuring diverse audio samples. It challenges multimodal L.MSs with tasks across 27 distinct skills, requiring advanced audio perception, reasoning, and comain-specifte knowledge.
ABSTRACT
The ability to comprehend audio, which includes speech, non-speech sounds, and music, is crucial for Al agents interact effectively with the world. We present MMAU, a novel benchmark designed to evaluate multimodal audio un- derstanding models tasks requiring expert-level knowledge and complex rea- soning. MMAU comprises 10k carefully curated audio clips paired with human- annotated natural language questions and answers spanning speech, environmen- tal sounds, and music. It includes information extraction]] and reasoning 2 ques- tions, requiring models demonstrate 27 distinct skills across unique and chal- lenging tasks. Unlike existing benchmarks, MMAU emphasizes advanced percep- tion and reasoning with domain-specific knowledge, challenging models to tackle tasks akin to those faced by experts. We assess 18 open-source and proprietary (Large) Audio-Language Models, demonstrating the significant challenges posed by MMAU. Notably, even the most advanced Gemini 2.0 Flash achieves only 59. 59.93% accuracy, and the state-of-the-arr open-source Qwen2-Audio achieves only 52.50%, highlighting considerable room for improvement. We believe MMAU will drive the audio and multimodal research community to develop more advanced audio understanding models capable of solving complex audio tasks.
INTRODUCTION
The recent advancements in Large Language Models (LLMs) have fueled discussions around the de- velopment of generalist Al agents, often referred to as Artificial General Intelligence (AGI), capable of solving a diverse range of complex understanding and reasoning tasks (Chowdhery et al., [023) Achiam al. 2023) Touvron al. 2023a). These developments have given rise to Al systems that can match or even surpass hwwan- expert performance in various natural language understanding and
We define information extraction question as one that requires deep understanding of audio, detailed content analysis, and the application external world knowledge when necessary.
We define reasoning question one that requires intentional, complex thinking beyond basic content understanding, analysis, and knowledge application, simulating expert-level cognitive processes.

---

Published as a conference paper at ICLR 2025
Sound
Speech
Music
Info-extraction- Eco-Acoustic Knowledge
Info-extraction Phosological Sequesee Dacading
lufo-cstraction Harmany Chord
Question: What natural environment most
Prograssins
likely reprosented by the audio?
Question: Fox the gives tongue tister ideatify which word appouss tist!
Question: Which choud progression is weed in the audio?
A. serene forest
B. quiet lihmay
aluminiuming
iron
A.
Am. Dm, E7
Am.7
C. A construction site
C. cogpetbononing
C.D.A, Bm, G. E, Fem.
D.A pesceful beach
D. noto of those
D. Féln, D) Rm, Chin
Anserr C.A construction site
Answer: aluminiuming
Answer: Em, B7, Cs, Am?
Reasoning- - Temporal Event Reasoning
Reasoning- Mult Speaker Role Mapping
Reasoning- - Ensctional Tone Interpretation
Question: For the given madio, ideniity which the following sounds can be heard for the
Question: Idemiiyy role
and
the overall emotional stmosphere created by the combination of
Question:
second spenker in the conversation
longest duration.
Parent and child
instruments in the audio?
A. Video game sound
B. Teacher and student
A. Ordinary and dull
Music
Doctor and pulient
R. Unique and hean--oucching
C. Sound effect
Coach and achlete
C. Chaotic and confusing
D. Background noise
Answer: C, Doctor and patient
Energeric fast-paced
Amwer:: A. Video game sound
Answer: B. Unique and heart-touching
Figure 2: Examples from the MMAU benchmark illustrating the diverse range of reasoning and information extraction tasks across the domains of sound, speech, and music. Each task involves rich, context- specific audio paired with human-annotated QA pairs that require expert-level knowledge and reasoning abilities. The benchmark covers a wide range of challenges, illustrating the breadth and depth of MMAU's evaluation scope.
reasoning benchmarks (y Arcas & Norvig 2023 Bubeck al., 2023
Ge al., 2024 Latif al.,
2023). Recently, Large Multimodal Models (LMMs), which extend LLMs by integrating multiple
modalities beyond text, have demonstrated enhanced general intelligence (Liu et al. 2024a]] 2023bb
Zhang et al. 2023a; Zhu et al. 2024] Ghosh et al., 2024c). These models excel at a broader set of tasks by improving their ability to observe and perceive the world more comprehensively.
Benchmarking has been cornerstone in advancing AI, providing structured challenges that drive the field forward (Raji et al. 202T). However, as highlighted by the AGI taxonomy proposed by (Moriis et al.) 2024), which defines AGI as system that performs at the "90th percentile of skilled adults" across a wide array of tasks, current benchmarks fall short of this standard. Tasks such as image and speech recognition, for instance, do not demand the expertise of skilled humans and can often be performed by young children (Lippmann) 1997 Gerhardstein & Rovee-Collier 2002). In response to this gap, researchers in natural language processing and vision have developed nu- merous benchmarks (Wang, 2018; Hendrycks et al., 2020. Yue et al. 2024: Lu et al. 2023), many of which require extensive world knowledge and complex reasoning solve. These benchmarks have pushed the boundaries of model capabilities, prompting incremental improvements. However, there is notable lack of comprehensive evaluation benchmarks specifically designed assess the perception and reasoning abilities of oudio-language models. Audio perception and reasoning are essential for achieving true AGI, as is one of the primary modalities through which humans inter- pret and engage with the world, capturing complex information about the environment, emotions, intentions, and context You et al., 2024; Gong 2024). Currently, advanced Large Audio-Language Models (LALMs) are primarily evaluated on foundational tasks such as Automatic Speech Recogni- tion (ASR), Acoustic Scene Classification, or Music Genre Classification (Rubenstein et al., 2023] Gong et al. |023c Ghosh et al.] 2024c). While these tasks are fundamental for assessing basic au- dio understanding, they do not require the deliberate and complex reasoning that characterizes more sophisticated forms of intelligence. This highlights significant gap in the evaluation of LALMs, limiting our ability to fully understand and quantify their true potential in advanced audio tasks.
Our Contributions. We present MMAU, the first comprehensive benchmark tailored for multi- modal audio understanding and reasoning. MMAU features over 10,000 expertly annotated audio- question-response pairs evenly distributed across speech, sound, and music domains. With informa- tion extraction and reasoning questions that require models to demonstrate proficiency in 27 distinct skills across unique tasks, MMAU achieves significant breadth. Additionally, it covers depth by in- cluding tasks that require advanced reasoning, such as multi-speaker role mapping, emotional shift detection, and temporal acoustic event analysis. Our audio data is sourced from a wide range of contexts, challenging models to jointly process auditory content and text, recall relevant knowledge, and engage in complex reasoning to solve the tasks. To summarize, our main contributions are:
1.
We introduce MMAU, the first benchmark specifically designed to evaluate advanced au- dio perception and reasoning in LALMs. With 10k expertly annotated instances spanning speech, sounds, and music, MMAU meets the highest standards of evaluation by covering both breadth and depth in multimodal audio understanding.
2

---

Published as a conference paper at ICLR 2025
2. We assess 18 open-source and proprietary models MMAU and demonstrate that even the most advanced LALMs struggle with tasks that humans easily excel at, achieving only 59% accuracy our benchmark, highlighting significant gaps in current model capabilities.
3. We conduct an in-depth analysis of model responses, uncovering key insights such as the effectiveness of audio captions for text-only models, skill-wise performance, and the chal- lenges LALMs face in attending to audio inputs and addressing complex tasks.
2 RELATED WORK
Audio-Language Models. Recent years have seen significant progress in audio understanding. driven by advances in (large) language models that enhance cross-modal interactions between au- dio and language. Early research focused on developing cross-modal audio-language encoders (ALE) that learn shared representations between the two modalities. Notable models include Audio- CLIP (Guzhov et al. 2022). CLAP (Elizalde et al. [2023), and CompA (Ghosh et al. 2023). CompA makes first attempt address reasoning in audio- language encoders by improving compositional reasoning through a novel training paradigm. More recently, efforts have shifted toward integrat- ing audio understanding with LLMs, leading the emergence of Large Audio-Language Models (LALMs). These include models such as Pengi (Ge et al. 2024), Qwen-Audio (Chu et al.) 2023), LTU (Gong et al. 2023c), and GAMA (Ghosh et al.) (024c). Leveraging the advanced reasoning capabilities of LLMs, LALMs can respond to complex queries involving audio inputs. For instance, models can accurately interpret intricate questions about acoustic scenes. However, unlike humans who can perceive and reason across diverse audio types, LALMs have largely evolved in isolation, with specialized models focusing separately on sounds (e.g.. Pengi. LTU, GAMA, etc.), speech (e.g.. SALM (Chen et al., 2024), AudioPalm (Rubenstein ct al. 2023), etc.), or music (LLark (Gardner et al. 2023), MERT (Li et al. 2023), and others (Liu
GAMA demonstrates that instruction-tune
ct al. 2024b; Doh et al., 2023 Won et al.] 2024 ). Few models are capable of comprehensively
understanding all three (e.g.. Qwen-Audio (Chu et al. 2024), Audio Flamingo (Kong et al., (024)).
Audio Benchmarks. With the rapid rise of multimodal LLMs, there has been significant surge in the development of comprehensive benchmarks for text and vision modalities to assess expert- level domain knowledge and advanced reasoning capabilities, including subject knowledge (Clark
ct al. 2018: Hendrycks et al. 2020), safety (Zhang et al. [023b] |eeth et al.) 2023). multilingual
proficiency [Ahuja al. 2023). multidisciplinary understanding Yuue al. [2024]| Hu al.
2024).
perception tests (Yuan et al. (023). mathematical reasoning (Li et al., 2024; Zhang et al. 2024
and video understanding (Li al. 2023) Ning et al., 2023 Fu al.]
2024ap. However. despite
this progress, there is still a notable lack of similarly complex benchmarks for the audio modality.
To address this gap, a few attempts have been made to build ladio-language benchmarks for speech (e.g.. OpenASQA (Gong et al. (2023)), sound (e.g., CompA (Ghosh et al. (003)), CompA-- (Ghosh)
cct al. (2024c)), music (e.g., MusicBench (Melechovsky et al. 2023), MuChin (Wang et al.) 2024b),
MuChoMusic (Weck et al.] )024)), and their combinations (e.g.. AIR-Bench Yang al. 2024))
AudioBench Wang et al. (2024a)). These benchmarks, however, either focus on limited reasoning tasks like compositional
temporal reasoning Ghosh et al. (2023) rely fundamental audio tasks like ASR and acoustic scene classification Yang et al. (2024). To the best of our knowledge, no existing benchmark fully addresses the breadth and depth of reasoning required to evaluate advanced audio understanding, leaving a critical gap in the assessment of LALLSs capabilities.
3
THE MMAU BENCHMARK
3.1 OVERVIEW OF MMAU
We introduce the Massive Multi-Task Audio Understanding and Reasoning Benchmark (MMAU), a novel benchmark designed to evaluate the expert-level multimodal reasoning and knowledge- retrieval capabilities of large audio-language models (LALMss. MMAU comprises carefully cu- rated audio clips paired with human-annotated natural language questions and answers meticulously crafted by domain experts. Spanning all three major audio domains speech, sounds, and mu- sic MM-MMMUU includes 27 distinct tasks, consisting of 16 reasoning and 11 information extraction tasks. MMAU is uniquely designed to test LALMs' advanced cognitive abilities, challenging mod- els with questions that require complex, deliberate reasoning and knowledge retrieval grounded in
3

---

Published as a conference paper at ICLR 2025
Teraore
Oinectr
setiaal
war and ieard
Fauggn
Sound 22%
Sperch
teadee Malio
Sound 42%
Music 49%
Iripp reation
Nill Sprdke
Acaatic Source
Mersel Noture
Heppetutie
Lecck
lorpa
Eve
famtring
Figure 3: (Left)) Distribution of skills required for information extraction questions in the MMAU benchmark across the domains of sound, speech, and music. (Right) Distribution of skills required for reasoning questions the MMAU benchmark across the same domains. Each question MMAU demands the model apply one or more of these skills to generate a reliable and accurate response. Appendix provides example questions demanding these skills and the specific tasks associated with them. This chart underscores the diverse cognitive abilities necessary for success in the benchmark, mirroring the complexity and expert-level reasoning involved.
audio perception. To our knowledge, MMAU stands as the first comprehensive benchmark to rigor- ously assess these capabilities, filling critical gap in the evaluation of LALMs.
Table provides an overview of MMAU, which consists of 10,000 multiple-choice questions (MCQs) di- vided into a test-mini set and main test set. The test-mini set, comprising 1,000 questions, reflects the task dis- tribution of the main test set and is in- tended for hyperparameter tuning. The multiple-choice format was selected to standardize evaluation and align with widely accepted practices in LLM eval- uation (Hendrycks et al.) 2020 Yue et al.] 2024). Just humans often struggle with closely related options in multiple-choice questions, we antici- pate that models may face similar difficulties. Each question in MMAU is manually categorized by domain experts into easy, medium, or hard difficulty levels. MMAU assesses models across 27 distinct skills, with questions focused either information extraction (3,936 questions) or reason- ing (6,064 questions). The detailed breakdown of skills for both question types shown in Fig. 3 For this benchmark, the skills required for information extraction and reasoning are kept disjoint, meaning skill used for an information extraction question will not be required for a reasoning question, although individual questions may require multiple skills from each respective category. MMAU is specifically designed to evaluate advanced audio comprehension, information retrieval (with without external knowledge), and complex reasoning, pushing models to not only perceive and understand multimodal information but also apply subject-specific knowledge and sophisticated reasoning to solve problems accurately.
Statistics
Number
Total Questions
10,000 3
Audio Domains
Domain Categories Difficulties (Easy: Medium: Hard)
10:10:7
22%::56%:229%
Splits (test-mini: test)
1000;9000
Information Extraction Based Questions Reasoning Based Questions
3499 (34.99%)
6501 (65.74%)
Average question length Average option length
9.28 words
5.23 words
Average audio length
10.14 sec
Table 1: Core statistics of the MMAU benchmark
3.2 DATA CURATION AND ANNOTATION
We follow rigorous 7-step pipeline for curating MMAU, described below:
1. Source Selection: We began by collecting diverse audio corpora, including speech, music, and environmental sounds, prioritizing real recordings over synthetic data. To ensure unbiased and ro- bust evaluation, we sourced audios exclusively from test sets or evaluation sets when test sets were unavailable. Preliminary checks were conducted ensure data quality and relevance before expert
4

---

Published as a conference paper at ICLR 2025
????
HMAM
Source
Task Curation
Expert Annotation & Filtering
Option
Expert Review
MMAU
Selection
Augmentation
Figure MMAU Benchmark Construction Pipeline.
refinement. Sound data was selected from AudioSet Strong, focusing on clips with 2-5 distinct acoustic events, each lasting at least two seconds, ensuring clear and distinguishable samples for reasoning tasks. For music, labeled test audio files were used to generate highly relevant questions. Speech data underwent additional checks for transcription clarity, adequate length, and accurate ground truth labels to facilitate meaningful question and answer generation. These steps (more de- tails in D.5) were critical, and we gathered 13 audio corpora to ensure strong foundation for task development (more details in Appendix
2. Task Curation:
Leveraging insights from these corpora, we consulted domain experts to select tasks that would challenge models with expert-level reasoning while maintaining real-world rele- vance. For this step, we also considered the possibility of generating synthetic audios. We curated tasks based on their potential to assess advanced reasoning and knowledge retrieval, carefully filter- ing an initial set of 90 tasks down 27, ensuring alignment with real-world applications and the constraints of current generative audio models.
3. Expert Annotation:
Domain experts, with help from the authors, crafted high-quality questions and answers for each audio clip. The authors helped curate the set of plausible audios for the experts (based on the final set of tasks selected) and went through multiple iterations. Questions were generated to ensure that each question required real-world complex reasoning and domain-spectie knowledge for a faithful question. Experts were asked to follow a set of pre-defined guidelines for QA generation, detailed in Appendix D.3
Expert Filtering: A separate team experts rigorously reviewed the QA pairs, removing am- biguous, overly complex instances, including instances with low-quality audio samples, to maintain high accuracy and relevance. This approach further enforces bias control the annotation pipeline by requiring all experts to adhere to standardized set of filtering guidelines.
5. Option Augmentation: We utilized GPT-4 (OpenAl et al., 2024) to augment each question with additional answer options, systematically increasing task complexity and further testing the models' reasoning skills. Options were not augmented randomly but generated based on the context of the audio and the question. The augmentation prompt is detailed in Fig. TO] After the option augmentation process, each annotator independently scores questions generated by other experts on 1-to-5 scale. Low scores are assigned questions with misleading or overly correlated options, as well as those with incorrect answers. This scoring ensures the filtering of subpar samples and contributes to the reliability of the dataset.
6. Expert Review: Final reviews by experts and authors included tagging every instance with the task that needs be completed and the specific skills required to complete that task.
7. MMAU Finalization: From the fully annotated and reviewed QA pairs, we selected 10,000 in- stances to create the final benchmark. This selection was made to ensure balanced representation of all 27 task types and equal coverage of speech, sound, and music. For evaluation, 1,000 instances were chosen to form the test-mini set, evenly distributed across all tasks, while the remaining stances were allocated to the main test set.
Details on the background of our expert annotators, generation model and annotation portal are provided in Appendix D
3.3 COMPARISON WITH OTHER BENCHIARKS
To highlight the distinction between current benchmarks and MMAU, we break down the informa- tion processing steps a Large Audio-Language Model (LALM):
5

---

Published as a conference paper at ICLR 2025
Duuminn
Tasks
Benchmark
Spaech
Sound
Music
Info Extraction
Reasaning
Expert Comments
Dafiienity
CompA
0.5k
Requires aqund event sequenee undentandiing, Limited
reasosing daych and knowledge scoge.
CompA-R
1.3
1.5k
Resrized to sounds and only contromull evere indersnniing
Limited in knowledge scope.
3.0
MuChis
1k
Roaricted mase with miniral manoning depth iniiedin
knowlodge scope.
2.5
miniral nasoning depth. Limitealin
Inowledge scope.
MoCheMuse
1.3k
074
0.4k
Reathiied mnviic with openetaaal snowets Limite knowledge scope.
OpeARQA
83k
888
Requies linited accuddic sccrc indaddaninng DaS rul
require esternal expert knowledge
N.0
AndieBennh
1034+
Basc samaiiic informatin retrical with ninimal ressinngg depth
complesio.
reqire
knowlelge.
3.5
AIR-Berch
1.2
Basc secustic informatio retrical with mininal
depti
comppesity aot require cocnmll expert knowledge
MMAU (ourg)
10K
4.5k
5.2k
Requies line anaieed udiio unde dinz with espert-lveel. multi step reasoning and specialiaed know across a broad range or toplcs.
4.5
Table 2: Comparison of MMAU with existing audio understanding and reasoning benchmarks across various statistics. MMAU covers all three domains-speech, sound, and music-while having the highest number of information extraction and complex reasoning tasks.
Audio Understanding
Knowledge Extraction (optional) -> Reasoning (optional)
Perception
Most existing benchmarks focus solely on audio understanding, assessing models on basic audio processing tasks like ASR, Speech Emotion Recognition, and other foundational tasks. These tasks primarily evaluate whether the model can comprehend the audio content, such as spoken words, emotional tones, or distinct sound events, but they do not challenge the model's broader cognitive abilities. We argue that this approach falls short in evaluating the true capabilities of LALMs, as simply mastering foundational tasks is insufficient for the next generation of AI agents that must go beyond basic understanding. MMAU targets this gap by moving beyond mere audio understand- ing to include tasks that require knowledge extraction and complex reasoning. This progression demands that models not only perceive the audio with respect the text prompt but also apply advanced cognitive skills to respond faithfully.
Table 2 provides a comparative analysis of MMAU with recent audio reasoning benchmarks. Un- like existing benchmarks, MMAU encompasses all three major audio domains-speeche music, and sounds-and and offers the largest set of tasks that integrate both knowledge extraction and reasoning. As illustrated in Fig. 3 MMAU sets itself apart with well-crafted reasoning tasks that are absent in other benchmarks (see Appendix [ for further comparisons). Notably, MMAU is the first bench- mark to incorporate snowledge-basee information extraction questions, pushing the boundaries of what LALMS can achieve.
To further illustrate the differences between MMAU and other benchmarks, we compare the diffi- culty levels based on expert ratings (1-5) across 500 randomly selected instances from each bench- mark (more details on this in Appendix [ Experts evaluated the benchmarks along two dimensions: Breadth (diversity of tasks and domains) and Depth (task complexity). In terms of breadth, previous benchmarks are often limited to specific domains task types. For instance, MusicBench (Mele- chovsky et al., 2023) and MuChin (Wang et al. 2024b) focus solely on basic music information retrieval tasks. When it comes to depth, many benchmarks emphasize specialized reasoning, such as temporal reasoning (Ghosh et al., 2023 2024c) or content-based reasoning (Gong et al., (003bb, but do not comprehensively evaluate LALMS ability to handle more complex tasks like contextual and causal reasoning. While benchmarks like AIR-Bench (Yang et al., 2024) and AudioBench (Wang 2024a) span multiple domains- sppechh, music, and sound they predominantly focus on foundational tasks and fail to fully capture the intricate reasoning capabilities of LALMs.
4 EXPERIMENTAL SETUP
LALMs. We compare a range of open-source, open-access, and closed-source Large Audio- Language Models (LALMs), including (i) Owhno-AAuio-Chhat (Chu et al. 2024): A open-access

---

Published as a conference paper at ICLR 2025
model (only checkpoints are available; training data and details is unknown) with strong capabilities in sound, speech, and music understanding and reasoning.
Lona--Audo--Inttruc is fine-tuned
version with chat abilities based Qwen2-7B its LLM. (ii) GAMA (Ghosh al. 2024c): A
leading fully open-source model focused on sound and music understanding, built on [. L--a----BBB
(iii) GAMA-ITT is its fine-tuned variant for complex reasoning. (iv) SALAMONN [aan al. (202))
One of the first open-source LALMs, excelling in speech and sound understanding, (v) LTU (Gong
al. 2023ch: A fully open- model with strong audio understanding and reasoning abili- ties. (vi) LTU-AS (Gong et al. 2023b) is an advanced version capable of joint speech and audio comprehension. Both models use Vicuna-7B as the base LLM. (vii) aadio-Flamingo-Ce -hat (Kong et al. 2024): A fine-tuned version of Audio-Flamingo with chat and instructioon
abilities. Unlike other models, it employs cross-attention and uses OPT-IML-MAX-I.33 as its base L.M. (viii) MusiLingo (Deng et al.] 2023): A music captioning and reasoning model that combines a
MERT encoder (Li ct al., 2023) with Vicuna 7B LLM. MusiLingo is fine-tuned on MusicInstruct (ix) M2UGen (Hussain al. )023): A framework capable of completing music understanding and multi-modal music generation tasks (x) MuLLaMa (Liu et al. 2024b): A Music Understanding Language Model designed with the purpose of answering questions based music. This model is based on MERT (Li et al. 2023) and Llama (Touvron et al. 2023b) (xi) Gemini-Flash and Gemini- Pro (Team al. 2024): Two proprietary LALMs known for advanced capabilities in speech, music, and sound reasoning. Gemini models are also some of the strongest multimodal systems overall, excelling in both vision and language tasks, though specific architectural details remain unknown. We do not evaluate non-instruct/noon chat versions of Qwen-2, Audio Flamingo, and Pengi as they cannot follow instructions and fail to respond by selecting options.
LLMs. To assess the robustness of MMAU, we also perform text-only evaluation using various open and closed- source Large Language Models (LLMs), including GPT-4o (OpenAl al.) (024)) a closed-source, state-of-the-ard LLM, and Llama 8B Instruct (Dubey et al. 2024), open-source, instruction-tunes model. Additionally, to evaluate whether incorporating external audio information can enhance LLM performance on MMAU, we provide these models with audio captions generated by Qwen2-Audio (Chu et al. 2024).
Evaluation Strategy. We use micro-averaged accuracy our evaluation metric. Specifically, we present the question along with the list choices to the models, instructing them to select the correct choice. Since most current LALMs are instruction-tumed for generating open-ended sponses (Ge et al.] 2024] Goog et al.] 2023b), we employ robust regular expressions and develop response-processie
workflows extract key information from the model outputs, which is then matched to one of the provided options using string matching. We discuss more about our string matching- based evaluation algorithm section [ tion selection due to ordering, we randomize the order of the options five times and select the option chosen most frequently. Additionally, we experiment with various prompt sets across all LALMS and report the best results.
To mitigate any potential bias in the model's op-
RESULTS AND DISCUSSION
5.1 MAIN RESULTS
Table 3 compares the results calculated using our evaluation algorithm on the raw output of various LALMs on the MMAU benchmark. Our key findings are:
1. MMAU poses significant challenge. The MMAU benchmark is highly demanding for current models, both open-source and proprietary. The top-performing LALM achieves only 59% accuracy, while the best-cascaded captioning + LLM approach reaches just 58%. In comparison, human performance achieves 82%.
2. Minimal gap between open-source and proprictary models. Unlike other domains, we observe only small performance gap between the best open-source and proprietary LALMs. As shown in Table Qwen2, the leading apen-sccess model, performs almost on par with the proprietary Gemini Pro, with just 0.47% difference in average performance. However, the top fully open-source model, GAMA, trails significantly behind, with larger performance gap of 28% compared to Gemini 2.0 Flash. This suggests that larger datasets
7

---

Published as a conference paper at ICLR 2025
Music
Speech
Avg
Models
Size
(So, Ma, Sp}
Test-mini
Test
Test-mini
Test
Test-mini
Test
Test-mini
Test
Random Guess
26.72
25.73
24.55
26.53
26.72
25.50
26.00
25.92
Most Frequent Choice
27.02
25.73
20.35
23.73
29.12
30.33
26.50
Human (test-mini)
86.31
78.22
82.17
82.23
Large Andio Language Models (LALMs)
323M
02.90
03.05
01.20
01.50
03.40
Audio Flamingo Chat
2.28
23.42
28.26
15.26
18.20
11.41
10. 10.16
16.69
18.87
I.TU
7R
22.52
25.86
09.69
12.83
17.71
16.37
16.89
18.51
LTU AS
7B
23.35
24.96
9.10
10:46
20.60
21.30
17.68
18.90
MusiLingo
7B
23.12
27.76
03.96
06.00
05.88
06.42
10.98
13.39
MuLLaMaa
7B
X
40.84
44.80
32,63
30.63
22.22
16.56
31.90
30.66
MZUGen
7B
03.60
03.69
32.93
30.40
06.36
04.53
14.28
12.87
GAMA
7B
41.44
45,40
32.33
30.83
18.91
19.21
30.90
31.81
GAMA-IT
7B
43.24
43.23
28.44
28.00 28.00
18.91
15.84
30.20
29.02
Owen
Owen-Audio Audio Chat
Owen2-Audh
8.4B
07.50
08.20
05.
06.16
03.10
04.24
05.24
06.20
Qwen2-Andio-Instaee
8.48
50.98
$3.26
42.04
45.90
49.20
52.50
SALAMONN
13B
41.00
40.30
34,80
33.76
25.50
24.24
33.70
32.77
Gomini Pro vi.
49,40
48.
58.55
$5.90
54.90
$2.97
Gemini 2.0 Flash
56.46
61.73
58.68
56.53
51.65
61.53
55.60
59.93
Large Language Models (LLMs)
GPT4o weak cap.
39.33
35.80
39.52
41.9
58.25
68.27
45.70
48.65
GPT40 strong cup.
57.35
55.83
49.70
51.73
64.86
68.66
57.30
58.74
Llama-3-Ias. + weak cap.
8B
34.23
33.73
38.02
42.36
54.05
61.54
42.10
45,87
Llama-3-Ios. strong cap.
8B
50.75
49.10
50.29
48.93
55.25
62,70
52,10
53,57
Table 3: Performance comparison of various LALMs and LLMs on the test subset of MMAU across sound, speech, and music domains. Human evaluation results shown for the MMAU test-mini split. We also mark the training data used models in each category are highlighted in bold, and the second- bestb scores are underlined. Note: These results are from the initial version of the benchmark version (and are calculated on raw LALM outputs). As of May 15, 2025, we're 'e updated the benchmark based on community feedback. For results the latest version, please refer the project website. We also report additional scores parsed version of the outputs, where we use GPT-40 to extract the selected option before evaluation.
these models includes either speech, sound, or music. The best-performing
and additional training resources likely contribute to enhanced audio perception and rea- soning performance on MMAU.
3. Generalized vs. Specialized Models. Generalized models trained across multiple do- speech, sounds, and music-such as Qwen2-Audio, LTU-AS, and Gemini, exhibit strong overall performance. This indicates that larger, more diverse training data leads to a more comprehensive understanding of audio. On the other hand, models fine-tuned for specific domains consistently outperform generalized models in their respective areas. For instance, M2UGen, designed for music understanding, surpasses general-purpos models like LTU and GAMA by up to 15% music-related benchmarks. This underscores the value of specialization in achieving higher task-specific accuracy.
4. Model size drives performance. Larger LLMs demonstrate superior reasoning capabili- ties and knowledge retention, resulting in significantly better performance on MMAU. For example, SALMONN, with billion more parameters than L.TU, achieves an average per- formance improvement of 14%, scale tackling complex audio-language reasoning tasks.
in Table This highlights the critical role of model
5. Models perform best on sound and worst speech. With average scores of 18% 30%, 23% across speech, sound, and music, models perform best on sound-related tasks and struggle the most with music. This suggests that while spoken language understanding has advanced, reasoning over spoken language-especially perception beyond mere con- temt-remains a challenge. On the other hand LALMS have mastered music reasoning, skill generally not possed non-experts.
6. Cascaded approaches outperform others. Captioning audios first and then prompting LLMs yields the best results. Enhancing the quality of the captions further improves over- all performance. This demonstrates the potential of scaling audio-language understanding through separate advancements in audio and language reasoning.
8

---

Published as a conference paper at ICLR 2025
7. ALEs perform well but have notable limitations. Despite their encoder-only architec- ture, ALEs demonstrate strong performance in our tailored evaluation setup, aligning with findings in Deshmukh al. ) where ALEs outperform LALMs in deductive reason- ing tasks. However, their success stems from their bag-of-words approach, excelling in tasks emphasizing lexical matching. Due to the distinet evaluation strategy used for ALEs compared to LALMs, we provide a detailed discussion of their performance in the App
5.2 ARE LALMS REALLY LISTENING?
Andio
Noise
Figure 5 compares the performance of vari- ous models on the MMAU test set, where the original audio input is replaced with random Gaussian noise. This experiment helps assess whether models truly attending the audio inputs or just responding using language priors. As shown, the performance of MuLLaMa and SALMONN remains largely unaffected, indi- cating that these models may not always rely on the audio input to generate responses. In con- trast, models like GAMA, Qwen2-Instruc and Gemini Pro vi. exhibit significant drop in per- formance, suggesting that they are more atten- tive to the audio content. We provide examples of incorrect outputs in Appendix K
40 300
Avcuracy
Qwene-Testract in
Figure 5: Performance comparison on the MMAU test with Gaussian noise replacing the original audio. Mod- els like MuLLaMa and SALMONN show little change performance, indicating limited reliance audio in- put, while others show a significant drop, suggesting greater audio dependence.
5.3 CAN CAPTIONS BRIDGE THE GAP FOR TEXT-ONLY MODELS?
Figure compares the performance of various models on the MMAU test set, where the origi- nal audio input is replaced with captions. We present results using two types of captions: weak captions (generated using EnCLAP (Kim et al. 2024) for sounds, MuLLaMa for music, and Whis- per buse (Radford et al. 2023) for speech transcripts) and strong, detailed captions (generated using Swen2-Audio-Insse Instruct with prompts detailed in Appendix N. As the results show, strong captions can indeed help bridge the audio understanding gap for text-only models, with GPT4o achieving the highest accuracy 59%. Additionally, icantly boosts the performance of text-only LLMS (i.e., when captions effectively capture acoustic details, text-only LLMs can reliably answer questions.) These findings are consistent with Ghosh et al. (0024)), who show that visual descriptions improve LVLM performance for reasoning prompts.
demonstrate that enhancing the quality of captions signif-
5.4 DEEP DIVE: SKILL-SPECIFE MODEL PERFORMANCE
The average scores for Gemini 2.0 Flash across easy, medium, and hard questions are 51.32, 66.16, and 50.91, respectively (detailed results for other models in Table While suggests that models perform consistently across diffi- culty levels, we wanted to dive deeper into skill- specific performance. Figure 6 illustrates the accuracy distribution across easy, medium, and hard questions for eight skills with the highest number of questions in the benchmark. Sur- prisingly, the reason for the uniformity across difficulty levels is that models excel in cer- tain skills across all difficulties (e.g., Phonemic Stress Pattern Analysis), but consistently strug- gle with others (e.g., Temporal Reasoning), re- gardless of the question's difficulty. This ob- servation highlights that rather than focusing on improving model performance in single skill,
Temporal Reasoning
Harmony and Chord Progressions
Instrumentation
Genre and Stylic Reusoning
Phonemic Stress Pattern Analysis
Rhythm and Tempo Understauding
kmotional Tone Interpretation
Historical and Cultural Reasoning
-Q- Easy
-D- -Q- Medium
- Hard
Figure 6: Accuracy distribution for Gemini 2.0 Flash acToss easy, medium, and hard questions, categorized by skill type. The graph highlights how LALMs excel some skills across all difficulty levels (e.g.. Phonemic Stress Pattern Analysis) but struggle with others (c.g.. Temporal Reasoning) regardless of difficulty.

---

Published as a conference paper at ICLR 2025
Other Eroos 0.4%
Annotation Erros 4%
Annotation Biros
6%
Reasoning Frors
Reasoning Errors
Extmetion Frrors 5%
Extraction Errors 17%
Pereeytual Froors 55%
Perceptual Errors 56%
Knowledge Erors
Know Emors
5%
Figure 7: Distribution of human-annotated error types 500 instances for Qwen2-Audio-Instrue (L.eft)
and Gemini Flash (Right). Appendix provides detailed definitions each error type.
future work should focus on developing broader range of competencies, ensuring they can handle complex reasoning across various tasks.
5.5 PINPOINTING LALM WEAKNESSES: WHERE ARE THEY FALLING SHORT?
and Gemini 2.0 Flash across 500 instances. The dominant error category for both models is Perceptual Errors, with showing 55% and Gemini 2.0 Flash at 56%. This indicates that both mod- els struggle primarily with understanding and accurately perceiving the audio inputs. Reasoning Errors and Answer Extraction Errors (Errors where model outputs and ground-truth answers the same but the model provided an ill-formatted response) account for significant portion mis- suggesting
Figure provides breakdown of the error types made by Qwen2-Audio-Ine
Owen2-Audio-Instuuee
takes, particularly for Qwen2-Audio-Instruec where 18% of errors are reasoning- that even when models correctly perceive the audio, they often fail to apply the necessary complex reasoning. For Gemini 2.0 Flash, reasoning errors account for 13%. while answer extraction errors are lower compared to Dowe2-AAuio-Insttuee Errors form smaller percentages in Qwen2. Overall, our error analysis highlights that improving perceptual understanding is crucial for better performance. This can be done through more training data (Liu et al. 2023a), better architectures (Ghosh et al. 2024c) or other methods (Fu et al. 2024b).
Interestingly, Knowledge Errors and Annotation
CONCLUSION, LIMITATIONS AND FUTURE WORK
In this paper, introduce MMAU, novel large-scale benchmark designed to comprehensively evaluate multimodal audio understanding and reasoning in AI models. MMAU challenges mod- els with diverse set of tasks that assess 27 distinct skills, emphasizing advanced perception and domain-specific reasoning. The benchmark presents tasks akin to those faced by experts, making a rigorous test for Al systems. Our evaluations of 18 open-source and proprietary LALMs reveal that even the overall best model achieves only 59% accuracy on MMAU, highlighting the significant challenges poses. We provide analysis of the unique hurdles presented by this benchmark.
As part of future work, we aim to address in future iterations of MMAU: (i) Currently, we treat skills required for information extraction and reasoning as disjoint sets. As part future work, we plan to incorporate tasks that require skills from both types. (ii) There is a risk of biases introduced during the human or ILLM-driven annotation process. We aim further refine the dataset to minimize any potential biases. (iii) MMAU focuses on multiple-choice tasks and does not evaluate open- ended generation, which allows models to reason more freely and exhibit errors such as language hallucinations. Including open-ended tasks will help better understand these kinds of errors. (iv) MMAU currently targets audio inputs up to 40 seconds, constrained by the input limitations of existing audio encoders. We aim to include tasks involving longer audio inputs to better evaluate models' capabilities handling extended audio contexts. (v) Lastly, we plan to broaden the range of tasks and skills covered by MMAU enhance its challenge and relevance future models.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
The benchmark will be publicly released upon paper acceptance. The test-mini subset will be com- pletely open-sourced on GitHub, together with ground-truth responses and all meta-data. The actual larger test set will be hosted eval.ai and GitHub, and only audios and questions and audios will be available. Researchers will be able to upload their predictions to evaluate their models. Our benchmark will be released with CC BY 4.0 license, and we only used existing audio datasets that allow redistribution.
8 ACKNOWLEDGMENT
This project is supported in part by NSF#1910940.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. urXiv preprint arXiv:2303.0877 087744 2023.
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301. 13355, 2023.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Ganu, Sameer Segal. Maxamed Axmed, et Mega: Multilingual evaluation of generative ai. arXiv preprint 2.XXi:22303.1222 2023.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for automatic music tagging. ICML, 2019.
Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12711 2023.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihal- cea, and Soujanya Poria. Towards multimodal sarcasm detection (an .Obviously.. perfect pa- per). In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4619-4629, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18653/11/P19-1 1455. URL h http://aaaaattooooooor///ee
1455
Zhehuai Chen, He Huang. Andrei Andrusenko, Oleksii Hrinchuk, Krishna C Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. Salm: Speech-augmented lan- guage model with in-context learning for speech recognition and translation. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13521-13525. IEEE, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 2023.

---

Published as a conference paper at ICLR 2025
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2- technical report. arXiv preprint arXiv:2407.10759 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: arXiv: 1803.05457 2018.
Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. Musilingo: Bridging music and text with pre-trained language models for music captioning and query response. arXiv preprint arXiv:2309. 08730, 2023.
Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. Audio entailment: Assessing deductive reasoning for audio understanding, arXiv preprint arxiv:2407. 18062, 2024.
SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.16372, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407. 14358, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal lims in video analysis. arXiv preprint arXiv:2405.2 2024a.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see not perceive. arXiv preprint 2404.22400.. 2024b.
Josh Gardner, Simon Durand, Daniel Stoller, and Rachel M Bittner. Llark: A multimodal foundation model for music. arXiv preprint arXiv:2310.07160, 2023.
Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When IIm meets domain experts. Advances in Neural Information Processing Systems, 36, 2024.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Peter Gerhardstein and Carolyn Rovee-Collier. The development visual search in infants and very young children. Journal of Experimental Child Psychology, 91():194--211 2002.
Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S Ramaneswaran, Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Compa: Addressing the gap in compositional reasoning in audio-language models. arXiv preprint arXiv:2310.08753, 2023.
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, Oriol Nieto, Zeyu Jin, and Dinesh Manocha. Vdgd: Mitigating Iylm hallucinations in cognitive prompts by bridging the visual perception gap. arXiv preprint 2.5i::2400.56688 2024a.
Sreyan Ghosh. Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Reclap: Improving zero shot audio classification by describing sounds. arXiv preprint ar arXiv:2409.0er 13, 2024b.
12

---

Published as a conference paper at ICLR 2025
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sak- shi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406. 11768, 2024c.
Yuan Gong. From audio perception to understanding: A path towards audio agi. In Al Seminar. Stony Brook University, 2024. URL httt::://a..stoonbbr.. edu/
Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU), pp. 1-8, 2023a. doi: N0.1199/ARRUST964.2222. 10389974..
Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU). pp. 1-8. IEEE, 2023b.
Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass, Listen, think, and understand. arXiv preprint arXiv:2305.100 2023c.
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-202 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370. IEEE, 2021.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20110-222111 221882221 2024.
Atin Sakkeer Hussain, Shansong Liu, Chenshuo Sun, and Ying Shan. M {22 ugen: Multi-modal music understanding and generation with the power of large language models. arXiv preprint arXiv:2311.11255 2023.
Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6735- 6739. IEEE, 2024.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with sew-shot learning and dialogue abilities. arXiv preprint arXiv:2402.0m 011801011,,,, 2024.
Ehsan Latif, Gengchen Mai, Matthew Nyaaba, Xuansheng Wu, Ninghao Liu, Guoyu Lu, Sheng Li, Tianming Liu, and Xiaoming Zhai. Artificial general intelligence (agi) for education. arXiv preprint arX4v:2304. 12479, 1, 2023.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.002. 2024.
Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, et al. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv preprint arXiv:2306.00100 2023.
13

---

Published as a conference paper at ICLR 2025
Richard P. Lippmann. Speech recognition by machines and humans. Speech Communii-
cation, 22(1):1-15, 1997. ISSN 0167-6393. doi: https://oo.oo////.00////06666669
00021-6. URL http://wwww.aaaooooeooooooo//oo/
S01 50166733397000211
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024a. URL https://
Llava-vi. ww..ithut... ht/pblcg///22-000000000
30- -lava-next
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding Ilama: Advancing text-to-music generation with question answering and captioning. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 286-290. IEHE, 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi. Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:23 2023.
Dan Lyth and Simon King. Natural language guidance of high-fidelity eext-to-specch with synthetic annotations, 2024.
Iariia Manco, Benno Week, Seungheon Doh, Minz Won, Yixiao Zhang. Dmitry Bodganov, Yusong
Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, et al. The song describer dataset: corpus of audio captions for music-and-language evaluation. arXiv preprint arXiv:2311.10057, 2023.
Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria. Mustango: Toward controllable text-to-music generation. arXiv preprint arXiv:2311.08355 2023.
Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Alek- sandra Faust, Clement Farabet, and Shane Legg. Position: Levels of AGI for operationalizing progress on the path to AGI. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st In- lernational Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 36308-3633221111 PMLR, 21-27 Jul 2024. URL https:://ppette httpss:/pproceeiiggg... mmr.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identifi- cation dataset. arXiv preprint urXiv:/706.000 2017.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprini urXiv:2311. 2023.
OpenAl, Josh Achiam, and Others. Gpt-4 technical report, 2024. URL aittps://arxitt /rg/
abs/2303. 08774
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv: 1800.02500 2018.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28518. PMLR, 2023.
Zafar Rafii, Antoine Liutkus, Fabian- Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The musdble 18 corpus for music separation. 2017.
14

---

Published as a conference paper at ICLR 2025
Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.1366, 2021.
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. /2925, 2023.
Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6820-6822 2023.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu. Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint ar wxXx:2310.132205 2023.
Gemini Team, Petko Georgiev, and Others. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL
05530
Touvron, T Lavril, G Izacard, Martinet, MA Lachaux, T Lacroix, B Rozière, N Goyal, E Ham- bro, F Azhar, al. Open and efficient foundation language models. Preprint at arXiv. https://doi. org/10. 48550/arrr. 2302, 2023a.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models, 2023b. URL httph:tt/////ct /ttp///wwww..gg///////2//.20 /ttg//abs//302.3371 13971
Harsh Trivedi, Heeyoung Kwon, Tushar Khot, Ashish Sabharwal, and Niranjan Balasubra- manian. Repurposing entailment for multi-hop question answering tasks. arXiv preprint 2019.
Alex Wang. Glue: multi-task benchmark and analysis platform for natural language understand- ing. arXiv preprint 1rXiv:1804.0740 2018.
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy Chen. Audiobench: A universal benchmark for audio large language models, 2024a. URL
16020
Zihao Wang, Shuyu Li, Tao Zhang, Qi Wang, Pengfei Yu, Jinyang Luo, Yan Liu, Ming Xi, and Kejun Zhang. Muchin: chinese colloquial description benchmark for evaluating language models in the field of music. arXiv preprint urXiv:2402.09871 2024b.
Benno Weck, Ilrria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, and Dmitry Bog- danov. Muchomusic: Evaluating music understanding in multimodal ludio-language models. arXiv preprint 2rXiv:2408.01332 2024.
Minz Won, Yun-Ning Hung, and Due Le. A foundation model for music informatics. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1226-1230 IEEE, 2024.
Yusong Wu*, Ke Chen*, Tianyu Zhang* Yuchen Hui Taylor Borg-Kirkpatrick and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP. 2023.
Qingyang Xi, Rachel M Bittner, Johan Pauwels, Xuzhou Ye, and Juan Pablo Bello. Guitarset: A dataset for guitar transcription. In ISMIR, pp. 453-460, 2018.
Blaise Agüera y Arcas and Peter Norvig. Artificial general intelligence is already here. Noema, October, 2023.
15

---

Published as a conference paper at ICLR 2025
Qian Yang. Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuan- jun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. Air-bench: Benchmarking large audio- language models via generative comprehension, 2024. URL https://azxv..xgg/
2402.0772.
Jiaxuan You, Ge Liu, Yunzhu Li, Song Han, and Dawn Song. How far are we from agi. In ICLR 2024 Workshops, 2024.
Ruibin Yuan, Yinghao Ma, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, et al. Marble: Music audio representation benchmark for universal evaluation. Advances in Neural Information Processing Systems, 36:39626 09647,,977,, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 9556-9567, 2024.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuna audio-visual language model for video understanding. arXiv preprint arXiv:2306.02888, 2023a. URL https://
mexiv.org/l abs 2306. 02858
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams visual math problems? arXiv preprint arXiv:2403. 2024.
Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309 07045, 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth Interna- tional Conference Learning Representations, 2024. URL https: / http///peeeeeeeeeeeeeee/e net/ Forum?id=1t2baBBE2.
A APPENDIX
Table of Contents:
1. B Additional Results
2. Synthetic Data Generation
3. D Annotation Details
4. E Model Details
5.
Dataset Details
6.
Annotation Tool
7.
Comparison
8.
Evaluation Algorithm
9.
Question Categories
10. K Failure Cases
B ADDITIONAL RESULTS
B. AUDIO-LANGUAGE ENCODERS (ALEs)
ALES To asses how CLAP-like Audio-Language Encoders (ALEs) perform on MMAU as
16

---

Published as a conference paper at ICLR 2025
4 evaluate sev- eral open-source ALEs, including (i) CLAP, fully open-source model de- signed primarily for sound and music comprehension. We tested different variants of CLAP, such as LAION- CLAP (Wu* et al.) 2023) and MS- CLAP Elizalde et al. (023). (ii) Re-
shown in Table
Models
Size
Sound
Music
Speech
Avg
CompA-CI.AP
416M
42.66
38.20
27.98
36.28
ReCLAP LAION-CLAY
416M
47.43
34.83
29,51
37.26
416M
45.10
35.19
25.61
35.30
MS CLAP 2023
159M
52.10
40.00
28.78
40.29
Table Performance comparison of ALES on MMAU benchmark.
CLAP Ghosh al. (2024b), an open-
source model enhanced with prompt augmentations for robust sound understanding. (iii) CompA-
CLAAPGhosh et al. (2023), a model that excels performing compositional reasoning with sound.
Evaluation Strategy To evaluate ALE on MMAU, we adopt methods similar to those used for as- sessing question-response performance in entailment models (Deshmukh al., 2024 Trivedi al.) 2019). First, we convert each question-choice pair into a hypothesis using GPT-4o (details in Ap- pendix Nj. We then encode the audio and hypotheses with ALE and select the best hypothesis based on the cosine similarity between the audio and hypothesis embeddings. Finally, accuracy to measure the performance across all data points.
micro-
Results Despite their encoder-only architecture, ALEs perform well in our evaluation setup, which is tailored for them. This is similar to findings in (Deshmukh et al.] 2024), where authors find ALEs to perform better than LALMs in deductive reasoning. However,
discuss that ALEs
benefit from acting as bag-of-words models in our evaluation scheme (and possibly in Deshmukh al. (2024) too). Future work could refine the evaluation process to better differentiate ALEs from LALMs.
Result Analysis While ALEs outperform LALMs in deductive reasoming, their advantage stems from the bag-of-woot wfordd nature of these models. To demonstrate this, we conduct qualitative analysis of responses generated by MS CLAP, shown in Fig. g Similar to (Ghosh et al. 2023), our findings reveal that these models struggle significantly when presented with counter-options containing the exact words in a different order, highlighting their lack of compositional reasoning. Future research should focus on improving the quality of options to assess the reasoning capabilities of ALEs better.
B.2 EVALUATING ALES AND LALMS ACROSS VARYING DIFFICULTY LEVELS
Table 5 provides the performance of ALES and LALMs across different dif- ficulty levels of MMAU. The models exhibit slightly better performance on medium tasks, with noticeable drop in performance for hard tasks. This trend suggests that while ALES and LALMs are capable of handling moder- ately complex challenges, they struggle with more intricate tasks. indicating po- tential limitations in reasoning or under- standing complex audio task dif- ficulty increases.
Easy (2237)
Medium (5573)
Hard
Models
(2190)
LAION-CLAP
38.72
36.97
27.60
SALMONN
20.31
39.33
30.63
GAMA
31.36
35.70
22.85
Qwen2
50.59
55.63
46.99
Gemini v1.5
57.04
51.49
52.07
Gemini 2.0 Flash
51.32
66.16
50.91
Average
41.55
47.55
38.51
Table 5: Performance Comparison of ALEs and LALMs at Different Difficulty Levels
B.3 FEW-SHOT RESULTS
We present the few-shot results of Qwen2-Audio on the test-mini subset in It supports multiple au- dio inputs, unlike most LALM baselines, which are limited to single audio inputs. This experiment tests the model's ability to leverage additional context from multiple audios. The model's perfor- mance degrades as provide more examples in the context. The model's performance degrades as we provide more examples in the context. Handling more audio inputs can increase complexity and introduce noise, making it harder for the model reason effectively.
17

---

Published as a conference paper at ICLR 2025
Skills
Questions
Based on the given audio, what is most likely happening in this scene?
A. It is most likely that person is hitting various bells with
rod in the scene depicted in the given andio.
B. It is most likely that rod is hitting various bells with a person
Acoustic Scene Reasoning
in the scene depicted in the given audio.
C. It is most likely that person is hitting various rod with bell
in the scene depicted in the given audio.
D. is most likely that a bell is hitting various person with a rod
in the scene depicted in the given audio.
Based on the given audio, what events are most likely occur- ring?
Based on the given audio, it most likely that horse moo- ing and a cow galloping.
Based on the given audio, it most likely that a cat is mooing
Acoustic Scene Reasoning
and a dog is galloping.
Based on the given audio, it is most likely that a horse is gal- loping and a cow is mooing.
D. Based on the given audio, it most likely that horse and cow are galloping.
Given the audio sample, what might have caused the bird chirp?
It might have been the birds speaking nearby that caused the person chirp.
Event-Baser Sound Reasoning
might have been the person speaking nearby that caused the birds to chirp.
The continuous rusting sounds in the audio sample could have caused the bird to chirp.
sudden hissing noise might have caused the bird chirp.
Based the given audio, what is likely happening?
It likely that wood is cutting based the given audio.
Acoustic Scene Reasoning
B. It is likely that saw cutting wood based on the given audio.
C. It likely that the animals making noise.
is likely that people conversing.
Figure 8: Qualitative analysis of the options selected by MS-CLAP. Correct results are highlighted in green, while predicted results are shown in red. MS CLAP behaves like bug-of-wordd model when selecting the correct options.
Domain Accuracy
Difficulty Accuracy
Qwen Audio Instruct
Total
Sound
Music
Speech
Easy
Medium
Hard
0 shot
54.35
$2.99
41.14
36.05
59.80
41.81
49.2
Shot
51.95
45.21
31.83
27.13
53.92
36.64
43.0
Shot
14.41
26.35
14.11
14.73
23.53
10.78
18.3
5 Shot
16.52
25.45
23.12
14.34
25.10
22.41
21.7
Table 6: Performance comparison of Qwen2-Audio on test-mini across different few-shot settings.
B.4 AUDIO Vs NOISE INPUT TO LALMS
Table 7 presents skill-wise results for two open-sourced LAMLs, GAMA Ghosh et al. (2024c) and SALMONN Tang et al. (2023), with audio and gaussian noise inputs. In general, our benchmark is robust, as most models perform rear-random chance under white noise. Interestingly, LALMS such as SALMONN exhibit minimal performance degradation with noise, suggesting reliance on language priors from their LLM counterparts or random guessing.
B.5 LLM BASED EVALUATION
18

---

Published as a conference paper at ICLR 2025
Skills
GAMA (Audio)
GAMA (Noise)
SALMONN (Audio)
SALMONN (Noise)
Acoustic Scene Reasoning
0.34
0.28
0.43
0.44
Acoustic-Sone SSurce Inference
0.64
0.26
0.39
0.41
Ambient Sound Interpretation
0.41
0.16
0.23
0.24
Conversational Fact Retrieval
0.31
0.14
0.31
0.30
Dissonant Emotion Interpretation
0.05
0.07
0.33
0.33
Eco-Acousti Knowledge
0.89
0.41
0.46
0,49
Emotion Flip Detection
0.26
0.29
0.09
0.11
Emotion State Summarisation
0.06
0.13
0.33
0.33
Emotional Tone Interpretation
0.18
0.21
0.35
Event-Based Knowledge Retrieval
0.07
0.15
0.10
0.10
Event-Basee Based Sound Reasoning
0.60
0.28
0.42
0,44
Musical Genre Reasoning
0.18
0.20
0.34
0.38
Harmony and Chord Progressions
0,20
0.18
0.23
0.23
Socio-cultural Interpretation
0.18
0.22
0.28
0.30
Instrumentation
0.23
0.14
0.30
0.32
Key Highlight Extraction
0.51
0.22
0.40
0,43
Lyrical Reasoning
0.22
0.22
0.31
0.32
Mclodic Structure Interpretation
0.27
0.19
0.34
0.36
Rhythm and Tempo Understanding
0.15
0.10
0.12
0.12
Sulak-Speaker Role Mapping
0.10
0.01
0.21
0.22
Phonemic Stress Pattern Analysis
0.21
0.14
0.01
0.01
Phonological Sequence Decoding
0.24
0,06
0.03
0,03
Musical Texture Interpretation Sound- Bused Event Recognition
0.12
0.16
0.30
0.31
0.81
0.33
0.35
0.40
Counting
0.08
0.10
Temporal Event Reasoning
0.24
0.34
0.28
0.27
Temporal Reasoning
0.15
0.23
0.22
0.20
Table 7: Comparison of performance across skills for GAMA and SALMONN models with and without noise.
Models
Ours
GPT-4o
The results in Table g show that our string- matching-based evaluation method produces scores comparable to those obtained when GPT-4o serves judge. Across different models - SALMONN, GAMA, and Qwen2- Audio-Instruct, the scores from our method closely align with those from GPT-4o, show- ing minimal variation, e.g., 2.1% difference for SALMONN and 2.08% difference for Qwen2-Audio- -nnttuut--nnttuctt.. This consistency indicates that our approach reliable and effectively cap- tures model performance in manner on par with sophisticated judgment mechanisms like GPT-4o. Consequently, our method offers a simpler, cost-effective, and robust alternative for evaluation.
SALMONN
33.70
31.60
GAMA
30.90
36.80
Owen2-Audio-Instruu
49.20
47.12
Table 8: Performance comparison of proposed string- matching based evaluation method and GPT-4o- as as-a--udge evaluation on the test-min eest-mini subset.
C SYNTHETIC DATA GENERATION
We outline the process for generating synthetic audio data for various tasks in the speech and sound domain. For speech domain tasks involving multi-speaker role mapping, expert annotators first craft concise synthetic conversations where roles, such as "doctor" and "patient," are discernible. We then utilize Parler-TTS (Lyth & King 2024), a lightweight, open-source text-to-speech (TTS) model, to synthesize naturalistic speech tailored to specific speaker attributes (e.g., gender, pitch, style) using textual prompts. Finally, manual filtering step performed to remove monotonous audio samples and those where speaker roles are not clearly distinguishable.
For sound data generation, we use the text-to-audio model (Evans al., 2024) to generate single audio events. These single-event audio samples then combined programmatically create multi- event compositions, reflecting realistic and complex auditory scenarios. The synthetic single-event and multi-event audio samples used for tasks such temporal event reasoning, including identi- fying sequences or durations of events. In the end, a thorough filtering is done to extract the quality audio samples and eliminate noisy audio where events are insufficiently distinguishable.
For sound domain tasks requiring skills such as ambient sound understanding, we employ audio overlay techniques create diverse pool of distinctive background sounds. Using 100 unique
19

---

Published as a conference paper at ICLR 2025
sounds from the AudioSet Strong evaluation set (Hershey al. 2021 overlayed each sound with every other sound, ensuring variation in attributes such as loudness to enhance distinctiveness. A thorough manual filtering process is then conducted filter out audio samples that might be mislaheled or where the individual sounds overlayed are not aurally distinctive.
Category
Tasks
Count
Sound
Temporal Event Reasoning
250
Sound
Ambient Sound Understanding
436
Speech
Event-Based Knowledge Retrieval
316
Speech Multi Speaker Role Mapping
260
Speech Phonological Sequence Understanding
150
Table 9: Distribution of synthetically generated audio data across different skills
D ANNOTATION DETAILS
D.I ANNOTATION
Figure 9 shows a snapshot of the tool used to annotate audio-question pairs and verify the answers. First, 3 expert annotators from each domain sound, speech, and music annotate and verify each answer for each audio-question pair as curated in the previous step. Once the annotations are done, these experts filter the most plausible samples from the annotated samples. During the annotation phase, the experts annotated = =11000 pairs of audio and question, out of which ==800 were dis- carded during filtering. During the Expert Review stage, the experts from each domain reviewed the question-answw -nnswer pair for each audio and disregarded #200 samples that either had misleading or very co-related options after the option augmentation stage had incorrect answers. The ex- perts went through the benchmark twice during the annotation & filtering stage to avoid any form of discrepancy.
D.2 ANNOTATOR DETAILS
Two sets of experts, 3 each, were separately involved during Expert Annotation & Filtering and Expert Review. Each domain, i.c. sound, speech,h and music, had expert for each Annotation & Filtering and the Review stage. The experts included 4 males and females. The experts involved in the Expert Annotation stage are MS/PhD students with a strong foundational understanding of their respective domains. The experts involved during the Expert Review stage were PhD students and industry practitioners. Their expertise was verified by their published research work and contri- bution to the domain. These experts brought with them a wealth of domain expertise and research experience. They have profound understanding of sound analysis and excel at discerning intri- cate details audio recordings. Their expertise both technical and theoretical, enabling them to approach the annotation process with nuanced insight. This background allows them handle complex audio data with precision, ensuring that the annotations are accurate and meaningful. Their combined experience in audio research is valuable asset to project, significantly enhancing the depth and reliability our annotated audio corpus.
D.3 ANNOTATION GUIDELINES
During annotation, the following guidelines were shared with the annotators:
1. Annotations must be accurate, consistent, and adhere high standard of academic rigour.
2. Listen to the complete audio before annotating the question-answer pair.
3. All questions must contain one audio, and the audio should not be corrupt.
4. All questions should be in the English language.
5. All questions must be tagged with 'task' type as defined.
6. All the questions must be tagged with 'difficulty' level.
20

---

Published as a conference paper at ICLR 2025
7. All questions must have "dataset" tag, which implies which dataset the audio actually comes from.
8. The answers to all the questions must be MCQ, and other types of question pairs must be discarded.
9. The questions should not mention the name of the audio or any information about the audio being used.
D.4 DIFFICULTY CATEGORISATION
The difficulty of each question in our dataset rated by domain experts on a scale of I to 10. For each question, we averaged the scores provided by the experts to determine the difficulty level. Questions with an average score 1-3 were categorized as "easy," those scoring 4-6 as "medium," and those scoring above 6 as "hard." These difficulty levels were assigned based on the level of ex- pertise or the amount of information required answer each question correctly. This categorization ensures structured evaluation of model performance across varying levels of complexity.
D.5 SOURCE SELECTION
To ensure unbiased and robust evaluation, audio was sourced exclusively from test or evaluation sets (when test sets were unavailable). Preliminary checks were applied to ensure quality and rele- vance before further expert-driven refinement. The key steps for each domain are outlined below:
Music: Labeled test audio files were used, ensuring that the corresponding questions were highly relevant the task.
Sound: Audio clips from the evaluation set of AudioSet Strong were selected based on the presence of minimum of two and maximum of five unique acoustic events, with each event lasting at least two seconds. This ensured high-quality, distinguishable audio samples suitable for reasoning tasks.
Speech:
For speech data, additional checks on transcription lengths and ground truth labels were applied to ensure clarity and adequate length, facilitating the generation of meaningful questions and answers.
These steps helped establish a diverse and high-quality dataset, forming a strong foundation for task development.
D.6 HUMAN EVALUATION
We recruit university students for a human evaluation study. Each participant was provided with detailed instructions and asked to carefully listen the audio samples before answering the cor- responding questions. This evaluation was designed to assess the accuracy and reliability of the benchmark, ensuring the human-leved performance for comparison with the models' outputs. The results from the human evaluators served as baseline for assessing the models' effectiveness on the task. This evaluation performed test-mini part of MMAU.
D.7 IRB
Our institution's Institutional Review Board (IRB) has granted approval for the human studies pre- sented in the paper.
MODEL. DETAILS
Audio Flamingo. Kong et al. (2024) is an audio language model that supports in-context learning (ICL), retrieval augmented generation (RAG), and multi-turn dialogues. It has shown state-of-the- art results on a variety open-ended and closed-ended audio understanding and few-shot learning tasks.
21

---

Published as a conference paper at ICLR 2025
Qwen-Audio. Chu et al. (2023) is large-scale audio language model supporting diverse audio types, languages, and tasks. achieves state-of-the-ane performance across various benchmarks, showing its universal audio understanding capabilities. Qwen-Audio also leverages its ability by supporting multilingual, multi-turn dialogues with flexible input from both audio and text through Qwen-Audio-Chat.
Qwen2-Audio. Chu et al. (2024) is Large Audio- Audio-Language Model (LALM) built on Qwen-Audio, designed to process both audio and text inputs generate textual outputs. Qwen2-Audio shows state-of-the-art performance in instruction-following capabilities across speech, sound, music, and mixed audio subsets, demonstrating its proficiency in audio understanding and dialogue capabilities.
LTU. Gong et al. (2023c) is a multi-modal large language model focusing on general audio under- standing, including reasoning and comprehension abilities. LTU is trained on a of closed-ended and open-ended questions with perception-tor -ooppiioontt-uddesttadddd training approach. LTU demonstrates strong performance and generalization ability on conventional audio tasks such as classification and captioning.
LTU-AS. Gong et al. (2023a) proposes joint audio and speech model. uses whisper as the audio encoder and Llama the reasoning model, combining strong perception and reasoning abil- ities, showing competitive performance on all tested closed-ended audio and speech benchmarks, particularly on tasks requiring joint audio and speech understanding.
SALMONN. Tang et al. (203)) is a multimodal large language model designed to perceive and understand speech, audio events, and music, showing significant step toward achieving generalized auditory capabilities for LLMs. excels in tasks such as speech recognition, audio captioning, and speech translation while generalizing to tasks like slot filling, keyword extraction, and speech translation for a variety of languages. It also exhibits remarkable emergent abilities, including audio- based storytelling and speech-audio co-reasoning.
Pengi. Ge et al. (024) was one of the first efforts to achieve general-purpose audio understanding through frre-form language generation with transfer learning. It excels at several close-ended and open-ended audio tasks. It leverages transfer learning by framing all audio tasks as text-generation problems. Pengi shows state-of-the-ar performance across 21 downstream tasks in various audio domains, demonstrating the capability of a
audio language model.
MusiLingo. Deng al. (2023 a music language model designed for music question- -anwering and captioning. Musiosngo's s framework includes a single projection layer, which aligns music representations with textual contexts, resulting in a competitive performance for a variety of music question-answering tasks and music captioning.
MU-LLaMa. Liu al. (2024b) a music language model for music -nnweeing and cap- tioning. It generates captions by answering music-related questions for the given music and demon- strates exceptional generalization capabilities, making it highly effective across various music- related tasks. It exhibits superior performance in both music annweeinn and music cap- tioning tasks, surpassing the current state-of-the-arr models.
M2UGen. Hussain et al. (2023) is music language model focusing music understanding and multi-modal music generation tasks, multi-modal music generation, and music editing. M2UGen shows sfate-of-the-art results various tasks, including music understanding, music editing, and tex/fimage/video-to- music generation.
GAMA. Ghosh al. (202c) is large audio language model with advanced audio understanding and complex reasoning abilities. By integrating an LLM with various audio representations, the model delivers comprehensive understanding of input audio. It demonstrates tftte-of-the-ar per- formance on 16 datasets spanning 4 tasks, significantly surpassing previous audio-language models on standard audio and music understanding.
MS CLAP Elizalde et al. (2023) is an audio language model trained with contrastive learning between audio data and their corresponding natural language descriptions. It extracts representations from both audio and text encoders.
CompA-CLAP. Ghosh et al. (2023) an extension of CLAP that is trained exclusively on open- source datasets. It is further fune-tuned with specialized algorithms and datasets to enhance compo- sitional reasoning capabilities.
22

---

Published as a conference paper at ICLR 2025
LAION-CLAP. Wu et al. (2023) proposes large-scale contrastive language-audio pretrain- ing model that leverages a newly introduced dataset called LAION-Audio-63: which includes over 630k audio-text pairs. The model combines audio and text encoders with feature fusion and keyword-to-caption augmentation, improving performance on text-to-audio retrieval, zero-shot au- dio classification, and supervised audio classification tasks.
ReCLAP. Ghosh et al. (2024b) builds on the work of LAION-CLAP, and introduces an enhanced CLAP model trained with rewritten audio captions to improve zero-shot audio classification (ZSAC) and retrieval tasks. The ReCLAP model is trained on *2.3M audio-caption pairs.
F DATASET DETAILS
Table [0 presents the frequency distribution of synthetic and real data, along with the sources from which the real data is pooled.
AudioSet. Gemmeke et al. 2017) Audioset a large-scale audio event dataset comprising over 2 million human-annotated TO-second video clips. The dataset is labelled using a hierarchi- cal ontology of 632 event classes, allowing the same sound to be tagged with different labels. AudioSet Strong. [eershey et al. (2021) The AudioSet Strong dataset is an extension of the original AudioSet, containing 67,000 clips with strong labels (precise, 0.1 annotations) from a subset of the original 1.8 million weakly labelled clips. It spans 356 sound classes with detailed start and end times for events, providing over 200 hours of audio. This dataset is used improve audio event classification and evaluate classifiers with both positive and challenging negative labels.
Dataset
Audios
Audioset
2788
AudioSet Strong
391
Mustard
405
MELD
540
VoxCeleb-1
633
MUStARD. Castro al. (2019) MUStARD a multi-modal video corpus for research in automated sarcasm discovery. MUStARD is curated from popular TV shows such as Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholies Anonymous. MUStARD comprises 690 videos with an even number of sarcastic and non-sarcastic labels.
IEMOCAP
515
MusicBench
1937
Jamendo
32
SDD
277
MusicCaps
514
GuitarSet
506
MELD. Poria et al. (0188) The Multimodal EmotionLines Dataset (MELD) is a multimodal dataset designed for emo- tion recognition in conversations. It contains around 13,000 utterances derived from 1,433 dialogues from the TV series Friends. These dialogues include audio, visual, and textual components. Each utterance is annotated with emotion and sentiment labels.
MUSDB 18
68
Synthetic
1394
Table 10: List of sources from where MMAU pooled.
VoxCeleb. Nagrani et al. (2017) The VoxCeleb dataset is large-scale speaker identification corpus containing over 100,000 utterances from 1,251 celebrities. The dataset is used for both speaker identification and speaker verification with noisy, unconstrained speech, making it useful for real- world speaker recognition tasks.
IEMOCAP. Busso et al. (2008) The IEMOCAP dataset is used for emotion recognition, consisting of 302 videos of dialogues recorded across sessions with pairs of speakers. includes emotion labels: angry, excited, fear, sad, surprised, frustrated, happy, disappointed, and neutral, as well as valence, arousal, and dominance annotations.
Agostinelli et al. 20233) MusicCaps is music caption dataset consisting of 5.5k music clips from AudioSet by focusing exclusively on music content, each paired with text descriptions written by ten professional musicians. For every 10-second clip. it provides free-text caption (four sentences on average) and a list of music aspects like genre, mood, tempo, and instrumentation. The dataset includes around eleven aspects per clip and genre- balanced split with 1k examples.
MusicCaps.
MusicBench. Melechovsky et al. 2023) MusicBench is a dataset for text-to- rext-to-music music generation, ex- panding the original MusicCaps dataset from 5,521 to 52,768 training samples and 400 test samples. enhances the dataset by adding music features such as chords, beats, tempo, and key, described via text templates, and by applying augmentations such pitch shifts, tempo, and volume changes.
23

---

Published as a conference paper at ICLR 2025
ransscipt
Ayrdies (smueed)
deyy' plane Lael
Queadin
mecrioe tfat bur par??
Choines:
liperae medge ormanse remrn
Correct
Speieeee winggn eneenet benin pecvocs.
Dent
Acoesi
Figure 9: Snapshot of the annotation tool used by the annotators to annotate the correct answers for each audio-question pair.
MTG-Jamendo. Bogdanov et al. (2019) The MTG-Jamendo Dataset is dataset for automatic music tagging, featuring over 55,000 full audio tracks, each annotated with 195 tags spanning gen- res, instruments, and moods/themess The dataset includes 3,565 artists with 3,777 hours of audio in high-quality 320 kbps MP3 format. It includes five predefined splits for training, validation, and testing, with no overlap of tracks from the same artist across sets.
SDD. Manco et al. (2023) The Song Describer Dataset (SDD) is used as an evaluation tool for music-and-language models, enabling benchmarking tasks such as music captioning and text-to- music retrieval. It contains 1,106 human-written captions for 706 music recordings collected from 142 annotators. The dataset features audio-caption pairs with deseriptions focused various musi- cal elements like genre, mood, and instrumentation.
GuitarSet. et al. (2018) The GuitarSet dataset contains hours of guitar recordings from ex- perienced guitarists, each performing 30 excerpts of various musical genres, including Rock, Jazz, Funk, Bossa Nova, and
It provides rich annotations like tempo, key, chords, beats, and note-level transcriptions. The dataset includes time-aligned data on string/fret positions, chords, and playing style, offering valuable resources for tasks such as guitar transcription, perfor- mance analysis, beat tracking, and chord estimation.
MUSDB18.
Rafi et al. (017)) The MUSDBIS 18 dataset is widely used for music source separation tasks. The dataset consists of 150 full-track songs across various styles. It includes 100 songs in the training set and 50 songs in the test set, with each track split into 5 stereo streams: mixture, drums, bass, accompaniment, and vocals.
G ANNOTATION TOOL
Figure shows snapshot of the tool used by the annotators. Annotators were shown the audio, questions, options, and answers. The annotators were asked to listen to the audio and annotate if the answer shown was correct and the option. The annotators had the option either accept or reject the question-answer pair for the given audio.
H COMPARISON
Table II highlights the differences between MMAU and previous benchmarks, particularly in terms of the increased difficulty and required complex reasoning ability that MMAU's questions present to the models.
24

---

Published as a conference paper at ICLR 2025
Category
Priur Benchmarks
MMAU
Tiskk Simple event idantiication
Tack: Ambiest Sound Understending
Sound
Example: "What's
of sound""
Example: "What materal is typically used the the instrument"
Difficalty:
Difficuity: Hard
Dataset: AirRench
MMAUI
Speaker demviiication, emotion detection
Tack: Conversatooal Anilysi
Speech
Example:
speaker's
Example:
surgeon responsible
mentioned"
Diflicalty: Easy
Diflicully:
Dataset: AirHench
Dataset; MMAU
Task: Gence Semilication MIDI pitch descetion
Task; Instracsent idemifleation, vocal characteristis asalysis
Music
Rxample: "What's 's the genre of this musi?"
Example: "Which instrument is playing the high notes?"
Difliculty: Easy
Diflicultt;; Medium
Dataset: AirRench
Dataset: MMAUI
Table II: Comparison of MMAU vs Prior Audio Benchmark
I EVALUATION ALGORITHM
The proposed algorithm evaluates the correctness of a prediction against given answer multiple-choice setting. It operates by tokenizing the input strings into of lowercase words, enabling a robust comparison by disregarding case and punctuation variations. The algorithm first extracts the tokens from the correct answer and the prediction.
also identifies tokens from in- correct choices, excluding any shared tokens with the correct answer to avoid penalizing common vocabulary. The evaluation hinges on two conditions: (i) All tokens from the correct answer must be present in the prediction. (ii) The prediction must not contain tokens unique to the incorrect choices. both conditions are satisfied, the algorithm returns true, indicating a correct prediction; otherwise, it returns false. This approach ensures balance between strict answer matching and resilience against irrelevant or misleading content the prediction.
Algorithm 1: String Match Evaluation Algorithm
Input answer: The correct answer string prediction: The predicted answer string
choices: A list of multiple-choice options (including the correct answer)
Output: boolean value indicating whether the prediction is correct.
Helper Function: Tokenize (text)
Convert text to lowercase;
Extract word tokens using word boundary regular expression : /bb (b;; Return the of tokens;
Main Algorithm:
answer tokens + Tokenize (answer);
pecdicttiontokknns Tokenize (prediction);
it' prediction tokens is empty then
return False:
Idetiffy Tokens from Incorrect Choices: incorrect tokens +- 0:
foreach choice in choices do
tokees Tokenize (choice);
schoice_tokent + answer_tokens then
Incorrect.Jokent +- iocorsect.tokens U (choice_tokene answer.tokens));
Evaluate Conditions:
condl + (answer S prediction_tokeess);
cond2 (prediction tokens incorrect tokens 0);
Return Result:
return cond! AND cond2;
25

---

Published as a conference paper at ICLR 2025
J ADDITIONAL INFORMATION ON SKILLS
Table 12 shows examples of questions in the benchmark that require more than one skill to solve. Approximately 8% of the questions involve overlap of information extraction and reasoning skills, while around 18% of the questions inherently require multiple skills to arrive at the cor- rect answer. The table illustrates specific examples where the overlap of information extraction and reasoning is essential for solving the questions effectively.
Domaín
Skills Involved
Question (with options)
Speech
Multi Speaker Emotion Reasoning
What makes the last comment
(Reasoning Based)
sarcastic in relation to the dialogue? Options:
Dissonant Emotion Interpretation
(Sarcasm Interpretation)
1. Criticism of scientific method.
2. Genuine admiration of intelligence.
3. Requesting further explanation.
4. Mocking exaggerated praise.
Music
Harmony and Chord Progressions
During what time frame you hear the chord G# in the audio?
Temporal Reasoning
Options: 1. 0.00 2.22
2. 2.22 4.44
3. 4.44 6.67
4. 6.67
Sound
Acoustic Scene Reasoning Eco-Acoustic Knowledge
What nature of the weather in the scene?
Options:
1. Snowing heavily
2. Sunny day
3. Windy
4. Heavy rain
Table 12: Examples of questions requiring both information extraction and reasoning skills.
The table 13 highlights the various skill challenges presented by the MMAU benchmark to the LALMs.
Domain
Skills
Tasks
Question (with option)
Temporal Event
Identify ordering and duration of various sounds
Identify the total number of drum beats in the audio. Choices:
Reasoning
Sound
A.
C. 5
D.3
Acoustic-Souret Inference
Identify the source various sounds
For the given audio sample, identify the source of the singing sound.
Choices:
A. People
B. Birds
C. Musical Instrument
D. Radio
26

---

Published as a conference paper at ICLR 2025
Eco-Acoustic Knowledge
Identify the environ- mental background based various
Based the audio, what is the likely set- ting?
Choices:
sounds
A. Beach
B. Mountain
C. City Park
D. Forest
Ambient Sound Interpretation
Extracting informa- tion about the back- ground sound
Name a famous musician known for play- ing the instrument heard in the back- ground.
Choices:
A. Yo-Yo Ma
B. Jimi Hendrix
C. Miles Davis
D. Flea
Acoustic Scene Reasoning
Answer the reason-
Based the given audio, what event is taking place?
ing questions based on the acoustic scene interpreted from multiple sounds.
Choices:
A. A person is playing percussive instru- ments simultancously.
B. Hard objects are being manipulated in various ways.
C. Someone is rolling and striking hard ob- jects.
D. A person is handling items and closing a container.
Causal reasoning question about what triggered source produce specific
Based on the given audio, what could have caused the dog's ' barking? Choices:
Sound Reasoning
A. A person approaching the dog.
sound.
B. A cat approaching the dog.
C. A laughter heard nearby
D. gentle splash of water.
Sound-Based Event Recogni- tion
Based on multiple sound, infer the most likely event from the
What type of emergency vehicle is indi- cated by the sirens the audio?
Choices:
audio
Fire truck.
B. Ambulance.
C. Police car
D. Garbage truck.
Dissonant Emo- Interpreta- tion
Identify
sarcasm
From the given conversation, What makes the last comment sarcastic in relation to the dialogue?
tion
multi-speaker
settings
Choices:
A. Criticism of scientific method
Speech
B. Genuine admiration intelligence.
C. &.Requsttig further explanation
Mocking exaggerated praise
Extract information about the event
Who the scientist behind the discovery mentioned by the speaker?
Knowledge Retrieval discussed conversation.
Choices:
A. Marie Curie
B. Albert Einstein
C. Alexander Fleming
D. Isaac Newton
27

---

Published as a conference paper at ICLR 2025
Counting
Count the number of speakers in a dia- logue
What's the number of speakers in the cur- rent conversation?
Choices:
A.
B. 4
C.
D.
Phonemic Stress Pattern Analysis
Identify the stress patterns of phonemes an utterance.
From the given utterance, identify a pair of words that contain similar sounding stressed and unstressed phonemes Choices:
A. Sometimes, want
B. hair,directing
C. first, second
D. few, blanks
Emotional State summarisation
Identify the emotions of all the speakers in a conversation
From the given conversation, Identify the emotion of each speaker
Choices:
first speaker shows neutral, anger; sec- ond speaker shows fear, neutral, disgust.
B. first speaker shows neutral, anger; sec- ond speaker seems neutral.
C. first speaker shows happiness; second speaker shows fear.
D. first speaker shows fear; second shows disgust
Conversational Fact Retrieval
Answer factual ques- tions based on the content discussed by
How much money did the second speaker
offer the first speaker marry her? Choices:
the speakers.
A. Twenty thousand dollars
B. Seventy thousand dollars
C. Fifty thousand dollars
D. One hundred thousand dollars
Multi Speaker Identify the role played by each speaker in a conver- sation
In the given conversation, identify the role of two speakers.
Role Mapping
Choices A. first speaker is voice coach and the second speaker is singer
B. both speakers are neighbors
C. first speaker is surgeon and the second
speaker surgical nurse
D. first speaker is a nurse and the second
speaker doctor
Phonological Se- quence Decoding
Identify the word order in similarly sounding words within tongue twisters.
For a given tongue twister, Identify which word came first
Choices: elves
B. elk
C. eve
D. elite
Emotion Flip De- tection
Identify which speakers showed emotion flip in conversation
From the given conversation, Identify the speakers that showed emotion flip. Choices:
A. both speakers
B. first speaker
C. second speaker
none of the speakers
28

---

Published as a conference paper at ICLR 2025
Key highlight Ex- traction
Identify the intent of the conversation
What is the main topic of discussion be- tween the speakers
Choice:
A. negative aspects of environmental pol- lution
B. improving one's relationship with sib- lings.
C. challenges of maintaining parent-child relationships
D. Impact of good communication skills
Temporal soning
Rea-
Extract information about the temporal structure of the music track/song
How does the male voice follow the strummed electric guitar in the audio? Choices:
A. It follows immediately after each strum
B. It starts before the guitar
Music
C. It overlaps with the guitar
D. It starts well after the guitar finishes
Musical Reasoning
Genre
Understanding musi- cal genre and song type
Considering the mood and elements of the audio, what the likely purpose of the song?
Choices:
A. A party anthem
B. A workout mix
C. A proposal song
D. A lullaby
Lyrical ing
Reason-
Involves analyz-
What day is mentioned in the lyrics? Choices:
ing song lyrics
interpret themes, emotions, and under- lying meanings.
A. Monday
B. Friday
C. Sunday
D. Wednesday
Socio-cultural terpretation
Analyzing how his- torical events and cultural contexts influence musical styles, genres, and themes.
In which cultural setting would the music in the audio most likely be performed? Choices:
A. Western classical concert hall
B. Indian classical music festival
C. Modern pop concert
D. Jazz club
Melodic Struc- ture tion
Infer the organiza- tion and progression
What type of bass line is playing in the au- dio?
Interpreta-
of melodies under- stand their patterns, forms, and emotional expressions.
Choices:
A. Acoustic bass line.
B. Groovy synth bass line.
C. Fretless bass line.
D. Double bass line
Harmony and Chord Progres- sions
Involve the study of how chords interact
What the chord progression in the audio?
Choices:
and transition cre- ate musical texture, mood, and overall structure.
A. C. G, Am,
B. G7, Fm, Ab, Eb, Bb
C. Dm, A7, G. Bm
D. F. C, Dm, Bb
29

---

Published as a conference paper at ICLR 2025
Rhythm Tempo standing
and Under-
Focuses on analyzing the timing, beats, and pace of piece
What is the tempo of the audio?
Choices:
A. 120 bpm.
B. 130 bpm.
C. 149 bpm.
D. 160 bpm
Musical Texture Interpretation
Analyzing the overall vocal quality of the singer.
What is the main characteristic of the male
voice in the audio?
Choices:
Soft and mellow
B. Loud and soulful
C. High-pitched and fast
D. Monotone and slow
Instrumentation
Extracting informa- tion about various in- struments present a musical piece
What is the primary instrument playing in the audio?
Choices:
A. Violin
B. Flute
C. Guitar
Piano
Emotional Tone Interpretation
Analyzing the feel- ings conveyed in music to understand the emotional impact and mood of a piece.
How would you describe the impact the simple guitar solo in the bridge the song's mood? Choices:
A. It introduces sense of calmness.
B. adds complexity and tension
C. enhances the upbeat and dynamic feel.
D. It makes the song sound more melan- cholic.
Table 13: Details on categories, type of questions with examples for each task
K
FAILURE CASES
The table below highlights the failure cases of the top-performing LALMs, with examples drawn from the Owen2-Audio-Instrenn model.
Domain
Category
Question (with options)
Answer
Model Response
Acoustic-Soues Source Inference
Based on the given audio, identify the source of the music.
Radio
Construction site
Sound
Choices:
A. Fire truck
Radio
C. Airplane
D. Construction site
Acoustic-Source Inference
Given the audio, identify the source of the mechanism sound.
Machine
Human
Choices:
A. Nature
Machine
C. Human
D. Animal
30

---

Published as a conference paper at ICLR 2025
Acoustic Reasoning
Scene
Based the given audio, what event is most likely oc- curring?
A bell tower is signal- ing an event.
An alarm clock is ringing intermit- tently.
Choices:
A. An alarm clock is ringing intermittently.
A small handbell is being rung.
C. bell tower is signaling event.
A doorbell is being re-
peatedly pressed.
Acoustic Reasoning
Scene
Given the audio, which event most likely occur- ring?
Rain patterns metal surface.
Water quickly slows down.
drips then
Choices:
A. Water drips quickly then slows down.
tap is dripping into a basin.
Rain falls patter beat then stops.
D. Rain patterns on a metal surface.
Ambient Sound Understanding
Identify the instrument play- ing in the background.
Guitar
Piano
Choices:
A. Guitar
Flute
C. Piano
Violin
Event-Basse
Who developed the vaccine mentioned by the speaker?
Dr.
Dr. Albert Sabin
Knowledge Retrieval
Jonas Salk
Choices:
A. Dr. Jonas Salk
Dr. Louis Pasteur
C. Dr. Albert Sabin
D. Dr. Robert Koch
Multi-Speaker Identity Profiling
How many speakers are
Three
Five
present in this conversation?
Speech
Choices:
Three
B. Four
Six
D. Five
Phonemic Stress
From the given utterance, count the number of words that contain at least one stressed phoneme.
Nine
One (incorrect reasoning)
Pattern Analysis
Choices:
A. Four
B. Nine
C. Seventeen
D. One
31

---

Published as a conference paper at ICLR 2025
Conversational Fact Retrieval
What is Second Speaker's
Jones
John
first name according to First Speaker?
Choices:
A. Jack
B. John
C. Jones
D. James
Conversational Fact Retrieval
Who directed First Speaker
Second Speaker
Third Speaker
to get line?
Choices:
A. Fourth Speaker
B. Third Speaker
C. Second Speaker
D. First Speaker
Metre
and
What is the tempo of the au- dio in bpm?
135.0
150.0
Rhythm
Music
Choices:
A. 160.0
135.0
C. 120.0
150.0
Melody
Which instrument pri- Electric marily responsible for the guitar melody in the audio? Choices:
Piano
A. Piano
B. Violin
Electric guitar
D. Flute
Historical and
Identify the lead instrument in the jazz track as described the audio.
Trumpet
Saxophone
Cultural Reason-
ing
Choices:
Piano
B. Guitar
Trumpet
D. Saxophone
Emotional Tone
What kind of emotional re- sponse is the audio most likely intended evoke?
Seriousnešs Calm and seren- and ur- ity gency
Choices:
Seriousness and urgency
B. Sadness and contempla-
tion
C. Joy and excitement
D. Calm and serenity
Table 14: Model Failures in Sound, Speech, and Music Categories with Sub- Sub-Category Information
L BENCHMARK EVALUATION
We asked domain experts to rate each existing benchmark scale of to 5 based the diffi- culty level of solving the questions. For each benchmark, we randomly selected 1,000 samples (or evaluated the entire benchmark if it contained fewer than 1,000 examples). Domain experts were
32

---

Published as a conference paper at ICLR 2025
instructed listen to the audio and answer the corresponding questions. following fixed set of guidelines. These guidelines included the breadth of the questions (e.g., variety, question type such open-ended or multiple-choice), domain coverage (speech, music, sound), and depth of the ques- tions (e.g., whether they required multi-step reasoning or involved different types of reasoning such content-based, causal, contextual).
To ensure unbiased evaluation, the benchmark names were not revealed in advance. Before assigning difficulty score. each expert was asked summarize their evaluation in one sentences. We aggregated the feedback and difficulty scores from all domain experts and presented our findings in Table 2
M ADDITIONAL DETAILS ON ERROR TYPES
Error Type
Definition
Question
Prediction Waterfall
Reason Misinterpreted the sound
Perceptual Er-
The model fails to perceive the audio correctly.
Based on the given au- dio, identify the source of the following sound.
ror
Choices:
A. Stream
B. Faucet
C. Waterfall
D. Rain
Knowledge Error
The model un- derstands the audio but lacks the knowledge to answer.
What the typical fre- quency range of the in- strument playing in the background?
20-200 Hz
Lacked specific frequency knowledge
Choices:
A. The bass typically ranges from 40 Hz to 400 Hz.
B. The bass typically ranges from 400 Hz to 4 kHz.
C. The bass typically ranges from 20 tz 200
Hz.
D. The bass typically ranges from kHz to 40 kHz.
Reasoning Er- ror
The model strug- gles with logical reasoning.
What weather condition Humid
Incorrect rea-
is indicated by the au- dio?
soning about sound
Choices:
A. Windy
B. Calm
C. Humid
D. Rainy
33

---

Published as a conference paper at ICLR 2025
Error Type Annotation
Defniiion
Question
Prediction Singing with music
Reason Answer key was incorrect
The model's sponse correct but the answer key is wrong.
Given the audio sample, what was the primary focus of the audio?
Error
Choices:
A. A man speaking with background mu- sic
B. A man breathing
heavily
C. Only music playing
continuously
D. A man singing with
music
Answer Extraction Error
The model's an- swer matches but formatting leads incorrect marking.
Based on the given audio, what could have led to the shout?
Whip sound
Incorrect for- mat in answer
Choices:
A. A whip sound oc- curring just before the shout
B. Continuous music playing in the back- ground
C. Human voice heard earlier in the audio
D. Whistling and ap- plause towards the end
Other Error
The model refuses to answer
Based on given audio, what is the most likely source the noise?
Refused answer
None of the options fit
counters another issue.
Choices:
A. A malfunctioning
electronic device
B. A gentle breeze
C. A calm river stream
D. distant bird chirp-
ing
Table 15: Additional details on Error types with some examples from MMAU, The model predic- tions are taken from Gemini Pro v1.5
34

---

Published as a conference paper at ICLR 2025
N
PROMPTS
Pzooepti
you taps,
optioms
vartery contrarciig oprions.
oppeoltes or Hoorsigoosttes
Eeaople: spsakez discsases paritive axpect of thecey, one option asy catcion the thacry's bensfite, siile anester apsion
could aupgeat drasbacke.
Bow is conconea: Teat: Lakers nightt mivinterppes the centast or oreziout bow toe apeakes is uatoraning beth sides or iase.
Dartiai Correctauzz
part
sonuraclly crostat detail
inacouracy incoepiete
Pacaperaning with
Eumple: The option nigus rxplicuse the spectuc suid but intzoduce aubicu chanpe reuning (eugu) "cequizes"" to *resoarende")
It confuees The zobete change sight seew Insigntriont, bet to alters the searing and teads the wrong chefoe,
Hisleading Steilacities
the Krapple: optiaza stailss, with amal: diffferenca in wording, laading tast-tatace
octez.
appиz
distinpusti, aaking ditticuit
Rengerated
Esmple: apsabez zanticra alnoz point, one sption night exaqpezate leve., tazning tright sffects inte "datinitaly atfecta").
How it concupea: The esigpesation or cnderstatemnt nigst align aist tae geseral topie but doese't acossately retlect the apeatects poist.
Tepl fed us. stated Informattor
Rasglle
correctly Infer ocweching from cald, shile aoother incorrently
explisisty
Dopie Brist Contunion
Respple: The rpeaker may thift foor ore toplo to another, and optiions night include infomatiin from both topice.
It confuees: Phot-takere Hight celect sn option relsted to different part of the conserssticn or lecture,
5. Owspore: DE Seguasce Conduster.
Seangler The speakez right, describe segietos events, but anaver cholices esuld nix the order timing.
Bow it concuneat The tust-taber select the interration but in the
Ynowledge
Esaglee
is conduana: bakazz night only on thair prior Inoviedge oz axauptiosa, aven the snawez dosan't siige liatening
passage.
30. tytiens with Betea Intorration
Reangle: an optiion right seen cornect but sdde soformation that was not nentioned by the rpasker,
Row to confuine: The aditetooal: dessll asy zeee plauetble but docrnte actually nefleot the content of the listening pacrage.
Seta that each contrsative sptier. mast not uxceed socda. the must
fornat, The baspiste
Baze queation: equestion), question equeation the
Figure 10: Prompts/Instuutions used for generating contrasting options for MMAU,
35

---

Published as a conference paper at ICLR 2025
APrompt2
Please transcribe the the spoken worda in the audio clip aocurately. Capture all spoken content verbatim, including any significant pauses, emotioas, oz emphasis expressed by the speaker, no the interpretations descriptions beyond the spoken words.
#Prompt3
Please provide a detailed descripticn of the muaic in the audio clip, Include information about the genre, instruments, tempo, sood, and notable nelodles harmonies. Describe any present, including lyrics they are the overall atmosphere and emotions conveyed by the music.
discernible.
#Prompt4
Please describe al: the events and sounds occurring in the audio clip in detail. Identify and describe each sound source, such objects, animala, weather, envi ronmentsl noises. Include infermation about the sequeace of evente and any interactions between sound sources. Mention the context or setting if it can be infecred from the sounds.
Figure 11: Prompts/Instructions used for generating captions using Qwen2-Audio.
Given question answer, reformulate
single premise statement.
Examples:
Question: Does the audio contain any mel melody?
Answer: It's hard to tell.
Premise: It is difficuls to determine whether the audio contains any melody,
Question:
What instrument plays the melady after the male vecal in the audio?
Plano.
instrument that pi ays the melody after the male vocal the audio
piano.
Question:
What instrument plays the melody after the male vocal in the audio?
Trumpet.
instrument plays melody after vocal audio is trumpet.
Provide premise follewing questien answer jaon format:
Question:
(question)
Answer:
[answer]
Premise:
Figure 12: Prompts/Instructions used for generating hypotheses using question-choice pairs.
You are judge to decide whether prediction matches with the criginel ansver. Just return 1 if the predicted onswer matches the true answer, else return 0. The predicted anawer must conceptually aliiged with the setuall answer.
For example:
Predictec: "sound of dog", Actual: "dog barking" Roth are conceptuslly
aligned, so the match la
Predicted: "a cat neowing", Actual: "dog barking" These are not conceptually aligned,
The output
strictly be in the JSON format: (I'match': 0)] (I'metch': 11:.
Here is the predicted answer: <nodel outpot>>
Berr is the correct answer: koriginall anawer>
Figure 13: Prompt used in GPT-4o for LLM as judge evaluation on MMAU benchmark across various LALMs
36


Reviews:
[{"review_id": "lH66A3DqLx", "reviewer": "Reviewer_W76o", "summary": "This paper introduces MMAU, a new benchmark designed for audio large language models, akin to the role of MMLU and MMMU for text and vision. The authors hired \"domain experts\" to curate a challenging set of 10,000 annotated audio-based question-response pairs spanning speech, music, and environmental sounds. The author further benchmarked 18 open-source and proprietary large audio-language models, and shows a few interesting observation and insights.", "questions": "Section 5.2\n\nI am wondering about the breakdown (e.g., categories) for this experiment. Do you provide the ground truth text of the speech to the model? It seems a bit strange to me that, for sound classification and speech recognition, the model could do anything meaningful with Gaussian noise.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The topic of this paper is very important — currently, most audio LLMs are benchmarked using different evals, and researchers must test other models for comparison, which is tedious and impractical for many. A unified benchmark is common in text and vision, such as MMLU, MMMU, and DocVQA, and has already demonstrated its effectiveness as a good measure of models. We need one for audio. \n\n2. The authors also benchmarked 18 LLMs, which is a non-trivial task as some open-source repositories are not easy to use. With this benchmark, we gain insights into performance comparisons among them, making this paper also serve as a very valuable review paper. The result numbers contain a lot of information.\n\n3. The paper includes some good discussions.\n\n4. Overall, the writing is clear, with detailed information provided in the appendix.", "weaknesses": "1. I would like to see more discussion for an ICLR submission. Although the authors did significant work, I encourage them to add more insights to the results, such as exploring why one model performs better than another in specific benchmarks.\n- e.g., is the difference of a specific test more likely from the training data (e.g., Google's model trained on more data than those from academia), or from the architecture/training (e.g., early fusion, late fusion, discrete token vs continuous embedding vs text input). And what's the potential way to improve. \n\n2. Evaluations are crucial but must be carefully designed. I did not notice a few-shot setting mentioned in the paper (unless I overlooked it), yet few-shot evaluation is quite common in LLM benchmarking (i.e., Q1->A1; Q2->A2; Q3-?).\n- If running few shot on all models / tests is hard, I suggest to run it for some representative models and report the gap between zero-shot and few-shot, this is particular important for model not post-trained on things like multi-choice questions (select one from ABCD). \n\n3. I understand this might be due to the timing of when the authors prepared the paper or potential rate limitations, so this is not a major flaw. However, given that OpenAI now offers real-time API and voice mode API, it would be a valuable addition to include these capabilities.\n- And I understand it might be quite expensive to run the full test, if that is the case, maybe you can run on a set of hard tests and compare with Google's model. But I feel a full run would greatly increase the value of the paper - it tells the reader which is the best audio model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ngzOaVHTyo", "reviewer": "Reviewer_hLoT", "summary": "This work introduces MMAU, an audio understanding benchmark designed to evaluate the advanced auditory perceptual and reasoning capabilities of LALMs. MMAU features 10,000 human-annotated audio-question-response pairs, encompassing 27 distinct skills across unique tasks. Most existing LALMs are tested on MMAU in the paper, highlighting the challenges in auditory understanding faced by current models.", "questions": "- What tasks is the synthesised data primarily aimed at? How is the data synthesised?\n- How is the difficulty (\"easy\", \"medium\", \"hard\") of a question determined? Since for some skills, existing models consistently perform badly regardless of the difficult levels of the questions, does this mean that the difficulty should be classified by the skill type, rather than the question?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- In this paper, a large-scale audio understanding benchamrk \"MMAU\" is built to evaluate LALMs. Different from previous benchmarks, MMAU not only pays more attention to deeper and more difficult auditory reasoning tasks, but also covers a wide range of sound signals as well.\n- I believe MMAU is a reliable benchmark, as it is carefully designed during the build process with human review at each step.\n- Most exsiting LALMs are evaluated on MMAU in this paper. Besides, the authors have analysed the weaknesses of existing LALMs in detail, providing directions for further studies.", "weaknesses": "- It seems that MMAU primarily focuses on short audio (around 10 seconds) and lacks evaluations involving perception and reasoning for long audio. Since long audio includes more contextual information, the model's ability to understand long audio might offer a broader indication of its overall performance.\n- MMAU is still predominantly multiple choice, but existing LALMs may not be good at multiple choice. Perhaps open-ended questions could be used as prompts instead when testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SteFvL2CCw", "reviewer": "Reviewer_ZoCS", "summary": "The paper presents MMAU, a new benchmark to evaluate large audio-language models (LALMs) on complex multimodal audio understanding tasks. It includes 10,000 human-annotated audio-question-response pairs covering speech, sound, and music. The goal is to benchmark knowledge extraction and advanced reasoning across 27 distinct skills. The authors evaluate 18 open-source and proprietary models, revealing a significant gap between current LALMs and human capabilities. The benchmark's difficulty and breadth emphasize the limitations of current audio-language models, encouraging future research.", "questions": "- The authors mention using robust regular expressions and response-processing workflows to extract key information, matching it to the provided options via string matching. Evaluating ALMs that produce open-ended responses is challenging as they often don't adhere to multiple-choice options or instructions, making regex unsuitable. Have the authors considered using an LLM to evaluate outputs for this task? More details on the regex method, including human annotation verification for a subset of randomly sampled data, would be helpful.\n- In Appendix E, the authors list dataset sources used. Among these, AudioSet and AudioSet-strong are commonly used to train ALMs. This overlap between training and test audio data could skew results and conclusions. Have the authors conducted experiments to ensure this isn't the case or used a separate test set to avoid this issue? Clarification would be appreciated.\n- Additional information on the synthetic data generation and its domain would be helpful, I could not find this in the paper", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- MMAU significantly improves upon existing benchmarks by covering 27 distinct skills across three domains (speech, sound, and music). The low accuracy scores of state-of-the-art models highlight the benchmark's difficulty and push for more advanced models to handle complex tasks.\n- It is the first benchmark for reasoning and expert-level knowledge extraction in literature, setting it apart from previous benchmarks focused on foundational audio processing tasks. This aligns with the growing demand for AI systems capable of human-like reasoning.\n- The paper includes a detailed analysis of error types (perceptual, reasoning, etc.) across different ALMs and domains, offering valuable insights into current model limitations and how to improve them.\n- The paper is well-written and easy to follow", "weaknesses": "- MMAU focuses solely on multiple-choice tasks, which could skew results towards models trained for MCQ-type question-answering and possibly even contrastive models. It would be beneficial to include an open-ended subset, even a small one, to contrast performance with the close-ended tasks.\n- The current version treats skills needed for information extraction and reasoning as separate, potentially oversimplifying the evaluation of tasks requiring a combination of skills.\n- [Minor] The contrastive models are tested on the MMAU dataset, but the results are added only in the appendix and Table 4. It would be better to add the Table 4 rows to the main Table 3 and include one main insight from the contrastive model results in the main paper.\n- [Overall] The paper is well-executed, delivers on its promises, and devoid of any major weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j80aRAlQUB", "reviewer": "Reviewer_2krx", "summary": "The paper introduces a new benchmark for evaluating multimodal audio-language models across speech, sound, and music domains, with tasks requiring both information extraction and complex reasoning. The benchmark comprises 10,000 audio clips paired with annotated questions and answers, challenging models to demonstrate proficiency in 27 distinct skills. The authors provide a comprehensive analysis of 18 models, revealing current limitations in multimodal understanding, with the best-performing model achieving only 53% accuracy, highlighting significant room for improvement in this area.", "questions": "1. The paper mentions the distinction between reasoning and extraction tasks. However, some tasks appear to require a combination of both skills inherently. What is the rationale behind maintaining this distinction?\n2. In Appendix L, the prompt requires the model to generate a correct option. What is the purpose of this approach, and how does it impact the evaluation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. MMAU is novel in its scope and comprehensiveness, covering a wide range of tasks that target specific skills in multimodal audio understanding and reasoning, addressing a critical gap in existing benchmarks.\n2. The authors enlisted real-world human experts, conducted thorough evaluations, and used GPT-4 to enrich the dataset with additional options. Detailed prompt information is also provided in the appendix.", "weaknesses": "1. While the authors claim that tasks may require multiple skills, a review of Appendix H reveals that most tasks can be completed using a single skill. This claim lacks precision and is not sufficiently supported in the paper.\n2. The data is sourced from 13 datasets, but the paper does not provide enough explanation regarding the selection of the 10,000 data points from these large datasets. In Section 3.2, the authors mention that experts were involved in classifying abilities and question-answer pairs, but there is no clear description of the selection process. How was the initial screening conducted, and how were quality and diversity ensured? Even Appendix E provides only a brief overview of the datasets, raising concerns about data quality.\n3. Although expert annotation is mentioned, there are no specific details on how bias was controlled during the annotation process. The accuracy and consistency of benchmark data are central to evaluation reliability, but Appendix C.2 lacks elaboration on how expert annotations were quality-checked and bias-mitigated.\n4. In the experimental section, while bias in model selection was avoided, the response quality of most models (e.g., SALAMONN and Qwen2-Audio) was significantly influenced by the prompts. These models were primarily trained in a QA format, and there is a noticeable difference between their open-ended responses and multiple-choice selections. This raises concerns about whether the evaluation method accurately reflects the models' limitations, and further discussion on this issue is necessary.\n5. The evaluation in the experimental section relies on multiple-choice questions and uses string matching to determine correct answers. This method seems overly simplistic. Given the openness of LLM responses, a more sophisticated approach should be used to evaluate whether a model's output is sufficiently similar to the correct answer. The authors should provide more details about the evaluation methodology to ensure results are reliably reproducible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)