Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
Authors: Chien-yu Huang
Wei-Chih Chen
Shu-wen Yang
Andy T. Liu
Chen-An Li
Yu-Xiang Lin
Wei-Cheng Tseng
Anuj Diwan
Yi-Jen Shih
Jiatong Shi
William Chen
Chih-Kai Yang
Xuanjun Chen
Chi-Yuan Hsiao
Puyuan Peng
Shih-Heng Wang
Chun-Yi Kuan
Ke-Han Lu
Kai-Wei Chang
Fabian Alejandro Ritter Gutierrez
Huang Kuan-Po
Siddhant Arora
You-Kuan Lin
CHUANG Ming To
Eunjung Yeo
Kalvin Chang
Chung-Ming Chien
Kwanghee Choi
Cheng-Hsiu Hsieh
Yi-Cheng Lin
Chee-En Yu
I-Hsiang Chiu
Heitor Guimarães
Jionghao Han
Tzu-Quan Lin
Tzu-Yuan Lin
Homu Chang
Ting-Wu Chang
Chun Wei Chen
Shou-Jen Chen
Yu-Hua Chen
Hsi-Chun Cheng
Kunal Dhawan
Jia-Lin Fang
Shi-Xin Fang
KUAN YU FANG CHIANG
Chi An Fu
Hsien-Fu Hsiao
Ching Yu Hsu
Shao-Syuan Huang
Lee Chen Wei
Hsi-Che Lin
Hsuan-Hao Lin
Hsuan-Ting Lin
Jian-Ren Lin
Ting-Chun Liu
Li-Chun Lu
Tsung-Min Pai
Ankita Pasad
Shih-Yun Shan Kuan
Suwon Shon
Yuxun Tang
Yun-Shao Tsai
Wei Jui Chiang
Tzu-Chieh Wei
Chengxi Wu
Dien-Ruei Wu
Chao-Han Huck Yang
Chieh-Chi Yang
Jia Qi Yip
Shao-Xiang Yuan
Haibin Wu
Karen Livescu
David Harwath
Shinji Watanabe
Hung-yi Lee
Venue: ICLR 2025 Poster (2025)
Keywords: speech
music
audio
benchmark
large language model
spoken language model
Primary area: datasets and benchmarks

Abstract: Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of a comprehensive evaluation benchmark poses a significant challenge. We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models. Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks, making it the largest benchmark for speech and audio evaluation. While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio. Evaluation results show that no model performed well universally. SALMONN-13B excelled in English ASR and Qwen2-Audio-7B-Instruct showed high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks. We open-source all task data and the evaluation pipeline at https://github.com/dynamic-superb/dynamic-superb.

Published as a conference paper at ICLR 2025
DYNAMIC-SUPERB PHASE-2: A COLLABORATIVELY EXPANDING BENCHMARK FOR MEASURING THE CA- PABILITIES OF SPOKEN LANGUAGE MODELS WITH 180 TASKS
Chien-yu Huang', Wei-Chih Chen', Shu-wen Yang', , Andy T. Liu', Chen-An Li Yu-Xiang Lin', Wei-Cheng Tseng", Anuj Diwan", Yi-Jen Shih?, Jiatong Shi William Chen Chih-Kai Yang' Xuanjun Chen', Chi-Yuan Hsiao', Puyuan Peng", Shin-Heng Wang'. Chun-Yi Kuan', Ke-Han Lu', Kai-Wel Wei Chang', Fabian Ritter-Gutierrez' Kuan-Po Huang', Siddhant Arora You-Kuan Lin', Ming To Chuang", Eunjung Yeo*, Kalvin Chang", Chung-Ming Chien", Kwanghee Chois, Cheng- Hssu Hsieh', Yi-Cheng Lin', Chee-En Yu', I-Hsiang Chiu Heitor R. Guimarães", Jionghao Han Tzu-Quan Lin Tzu-Yuan Lin", Homu Chang', Tu Chang', Chun Wei Chen', Shou-Jen Chen', Yu-Hua Chen', Hsi-Chun Cheng', Kunal Dhawan", Jia-Lin Fang", Shi-Xin Fang', Kuan- Yu Fang Chiang', Chi An Fu', Hsien-Fu Hsiao', Ching Yu Hsu' Shao-Syuan Huang' Lee Chen Wei Hsi-Che Lin Hsuan-Hao Lin't Hsuan-Ting Lin't Jian-Ren Lin', Ting-Chun Liu Li-Chun Lu', Tsung-Min Pait, Ankita Pasad Shih- Shih-Yun Shan Kuan', Suwon Shon', Yuxun Tanglo; Yun-Shao Tsai', Jui- Wei Tzu-Chich Wei Chengxi Wu', Dien-Ruei Wu', Chao-Han Huck Yang", Chieh-Chi Yang' Jia Qi Yip', Shao-Xiang Yuan', Haibin Wu', Karen Livescu David Harwath", Shinji Watanabe", Hung-yi Lee
National Taiwan University University of Texas at Austin Carnegie Mellon University
"Nanyang Technological University Université du Québec (INRS-EMT) IIndependent Researcher
TTyyot Technological Institute at Chicago
#NVIDIA ASSAP
Renmin University of China
ABSTRACT
Multimodal foundation models, such as Gemini and ChatGPT, have revolution- ized human-machine interactions by seamlessly integrating various forms of data. Developing universal spoken language model that comprehends wide range natural language instructions critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of comprehen- sive evaluation benchmark poses a significant challenge. We present Dynamic- SUPERB Phase-2, open and evolving benchmark for the comprehensive eval- uation of instruction-baset universal speech models. Building upon the first generation, this second version incorporates 125 tasks contributed collab- oratively by the global research community, expanding the benchmark to a to- tal of 180 tasks, making it the largest benchmark for speech and audio evalu- ation. While the first generation of cation tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing wide array of novel and diverse tasks, including regression and se- quence generation, across speech, music, and environmental audio. Evaluation results show that no model performed well universally. SALMONN-1E 10BN-13B excelled in English ASR and Qwen2-Audio- 7B-ni--r showed high accuracy in emo- tion recognition, but current models still require further innovations to handle a broader range of tasks. We open-source all task data and the evaluation pipeline at /https:///itthae com/ http:/dddaammmmsmmeeeeeae superbb dynami cuspebb
was limited to classifi-
INTRODUCTION
Recent advancements in large language models (LLMs) have accelerated the development of natural
language processing (NLP) (Touvron al. 2023a Achiam al. 2023) Li et al. 2023b Anthropic

---

Published as a conference paper at ICLR 2025
2023 Bai et al. 2023). These models can follow natural language instructions, making users quickly adopt them for a variety of applications. They have been integrated into commercial products, such as ChatGPT (Achiam et al. 2023) and Claude (Anthropic 2023). well as in the open-souree research community, including the LLaMA series (Touvron et al. 2023a by Dubey et al. (024). Yet, they are primarily text-based models, meaning they cannot process speech audio, which are essential for more natural ways of communication and interaction with the real world.
Compared to written text, spoken language has always been more natural and convenient way for humans to communicate. Spoken language conveys a wealth of information, including semantics, prosody, emotion, and speaker characteristics, while text is limited to representing semantic infor- mation, which can sometimes even depend on the prosodic cues present in spoken language (Lin) et al. 2024). This highlights the need for universal speech models and explains why automatic speech recognition (ASR) systems using text-based language models are not optimal. Several at- tempts have been made to develop instruction-bar based universal speech or audio models capable of performing various tasks, such as LTU-AS (Gong et al.) (023). SALMONN (Tang et al. 2024a).
Qwen-Audio (Chu et al. 2023 2024). and WavLLM (Hu et al. (024).
Despite significant research in universal speech models, evaluating them effectively and compre- hensively remains a major challenge. In NLP, benchmarks for text LLMs include a large number of tasks. CrossFit (Ye al. 202I) includes 160 tasks, BIG-bench (Srivastava et al. 2022) comprises 204 tasks, and Natural-Instructions (Mishra et al., 2022: Wang et al. 2022) provides 1,616 tasks. For universal speech models, several benchmarks have been proposed but with a fixed and limited set of tasks (typically around a dozen) to evaluate specific capabilities of models (Yang et al. [021]] Dunbar et al. 2022; Maimon al. 2024). This is insufficient for evaluating a universal model, as we aim to investigate whether models s can handle broader range of tasks beyond fixed benchmarks. There is a strong demand for benchmark that can evaluate these models across various aspects with a wide range of speech tasks, which led to the creation of Dynamic-SUPER SUPERB (Huang et al. (2024)).
Dynamic-SUPERB is the first benchmark for evaluating universal instruction-based speech and au- dio models. As models advance, we dynamically expand the benchmark by adding new tasks to provide better guidance for researchers in model development. We have designed well- strl-structured pipeline that harnesses the collective efforts of the community to gather diverse challenging, novel, and creative tasks. The first generation of Dynamic-SUPERI consists of 55 tasks, covering var- ious aspects of speech (e.g., semantics, speakers, etc.), making it the speech benchmark with the most tasks. However, this is not sufficient to pave the way for developing universal speech models, especially given the complexity and richness of the information conveyed by spoken language.
This paper presents the Dynamic-SUPERB Phase-2. We expanded it to 180 tasks, more than dou- ble the size of its first iteration, with contributions from the global research community. They cover a broad range of types, and some tasks present novel challenges that have not been explored in any previous research. Besides, previous speech research has typically treated music and environmental audio as background noise to be ignored. However, these sounds contain rich information and share overlapping elements that complement each other. Thus, in Dynamic-SUPERS. we also introduced preliminary music and audio tasks from established music and audio benchmarks (Yuan et al., 2023] Turrian et al., 2022). We define the core tasks by reformulating the SUPERB (Yang et al. 2021] (speech), MARBLE (Yuan et al.] 2023) (music). and HEAR (Turian et al., 2022) (audio) bench- marks for quick-round research experiments. One challenge of evaluating with such a large-scale benchmark is deriving concrete and useful insights from hundreds of evaluation results. We provide a taxonomy for every task in Dynamic-SUPERD where tasks are clustered by the specific model capabilities they probe. Researchers can follow this taxonomy to develop or reinforee specific ca- pabilities of the models they build. To our knowledge, Dynamic-SUPer SUPERB is the largest benchmark and the first to provide such a detailed task taxonomy in speech processing.
We conducted a comprehensive evaluation of several models using Synamic-SUPERE Phase-2. To handle the diverse output formats of these models, propose an automated L.LM-based pipeline for general evaluation across tasks. The evaluation results show that these models perform well only on a limited range of tasks. For example, SALMONN-13B performed well on English ASR, while Qwen2-Audio-- -Inttrcc achieved high accuracy in emotion recognition. However, they do not generalize well to a much broader range of tasks in speech recognition and paralinguistics. Notably, training on diverse data, even with sigmificant differences in signal-level characteristics, can enhance performance across domains. We observed that spoken language models outperformed
2

---

Published as a conference paper at ICLR 2025
music models in certain music tasks. This highlights the potential for developing unified models for speech, music, and general audio. We have open-sourced all materials to support reproducibility and further research. We invite researchers to help expand Synammc-SUPERB, making it more diverse and comprehensive to advance universal spoken language models.
2 RELATED WORKS
2.1
UNIVERSAL SPEECH MODELS
LLMs have shown strong natural language processing abilities and are used in speech, audio, and music applications. Recent frameworks integrate a pre-trained speech encoder with an L.LM through fine-tuning techniques such as LoRA (Hu et al,, (02T). LTU-AS (Gong et al.) (023) combines Whisper (Radford et al.) 2023) with LLaMA (Touvron et al. 2023a), and it is fine-tuned on open- ended speech and audio question-answer anwwering.. SALMONN adopts window-level Q-former (Li al. 2023a) to generate soft prompts fusing representations from the speech and audio encoders. Qwen- Audio (Chu et al., 2023) introduces task- tags into Qwen to encourage knowledge sharing and prevent interference across diverse tasks, and it supports multi-turn dialogues for both audio and text inputs. WavL.LM (Hu et al. 2024) uses curriculum learning prevent overfitting on specific tasks while maintaining the LLM's s original capabilities. All these models accept speech, audio, and text as input but output only text, focusing on understanding rather than generation in speech and audio. Several attempts have also used prompts for speech, music, and audio generation (Guo et al., 2023 Agostinelli et 2023: Liu al. 2023a). However, to our knowledge, there no universal model capable of handling both generation and understanding tasks. Moreover, these models have not been comprehensively evaluated on a benchmark, which hinders fair comparisons between them.
2.2
EVALUATION BENCHMARKS
Several benchmarks have been developed to evaluate speech models. SUPERB (Yang al. 102 is the most widely used benchmark for assessing the performance of speech foundation models across various tasks that cover different aspects of speech. On the other hand, SLUE (Shon et al. 2022 2023) focuses more specifically on spoken language understanding. Yet, they are primarily limited
to English. LeBenchmark (Evain al. 2021 Parcollet et al. (023)) evaluates speech foundation
models specifically for French, while IndicSUPERT (Javed et al. 2023) is dedicated to Indian lan-
guages. MI-SUPÈRB (Shi et al. | 2023) 2024b) offers ASR and language identification tasks in 100+ languages. Aside from speech, MARBLE (Yuan et al. (023) and HEAR (Turian et al. 2022) offer platforms for evaluating various music and a wide range of audio tasks. Using these benchmarks, researchers build a specialized model for each task. Thus, the number of tasks is limited due to the growing costs associated with adding more tasks. Conversely, universal models are expected to do various tasks without fine-tuning for each one, allowing users to engage in far more diverse appli- cations and enabling benchmark developers to include more tasks for comprehensive evaluation. In NLP and CV, benchmarks have been developed to evaluate models across much broader range of tasks (Bitton et al., 2023 Srivastava et al. 2022] Mishra et al., 2022). However, in speech, we lack benchmark of comparable scale. Dynamic-SUPERB SUPERB (Huang al. 2024a) is the first benchmark for instruction-bassd universal speech models. While its first version includes far more tasks than all other speech benchmarks, they are all classification tasks. AlR-bench (Yang et al., (024b). on the other hand, includes tasks from speech, music, and audio, and expands beyond classification, although the number of tasks available for evaluating universal models remains limited (19 tasks in foundation track). Thus, there remains a critical need for comprehensive benchmark that evaluates universal models across broader range of tasks fully assess their capabilities. Consequently, we developed Dynamic-SUPERt Phase-2, substantially upgrading the first generation with detailed taxonomy and establishing community contribution protocols to facilitate continual task integration.
DYNAMIC-SUPERS PHASE-2
3.1 OVERVIEW
Figure depicts the framework of Dynamic-SUPERB Phase-2. Our goal is to evaluate universal models that meet the following criteria: (1) The model accept speech, music, or other audio as
3

---

Published as a conference paper at ICLR 2025
Speech
Audio
(a) task collection & taxonomy
Speech Recoggittin
Sprech Translation
Phoname Recogrition
Speaker Varifoattion
Name Entily Recognition
Music Genre Classificttio
180 tasks
New Taskks
Biot Filing
Language Identification
Environmental Sound Clasefiication
Audio Spaatiel
Distance Prediction
Instrument Classflication
Guidance
Benchmarking
Review
Vartly the uttaraneis from carne speaker
According spedierr R silkkty...
Universal
Editors
the audio,
She
in thise sed begss and wed go
Research Community
collaboration
(c) task formulation
Figure 1: An overview of Dynamic-SUPERB.
input. (2) The model can perform specific tasks without requiring fine-tuning, (3) The model follows natural language instructions execute the corresponding tasks. To comprehensively evaluate these universal models, we have collected hundreds of tasks and developed a task taxonomy to better guide benchmark users (Figure [b). All tasks Dynamic-SUPERB Phase-2 are intended solely for testing purposes; we do not provide training data for two main reasons. First, current models are trained on large-scale, open-source, or proprietary datasets, making it challenging to ensure that all models are trained under consistent conditions. Second, aim to evaluate universal models that do not require fine-tuning for downstream tasks.
The Dynamic-SUPERB project is designed to evolve dynamically with research advancements by incorporating novel tasks from the research community (Figure (Tp). With the first call for tasks, Dynamic-SUPERS Phase-2 has grown from its first generation, expanding from 55 to 180 tasks. several benchmarks used in speech, music, and audio research, demonstrating that DynamicSSUPERB Phase-2 is the largest benchmark covering all three areas. It provides more fine-grained evaluation than any other benchmark.
Table 1: Comparison of popular benchmarks with Conamic-SUPERRE
Dynamic-
Dynamic-
Benchmark SUPERB
SLUE
HEAR
MARBLE
AIR-Bench
SUPERB Phase-1
SUPERB Phase-2
of tasks
13
7
19
13
19*
55
180
Speech Music
Audio
foundation track
3.2 TASK FORMULATION
As Figure T depicts, in Dynamic-SUPERB Phase-2, each task is structured to include: (1) Text instruction: A natural language instruction that guides the model on the task to perform. Each task has multiple different instructions to evaluate the model's ability to understand instructions. least one audio element, which can be the input, the output, or both. (3) (Optional) Text component: Text elements (other than instruction) that may serve as inputs or outputs. The number of audio elements in the inputs or outputs may vary depending on the task. For example, in speaker verification, two utterances are used to determine whether they were produced by the same speaker (green blocks Figure [ Besides, text inputs are not always required; for instance, ASR does not involve any text inputs. We use text format for instructions instead of spoken format. Spoken instructions involve varying content, speaker characteristics, and prosody, making them more complex. Text instructions act models. These designs ensure consistency and simplify the model's understanding and processing of diverse tasks while maintaining the benchmark's extendibility with minimal constraints.
(2) Audio component:
intermediary, bridging text and spoken universal

---

Published as a conference paper at ICLR 2025
Another problem in evaluating universal models is the format of classification and regression tasks. Generally, a task-specialized model generates predicted labels (soft distributions or hard labels) within pre-defined set of labels for classification numeric values for regression within pre- defined range. However, a universal model does not necessarily have access to this information, and thus is hard evaluate them in the same way, especially using it perform several different tasks. To address this, we believe that universal speech model should produce outputs in natural language all classification labels are represented as text. For regression tasks with specific formats (e.g., scalars, JSON, or Python- style lists), we can parse the natural language outputs using pest-proeessing pipeline (Section 4.2)
for all tasks (model outputs Figure 1+). Therefore, in
3.3 CALL TASKS
Dynamic-SUPERB fosters community collaboration, encouraging the addition of innovative tasks to remain relevant in this rapidly changing field. In Phase-2, we initiated a call for tasks in March 2024 to invite contributions from the research community. We established organized and transparent submission process on our GitHub portal, where we organize members who serve as editors to guide contributors. Contributors propose tasks by providing relevant information GitHub. Each proposal is assigned to an editor who checks for major issues and prevents duplicated efforts. Then, task proposers upload their data with proper licenses. They complete submissions by opening a pull request on GitHub with the necessary files. Afterward, editors review submissions on rolling basis, offering suggestions for refinement rather than immediate acceptance or rejection. After iterative improvements, accepted tasks are merged into the repository. Between March and July 2024, we received more than 140 task proposals and accepted over 120 new tasks. Tasks still under review are not included here but will be featured in the next phase. For details, please refer to Appendix
Huggingface space in specified formats, using only datasets
3.4
TASK TAXONOMY
Elagetate
Grainner
Plometics and Phoneme
Sound
Rhyrium
Pricesing
Intent & Meaning
Encken manaaain
Lakag Lahggrange
Spoden
Analysts
Singing
Phossolagy:
and Presedly
Prosody
Diander Clandiicatos
Automatic Sprech
Ricnn seeaggetiin
Speech
Speea
Rafely
Audio
Audio
Music
Harmony
& Pitch
adatioo Dutection &
Recoquition
seraritt
Siaeety and
Prodiction
Pack-
Spentin Audd foeelee
Speciflc
eccognition
Spaaker &
Language
Charsteristies
Tacks
Emotitn
Amilysis
Spocsker
Eveut
Nevil
Spatisl Audio
Quality Ansemment
Music Classification
(a) Task Taxonomy for speech tasks.
(b) Task Taxonomy for audio and music tasks.
Figure Task taxonomy Ceynamic-SUPERR
One primary challenge in building a large-scale benchmark offering valuable insights to users. To address this, we developed task taxonomy that that helps researchers interpret performance results across various tasks. Researchers can leverage this taxonomy to select specific tasks for model development instead of evaluating every task in the benchmark.
Figures Z shows the high-level task taxonomy in Dynamic-SUPERB Phase-2. Due to the space limitation, only show representative tasks. Each leaf node includes at least one task and
We developed the taxonomy by referencing sessions from the INTERSPEECH conference and EDICS of IEEE SPS. Though not identical, they have very similar structure (please refer to Appendix foro comparison).

---

Published as a conference paper at ICLR 2025
may be further categorized into more fine-graines sub-domains, which are not shown here due to space constraints. For example, within 'Speaker & LLaguuge/Speakeer, we have two categories: "Speaker Characteristics' and 'Speaker Identification', each containing several tasks. Please refer to Appendix [ for the complete task list. We first categorize tasks into two primary fields: (I) speech and (II) music & audio, which are generally distinguished by the source of the sound. For instance, speech is produced by human vocal cords, music is created using instruments, and audio includes sounds generated by other creatures, materials, or natural phenomena. We then split each field into several domains based on the key attributes and challenges that the tasks within them present.
3.4.1 SPEECH
As Figure 2a shows, there are 8 domains within the speech field. Speech Recognition focuses on converting spoken language into text. This includes tackling various challenges like multilingual and code-switching ASR, as well as spontaneous ASR which is very different from audiobook- style ones. It also contains specialized tasks such as command recognition and keyword spotting. Speaker and Language addresses the analysis of speaker characteristics and languages, covering tasks such as speaker verification, diarization, and language identification. Spoken Language Un- derstanding deals with understanding and analysis of the content and semantics of spoken language. It covers tasks like sentiment analysis and speech translation. Phoneties, Phonology, and Prosody focuses on the sound structure of speech, including phoneme recogmition, pronunciation evaluation, and prosodic features like stress and accent classification. Paralinguistics explores non-verbal pects of speech, such as emotion recognition and vocal event detection, which can capture nuances like screaming or coughing. Speech Enhancement aims to improve speech quality by detecting and mitigating noise, reverberation, and other degradations, but currently, we only have understanding tasks. Speech, Voice, and Hearing Disorders is dedicated to identifying and classifying disorders such as stuttering and so on. Safety and Security focuses on detecting synthetic or manipulated speech, addressing tasks like spoof detection and recognizing deepfake voices.
3.4.2 AUDIO MUSIC
The audio and music field includes a wide range of tasks that focus on various attributes of sound beyond speech, such as musical elements, environmental sounds, and advanced sound analysis. This field is divided into 9 domains, each addressing a specific aspect of audio or music process- ing. Music Classification tasks focus on categorizing musical elements such as instruments, genres, and emotions, providing foundation for recognizing and analyzing different types of musical con- tent. Pitch Analysis delves into identifying the pitch and harmony within music, including tasks like pitch extraction, chord classification, and key detection. Rhythm Analysis involves tasks such as beat tracking, which is critical for understanding the temporal structure of music. In Singing Analysis, tasks address both lyric recognition and translation, as well as the classification of vocal techniques used in singing. Quality Assessment evaluates the perceived quality of singing, includ- ing automated predictions of Mean Opinion Scores (MOS). The Sound Event domain is broader, covering various sound sources such as animals, the environment, and human activities. Tasks range from animal sound classification emergency traffic detection and even advanced tasks like multi- channel sound event understanding. Safety domain includes detecting deepfakes in singing voices and identifying manipulated audio files, ensuring sound authenticity and integrity. Spatial Audio covers all tasks related to understanding spatial information, such as estimating the distance or posi- tion of sounds in the real world. Finally, Signal Characteristics Analysis domain addresses general signal-level characteristics of audio, including sound effect detection, and duration prediction.
3.4.3 CORE TASKS
While our task taxonomy provides a comprehensive, hierarchical, and systematic approach for benchmark users to easily get started with Dynamic-SUPERB Phase-2, evaluating all tasks with limited resources remains a challenge. To address this, alongside the task taxonomy, we have se- lected core tasks for speech, music, and audio. These core tasks are reduced subsets of essential tasks that have been widely used or studied within the research community. We include tasks from three popular benchmarks: SUPERB (speech), MARBLE (music), and HEAR (audio). The three benchmarks were originally designed for evaluating encoders, not for an instruction instruction-folloving uni- versal model, so modifications are required. We crafted instructions for each task, reduced the data

---

Published as a conference paper at ICLR 2025
size, and reformulated them into the Dynamic-SUPERI format. Besides, we replaced the datasets in some tasks to address licensing issues. Since the core task set is much smaller than the full bench- mark, researchers can more efficiently evaluate their models across a reasonable range of domains.
4 EXPERIMENTAL SETTINGS
4.1 MODELS
We evaluated several publicly-availabe models on Dynamic-SUPERB Phase-2: SALMONN, LTU- AS, Owen-Audio-Ca Chat.
WavLLLM, MU-LLaMA (Liu et al. 2024). GAMA (Ghosh et al. (024). SALMONN is further categorized into 7B and 13B versions based on the size of its LLM component. All of these models are publicly available, and we utilized their official implementations for inference without any modifications. The first five models are instruction-basee based speech models, each also trained with audio understanding capabilities. However, as they were not trained on music data, do not expect them to perform well on music-related tasks. Hence, we further included MU-LLaMA and GAMA, MU-LLaMA is specifically designed for various music-underssanding tasks, and GAMA is trained with both general audio data and mu- sic corpus. Similar to the speech models. MU-LLaMA and GAMA adopt an LLM-based framework. They use a music audio encoder, such as MERT (LI et al. 2024 or Q-former, to convert music or audio into features, which the LLM then uses as prompts for subsequent reasoning. The sam- pling strategies used for generating outputs from the LLMs were retained as defined in their official implementations. Finally, we implemented cascaded system baseline called Whisper-LLaMe The system first transcribes audio using Whisper-v3-large and then LLaMA3.1-8D processes the transcriptions to perform various tasks based on the provided instructions.
When evaluating models on these diverse tasks, several challenges inevitably arise. One challenge was testing the baseline models on tasks involving multiple audio inputs. For example, in speaker verification, a model determines whether two utterances are produced by the same speaker. Among the evaluated models, only Owen-Audi and Qwen2-Audin provide interfaces that allow inputting multiple audio files. For the remaining models, we concatenated all audio files, separated by 0.5 seconds of silence, and used the concatenated file as input.
Last, different models have their own maximum supported audio durations. Models using Whisper to encode speech face inherent limitation: Whisper can process audio files of up to 30 seconds, and some models are restricted to handling even shorter durations. Consequently, we retained the accommodate longer audio
original model settings and did not modify the model architecture clips. A preliminary analysis of all audio (Appendix G) in Dynamic-SUPERB Phase-2 shows that only small proportion (around 6.57%) exceeds 30 seconds in length. Hence, we believe our settings do not largely impact the evaluation results and ensure reasonable inference efficiency.
4.2 EVALUATION METRICS
The outputs of universal speech models are natural language sentences, making it difficult to assess their correctness using conventional metries. For classification tasks, such as emotion recognition, a task-specific model outputs a label from predefined emotions in the dataset, while a universal model generates sentences like "The speaker sounds happy." In this case, we can easily assess the correctness of the former by comparing labels, but this approach does not apply to the latter. Similarly, in regression tasks, a natural language response like "Using MOS scoring criteria, give the audio a score of 3 out of 5" makes it difficult to directly use metrics such as mean square error.
In evaluation, we categorize tasks into three types: (1) classification, (2) regression, and (3) se- quence generation. For each type, we use different pipelines to evaluate the models' outputs. For classification tasks, we utilize external LLMs (GPT-40) as referees, with the temperature set to be zero for evaluation consistency, to evaluate whether the outputs from speech music model match the ground truth. This approach has been widely adopted in the NLP community (Wang et al. 2023; Liu et al.) 2023b) Chiang & Lee (023), and extend its application speech search. We design a prompt that includes the task instructions, the model's output to be evaluated,
TTo avoid ambiguity, unless specified otherwise, the terms "output" or "model output" in this section refer the results generated by the models in Sec. 4.1
7

---

Published as a conference paper at ICLR 2025
and the corresponding ground truth. The LLM judge gets this prompt and determines if the output aligns with the ground truth using a chain-of- reasoning process. By leveraging the strong cntruuctionfoollwwinn capabilities of these LLMs, constrain them to provide the final decision in a fixed format, allowing us to extract the answer using simple methods such as regular expressions We then define accuracy as the percentage of outputs that the LLM judge considers aligned with the ground truth. Please refer to Appendix [ for details on the prompts and the alignment between LLMs and humans. Importantly, although we used GPT-40 for evaluation in the main text, to support reproducibility we have also included comprehensive evaluation results using the open-source LLM (LLaMA3. 1- -b-IInntrucr in Appendix C
For regression tasks, the above method cannot be applied because performance is not assessed us- ing hard labels. Instead, we use L.MSs (GPT-4o) as a post-processor to transform natural language responses into format compatible with the original metrics used in these tasks. Specifically, we developed a prompt that directs the LLMS to extract essential information from the output and con- vert it into the same format as the ground truth. If the output lacks correct or relevant information that can be converted to match the ground truth format, the LLM returns "N/A" to mark it as invalid. Since conventional metrics cannot accommodate "N/A" and setting default value is unreasonable due to the variability of metrics, we also calculate the N/A rate for each regression task, defined as the percentage of invalid outputs within a task. A higher N/A rate indicates that the model struggles with following instructions, which may indirectly affect its performance on the task.
For sequence generation tasks such as ASR and captioning, we apply their original metries directly to the raw outputs from baseline models. This is because identifying redundant prefixes or sections unrelated the task objectives is highly challenging in sequence generation, even with human in- volvement. Besides, our review of the original evaluation results reported by these baseline models revealed no explicit mention of post-pros poot-processing procedures. Therefore, we base our evaluations solely on the unprocessed outputs, ensuring consistency and objectivity across all models.
5 RESULTS
Audio & Music
Harmany & Pitch-
40.52
3.21
3.02
9.38
2.46
0.53
3,13
0.22
0.50
Music Classifiattion
1.09
13:28
16.73
15.58
5.73
8.43
14.22
4.79
0.00
-15
Quality
Rlythi Analysis
0.31
.077
0.16
0.44
0.43
0.13
0.18
0.35
0.00
Sanety
0.87
1.35
1.03
1.61
3.03
1.43
1.01
1.32
0.00
10
Signal Character Analysis
0.92
0.30
Singing Anslysis
-0.75
-0.13
-0.3
0.26
-0.76
-0.61
-1.37
40.88
0.00
Agcroge Retedvv scoe
Sound Event
0.68
6.00
0.98
Spartal
0.90
-0.31
0:25
0.00
0.699
0,50
Speech
Paralinguisikss
4.52
2.89
2.81
5.35
1.62
1.73
0.63
0.26
0.00
Phonetics, Phonology. Prosedty
-.33
0.21
21.03
Safety Security
-0.34
-0.08
.0.02
2.08
-0.05
4.12
-0.15
0.66
0.50
Speaker Language
8.37
8.32
9:28
15.05
3.64
9.48
6.40
10:40
0.00
-10
Speech
Speech Recognition
3.24
.198
1.29
0.36
0.59
4.93
4.82
1.80
0.00
Sceech, Voice, Hearing Disorder
0.27
0.29
0.01
014
3.16
0.01
0.24
4,44
0.00
Sookin Language Understanding
0.78
0.
0.07
0.14
0.05
4.09
0.55
-20
GAMA-IT
Mu AuAaa
Owee Austen-Chhl Chie
Qenest Aealio Instrect
SALMONN- 176
SAUAOW 78
unAss
whispestate LaMk
Figure 3: Domain-level relative-score-base performance comparison across different models. We exclude tasks where the models have a 100% NA rate or where Whisper-LLaMA scores zero, as these make it impossible to average their scores with other tasks.
We have tried to ask the speech universal models to directly output their decisions in fixed format, but they do not always follow the instructions. Therefore, the evaluation pipeline descrihed here is necessary.
8

---

Published as a conference paper at ICLR 2025
5.1 DOMAIN-LEVEL PERFORMANCE COMPARISON TAXONOMY
Here, we analyze the results based on the taxonomy. Due to space limitations, we can only report and compare the results of different models at the domain level in this subsection. Since different metrics are used across tasks, we adopted a relative-score-basedd method to summarize task performance. For each task, we calculated the relative improvement of cach model compared to the cascaded system baseline (Whisper + LLaMA) and then obtained the domain-level scores by averaging all its improvements across the tasks within each domain. For regression tasks, we introduced the N/A rate measure how well model meets the task requirements. The original task metric was only computed on instances that followed the task format (after post-processing by the LLM). To account for whether a model can follow instructions, we incorporated the N/A rate into the reported scores. Accordingly, we calculated the improvement based on scaled values. For metrics where a higher value indicates better performance (such as the Fl score), we multiplied the metric value by (1 N/A rate), Conversely, for metrics where a lower value indicates better performance (such as word error rate), we divided the metric value by (1 N/A rate).
Figure presents a domain-level comparison of relative scores across different models. Whisper- LLaMA consistently has zero scores throughout all domains, serving as the reference baseline for evaluating other models. In speech domains, no model outperforms Whisper-LLaMA in speech recognition and spoken language understanding. This shows that using ASR remains a strong baseline for language understanding, as text more explicitly represents semantic information than speech. However. Whisper-LLaMA lags behind SALMOŇN in phoneme recognition, a task for which SALMONN is explicitly trained, indicating that task-specific training still provides superior performance in specialized domains.
Conversely, in the speaker and paralinguistics domains, all models surpass the baseline. This im- provement occurs because the ASR process in the cascaded system tends to discard critical informa- tion from the speech signal, such as speaker characteristics, pitch, and emotion. In contrast, other models utilize soft representations that retain more speech information, giving them the potential to learn beyond ASR, and some of these models were also trained perform specific tasks within cer- tain domains F Additionally, music-specific large language models like GAMA-IT and Mu-LLaMA perform poorly on speech recognition and understanding tasks, which is expected since they were primarily designed for music understanding. Nevertheless, they surprisingly achieved scores com- parable to the speech models in speaker and paralinguistics.
Turning to audio and music domains, where ASR models cannot transcribe non-speech informa- tion into text, most models outperform the baseline in several areas, such as music classification and sound event detection. Surprisingly, spoken language models primarily trained on speech data (Qwen models, SALMONN, WavLLM WavLLMyooupperfn outperform the two music models (GAMA-IT and Mu- LLaMA) in various music-related domains such as harmony and pitch, music classification, and rhythm analysis. We speculate that training on diverse data, even with significant differences in signal-level characteristics, enhances performance across domains, emphasizing the importance of developing unified models for speech, music, and general audio processing.
Lastly, we observe some outliers with significantly higher or lower values in the figure, typically resulting from unbounded evaluation metrics. For instance, in the phonetics and prosody domain, WavLLM and LTU-AS exhibit poor scores due their erroneous outputs in phone/phoneme seg- ment counting tasks, predicting thousands of segments in utterances lasting only seconds and result- ing in an excessively high mean square error. While the relative score-based approach effectively summarizes model performance, domain-level scores can be distorted by specific tasks within do- main. Thus, we encourage researchers aiming to develop model performance in specific domains to comprehensively report task- scores to more accurately reflect their models" capabilities.
5.2 CORE TASKS RESULTS
We tested the performance of the models on core tasks that are widely studied and commonly used. Table 2 presents the detailed results of the models on SUPERB tasks, displaying each task's perfor-
'Athhough the models have been trained on some tasks the same domains, the training tasks of these models generally do not use the same instructions as Dynamic-SUPERB and do not offer as broad coverage. Owen and MALMONN are trained mixture of speech (dominant component), music, and audio data.

---

Published as a conference paper at ICLR 2025
Table Evaluation on SUPERB tasks. A **" indicates that the metric scaled with NA rate, and means a model has a 100% NA rate.
PR
KS
IC
ER
ASR
QbF
SV
SD
PER 4
Ace
Acc
f
WER
Acc f
F
CFR*
Acc
DER 1
Whisper-T.
LMM LaMA
100.1
36.3
36.5
12.5
34.0
46.5
53.6
ST.3
45.0
T068.7
SALMONN-7B
25.4
30.5
21.0
12.1
15.1
49.0
61.6
61.5
45.0
SALMONN- 13B
24.6
2.0
12.5
2.8
51.5
29.6
106.3
51.0
Qwen-Audio-CC Audio-Chat
100.6
60.5
26.5
70.4
69.4
48.0
42.2
90.1
49.0
8852.2
Qwen2-Audio-7B A--io-B-----
101.0
47.0
26.0
75.8
36.7
53.5
42.4
55.7
51.0
4664,4
WavLLM
100.0
43.0
28.0
12.1
6.9
49.5
50.8
84,9
49.0
3621.2
LTU-AS
102.8
1.0
0.1
25.8
96.3
45.0
10.2
559.3
44.0
14923.3
GAMA-IT
100.4
2.0
1.0
0.0
116.7
2.5
2.7
4750.0
45.0
187.6
Mu-LLaMM
110.3
4.0
3.5
12.9
103.7
51.0
1.0
15869.6
44.0
PR: Phoneme Recognition, KS; Keyword Spotting, IC: Intent Classification, ER: Emotion Recognition, QbE:
Query-by- by-Exampee, SF: Slot Filling, SV: Speaker Verification, SD: Speaker Diarization
mance in its own metric rather than relative scores. Please refer to Appendix for results on HEAR and MARBLE, It is worth noting that these tasks have been adapted to the Dynamic-SUPERB framework. Therefore, the results are not directly comparable to previous studies that tested on the original SUPERB. However, they indicate current performance levels on these tasks.
No single model excels across all tasks. In phoneme recognition (PR), the SALMONN models were the only ones to achieve a relatively lower phoneme error rate (PER) compared to the others. For keyword spotting (KS), Owen-Audio-GGn a random guess (50%). Whisper-LLate LLaMA surpassed all other models intent classification (IC). Regarding emotion recognition (ER), Qwen-Audio-Chaa and -un-n---Audo--7--nsttte stood out, with the latter even achieving about 75% accuracy. In ASR, SALMONN-13B and WavLLM are the only two models that achieved word eтror rate (WER) lower than 10%. Query-by-example (QbE) appears to be very challenging for all models, as none performed better than a random guess (50%). Slot-filling (SF) is evaluated using two metries: Fl for slot type and CER for slot value. While Whisper-LLaMe and SALMONN-7B are the top two in this task, their results are still far from ideal. In speaker verification (SV), all models performed badly, as even the best ones achieved results close to random guessing (50%). In speaker diarization (SD), results were similarly poor, with some even reaching a 100% N/A rate. Comparing Figure 3 and Table 2 we observed that per- formance on core tasks sometimes deviates from trends observed at the domain level. For example, although SALMONN-13B outperforms other models ASR, it lags behind Whisper-LLaMA speech recognition domain, revealing its limited capabilities and highlighting the need to enhance its generalizability. In summary, among tested models, some excel in specific tasks, while others perform poorly across the board, and none dominate across all tasks. We believe these findings provide valuable insights for researchers aiming to improve models, starting from core tasks and progressively tackling each domain in the benchmark for broader applicability.
was the only model to perform slightly better than
the
CONCLUSIONS
This paper presents Dynamic-SUPERB Phase-2, the largest henchmark for evaluating instruction- based universal spoken language models, building upon collaborative efforts across the research community. Dynamic-SUPERB covers a wide range of diverse tasks and offers a fine-grained task taxonomy. The recent models show good performance on specific tasks but poor generalization across common tasks like those in SUPERB, highlighting the need for further research on universal models. All materials are made openly available to facilitate reproduction and benchmarking. We sincerely invite researchers to join our vibrant community and collaborate to advance the field.
Although Synamic-SUPERY Phase-2 is the largest and most comprehensive bench- mark, we acknowledge its limitations. It lacks comprehensive speech-generation tasks, as Phase-2 focused on understanding tasks due to the few universal generation models. Despite our efforts to develop the task taxonomy scientifically, new domains may emerge the benchmark grows, and tasks can be categorized in various ways. While our automatic evaluation pipeline using LLMs correlates well with human evaluations (Appendix E) for current tasks, it may not generalize to all future tasks. Upholding the core spirit of the Caaacc-SPPERR project, we are striving to address these issues and enhance the benchmark for the next phase.
Limitations:
10

---

Published as a conference paper at ICLR 2025
7 ETHICS STATEMENT
Dynamic-SUPERB Phase-2 includes several datasets, and we asked contributors to describe how they used them. Most contributors derived task data by uniformly sampling from the origi- nal datasets without applying any special processing. Therefore, Dynamic-SPPERB Phase-2 in- evitably inherits the biases present in these datasets across tasks. Additionally, we asked all contributors to provide the necessary license information to ensure that the benchmark properly. More specifically, all task data in Dynamic- -UPERB are derived from datasets with licenses that permit remixing and redistribution. As expect the benchmark to grow dynamically in the future, we maintain a complete list of dataset licenses on our offi- cial GitHub page (nttps: /gtthubb com/ dynamic superb/ superb/dynami dynamic c corcccssseeei superb/blob/ mmaim /ocss/ aeloo///aaaseeeeeeeeeeeeeeeeeeee dataset__aaaiaaaeee license md), which will be updated as new tasks are introduced. If there are any changes to the dataset license the future, we may adjust the benchmark task accord-
include them in
ingly.
REPRODUCIBILITY STATEMENT
task data and the evaluation pipeline at htttps://githuu: com/ dynamic- superb/ /oyeami/dyymiicoomm ture to 0 to ensure a stable evaluation process. However, we acknowledge that GPT-4o's behavior may change with future updates. Therefore, using 1.LMA-3.1--0 1-8b-Instruce in Appendix (Table (). Additionally, in Appendix E we present a preliminary study using open-source LLMs for evaluation, demonstrating promising correlations with human annotations. We encourage researchers to use these open-source LLMSs when evaluating their own models to support reproducibility.
We open-source dynami c- superb For LLM-based evaluation, we set GPT-4o's tempera-
provide comprehensive set of evaluation results
REFERENCES
Librispeech word. http:://wwgggggaccccc/aaaeeeeeee word Accessed: 2024-10-02.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint axXiv:2303.08774 2023.
Eneko Agirre, Daniel Cer, Mona Diab. and Aitor Gonzalez-Agire, SemEval-2012 task 6: A pilot on semantic textual similarity. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret (eds.), *SEM 2012: The First Joint Conference putational Semantics Volume I: Proceedings of the main conference and the shared task, and Volume Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385 393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguis- 012-1000
Lexical and Com-
tics. URL https:://///// http:ts///claaaollloooooooooo
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301. 11325, 2023.
Afroz Ahamad, Ankit Anand, and Pranesh Bhargava. Accentdb: A database of non-native english accents to assist neural speech recognition. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 5353 5360, Marseille, France, May 2020. European Language Resources Association. htt.t//ooo//.....eeeeeerrroooooo
wwwp://wwwwe...ogg//aoooooooooo. 1ttp//2200.....11...111 1rec-1.
659
Anna Aljanaki, Frans Wiering, and Remco C. Veltkamp. Studying emotion induced by mu- sic through a crowdsourcing game. Information Processing & Management, 52(1):115-128 2016. ISSN 0306-4573. doi: oociine//rrrccee///////665513130 Isciencedirect com/ www.cccicciieeeeecc.coooeeocccce scts/cecccc/eeeeeeeeeee Sentiment in Social and Expressive Media.
URL ht : /www.
Emotion and

---

Published as a conference paper at ICLR 2025
Akshay Anantapadmanabhaa, Ashwin Bellur, and Hema A Murthy. Modal analysis and transcrip- tion of strokes of the mridangam using non non-negative matrix factorization. In 2013 IEEE interna- tional conference acoustics, speech and signal processing, pp. 181 181-185. IEEE, 2013.
Xavier Anguera, Luis-J. Rodriguez- Fubriguez-FFuetes,,
Fuentes, Andi Buzo, Florian Metze, Igor Szöke, and Mikel Pe- nagarikano. Quesst201:: Evaluating query-by-exampe speech search in a zero-resource setting with real-life queries. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5833-5837, 2015. doi: (0.110//CASS922155577999
Anthropic. Claude. https: hwwp::://www.aae httts://ww..mhhropic..com 2023. Large language model.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A. mastively-mulilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4218-4222, 2020.
Muhammad Asif, Muhammad Usaid, Munaf Rashid, Tabarka Rajab, Samreen Hussain, and Sarwar Wasi. Large-scale audio dataset for emergency vehicle sirens and road noises. Scientific data, (1):599, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Jon Barker, Shinji Watanabe, Emmanuel Vincent, and Jan Trmal. The Fifth 'CHiME' Speech Sep- aration and Recognition Challenge: Dataset, Task and Baselines. In Proc. Interspeech 2018, pp. 1561-1565, 2018.
doi: 10.21437//nterspecch200 1768.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A Spoken Language Understanding Resource Package. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Sören Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Robert Müller, Sebastian Lapuschkin, and Wojeiech Samek. Audiomnist: Exploring explainable artificial intelligence for audio analysis on simple benchmark. Journal of the Franklin Institute, 361 (1):4184 11:41884220 428,, 2024.
Jordan J Bird and Ahmad Lotfi Real-time time detection of ai- ai-generated speech for deepfake voice conversion. arXiv preprint 2rXiv::308. 12734, 2023.
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gard- ner, Rohan Taori, and Ludwig Schimdt. Visit-bench: benchmark for vision-language instruction following inspired by real-world use. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 26898-26922. 2023.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg- jamendo dataset for automatic music tagging. ICML, 2019.
Cano, N. Wack, and P. Herrera. ISMIR04 Genre Identification task dataset (1.0), 2018. URL https://doi org/10 2088//eendo..1... 1300
Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on doi: 10.1 N0.1009TAFFC..20422366
Affective Computing, 377)337,, 2014.
Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. The ami meet- ing corpus: A pre-announcement. In International workshop machine learning for multinodal interaction, pp. 28-39. Springer, 2005.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihal- cea, and Soujanya Poria. Towards multimodal sarcasm detection (an .Obviously perfect pa- per). In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4619-4629, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.118533v1PPPP 1/119- 1155. URL https: //acl antho ogy.oog/P19-1145
12

---

Published as a conference paper at ICLR 2025
CEEC. High school english listening exam. 11-08.
https://ww..ceeet
Accessed: 2024-
Liang-Yo Yu CHEN and Jyh-Shing Roger JANG. Stress detection of english words for capt sys- tem using word-length dependent gmm-based bayesian classifiers. Interdisciplinary Information Sciences, 18(2):65-70, 2012. doi: 20.4066/iss2012.65.
Cheng-Han Chiang and Hung-Yo Yi Lee. Can large language models an alternative to human eval- uations? In Proceedings of the 61st Annual Meeling of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pp. 15607-15631, 2023.
Eleanor Chodroff, Blaž Pažon, Annie Baker, and Steven Moran. Phonetic segmentation of the UCLA phonetics lab archive. Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 27724-12733 Torino, Italia, May 2024. ELRA and ICCL. URL https:tt//////c hatp::://aalaatthooogooo org/2024. wree-rreccee ma in. 1114
Soonbeom Choi, Won II Kim, Sae Byul Park, Sangeon Yong, and Juhan Nam. Children's song dataset for singing voice research. In The 21th International Society for Music Information Re- trieval Conference (ISMIR). International Society for Music Information Retrieval, 2020.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.099) 2023.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, al. Qwen2-audin technical report. arXiv preprint 10775 2024.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. Librimix: An open-source dataset for generalizable speech separation, 2020.
Maureen de Seyssel, Marvin Lavechin, Hadrien Titeux, Arthur Thomas, Gwendal Virlet, An- drea Santos Revilla, Guillaume Wisniewski, Bogdan Ludusan, and Emmanuel Dupoux. Prosaudit, prosodic benchmark for self-supervised speech models. INTERSPEECH 2023, pp. 2963- 2967, 2023. doi: 221443//mexspeech22233333
DeepContractor.
Musical
instrument
chord
classification, 2024.
URL
https: www. kaggle. com /wtp///aaaaastteee /deppcottaaattoo/ http:///tttttttteeettttttttttaaa
Accessed: 2024-10-01
Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. Fma: A dataset for music analysis. In 18th International Society for Music Information Retrieval Conference, 2017.
Héctor Delgado, Massimiliano Todisco, Md Sahidullah, Nicholas Evans, Tomi Kinnunen, Kong Aik Lee, and Junichi Yamagishi. Asvspoof 2017 version 2.0: meta-data analysis and baseline hancements. In The Speaker and Language Recognition Workshop (Odyssey 2018), pp. 296-303, 2018. doi: (0.21437/0dyssey20113
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning dataset. In ICASSP 2020 -202 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 736-740, 2020. doi: 10. 11119/AAA9944776222099999999
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprini arXiv:2407.22 /700072..78... 2024.
Ewan Dunbar, Mathieu Bernard, Nicolas Hamilakis, Tu Anh Nguyen, Maureen de Seyssel, Pa- tricia Rozé, Morgane Rivière, Eugene Kharitonov, and Emmanuel Dupoux. The Zero Re- source Speech Challenge 2021: Spoken Language Modelling. In INTERSPEECH 2021. doi: 10.21 2221443/nneespeech222---1
13

---

Published as a conference paper at ICLR 2025
Ewan Dunbar, Nicolas Hamilakis, and Emmanuel Dupoux. Self-supervised language learning from raw audio: Lessons from the zero resource speech challenge. IEEE Journal of Selected Topics in Signal Processing, (666):1211-1228 2022.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In International Conference on Machine Learning, pp. 1068-1077, PMLR, 2017.
Solène Evain, Ha Nguyen, Hang Le, Marcely Zanon Boito, Salima Mdhaffar, Sina Alisamir, Ziyi Tong, Natalia Tomashenko, Marco Dinarelli, Titouan Parcollet, Alexandre Allauzen, Yannick Estève, Benjamin Lecouteux, François Portet, Solange Rossato, Fabien Ringeval, Didier Schwab, and Laurent Besacier. Lebenchmark: A reproducible framework for assessing self-supervised representation learning from speech. Interspeech 2021, pp. 1439-1443, 2021. doi: 10.21437/ Interspeech.2021-555
Francesco Foscarin, Andrew McLeod, Philippe Rigaux, Florent Jacquemard, and Masahiko Sakai. ASAP: dataset of aligned scores and performances for piano transcription. In International Society for Music Information Retrieval Conference (ISMIR), pp. 534-541, 2020.
Joel Frank and Lea Schönherr. Wavefake: A data set facilitate audio deepfake detection. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Daniel Galvez, Greg Diamos, Juan Torres, Keith Achom, Juan Cerón, Anjali Gopi, David Kanter, Max Lam, Mark Mazumder, and Vijay Janapa Reddi. The people's 's speech:
A large-scale diverse english speech recognition dataset for commercial usage. In
Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Process- ing Systems Track Datasets and Benchmarks, volume 1, 2021. URL https:
fiirapprrfiieseppppeeeeee 2021 200/7711172222220011166653333333
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sak- shi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406. 1768, 2024.
Mandar Gogate, Kia Dashtipour, Ahsan Adeel, and Amir Hussain. Cochleanet: A robust language- independent audio-visual model for real-time speech enhancement. Information Fusion, 63:273- 285, 2020.
Ary I. Goldberger, Luis AN Amaral, Leon Glass, Jeffrey Hausdorff, Plamen Ch Ivanov, Roger Mark, Joseph E Mietus, George B Moody, Chung Kang Peng. and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23):e215-e22 2000.
Yuan Gong, Jin Yu, and James Glass. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 151-155, 2022. doi: AAL110//CAASSP492222000.2.771
Yuan Gong, Alexander Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU). pp. 1-8. IEEE, 2023.
Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to- speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel. and Douglas Eck. Enabling factorized piano music modeling and generation with the MAESTRO dataset. In International Conference on Learning Represen- fations, 2019. ht.httts::11 / wttp://wwwwweetetwteeeeeee http://www....eeeeeeee/////ee//a
14

---

Published as a conference paper at ICLR 2025
Addison Howard et al. Comelll birdcall identification, 2020. URL https://kaggle... hammppttiiiiossbbbrrdoo-eeeeeeeo
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, al. Wavllm: Towards robust and adaptive speech large language model. arXiv preprint M.XXx:24440006555 2024.
Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi- Hsiao, Chun- Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, al. Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuninn benchmark for speech. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12136-12140, IEEE, 2024a.
Kuan-Po Huang, Chih-Kai Yang, YuaK Fu, Ewan Dunbar, and Hung-yi Lee. Zero resource code-switched speech benchmark using speech utterance pairs for multiple spoken languages, In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10006-10010. IEEE, 2024b.
Eric Humphrey, Simon Durand, and Brian McFee. Openmic-2011 An open data-set for multiple instrument recognition. In ISMIR, pp. 438 438-444, 2018.
IEEE DCASE 2016 Challenge. IEEE DCASE 2016 Challenge. http: /wwp///wwwww./// www.//www.c.ooooo.ccooooooo.oo.. wwt...tu.t.... fi/ /gn//arg//////e.g dcase2016/ 2016. Accessed: 2025-03-04.
Keith Ito and Linda Johnson. The lj speech dataset.
bttps://keithito.co com/
LJ Swwpppppeeeeaaeseeeeeee 2017.
Tahir Javed, Kaushal Bhogale, Abhigyan Raman, Pratyush Kumar, Anoop Kunchukuttan, and Mitesh M Khapra. Indicsuperb: A speech processing universal performance benchmark for in- dian languages. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12942-12950, 2023.
Denys Katerenchuk, David Guy Brizan, and Andrew Rosenberg. Interpersonal relationship labels for the CALLHOME corpus. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceed- ings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018 European Language Resources Association (ELRA). URL https://aclantinn
wwg. 1592
Tyler Kendall and Charlie Farrington. The corpus of regional african american language, 2023. URL https:://di.....
7264/1 ad5 6t35
Bongjun Kim, Madhav Ghei, Bryan Pardo, and Zhiyao Duan. Vocal imitation set: dataset of vocally imitated sound events using the audioset ontology. In DCASE, pp. 148-152, 2018
Peter Knees, Ángel Faraldo Pérez, Herrera Boyer. Richard Vogl, Sebastian Böck, Florian Hörschläger, Mickael Le Goff, et al. Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections. In Proceedings of the 16th Interna- tional Society for Music Information Retrieval Conference (ISMIR); 2015 Oct 26-30; Málaga, Spain.(Malagaa]: International Society for Music Information Retrieval, 2015. p. 364-70. Inter- national Society for Music Information Retrieval (ISMIR), 2015.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L. Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. $220-5224 2017. doi: 10.109/CCASS222077.9933
15

---

Published as a conference paper at ICLR 2025
Edith Law, Kris West, Michael I Mandel, Mert Bay. and J Stephen Downie. Evaluation of algorithms using games: The case of music tagging. In ISMIR, pp. 387-392. Citeseer, 2009.
Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jeffrey P. Bigham. Sep-28k: A dataset for stuttering event detection from podeasts with people who stutter. In ICASSP 202/ 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 6798-6802. 2021.
doi: 10. 10.1109//CASSP397288.20 20w.99977222222111133555
Keon Lee, Kyumin Park, and Daeyoung Kim. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Yizhi LI, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu. MERT: Acoustic music understanding model with large-scale self-supervised training. In The Twelfth ternational Conference Learning Representations, 2024. URL pettps://openreettt
net/Eorum? 33229M
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05460 2023b.
Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko. Paralinguistic--enhanedd large language modeling of spoken dialogue. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10316-10322 IEEE, 2024.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: text-to-audio generation with latent diffusion models. In Proceed- ings of the 40th International Conference on Machine Learning, pp. 21450-21474, 2023a.
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding llama: Advancing text-to-music generation with question answering and captioning. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 286-290. IEBE, 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522222 2023b.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)x0016331, 2018
Holy Lovenia, Samuel Cahyawijaya, Genta Indra Winata, Peng Xu, Xu Yan, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham Barezi, et al. Ascend: A spontaneous thinese-english dataset for code-switching in multi-turn conversation. In Proceedings of the 13th Language Resources and Evaluation Conference (LREC), 2022.
Ken MacLean.
Voxforge, 2018.
Ken MacLeaan[Ontinne]
Available:
http://wwww.oodoorg..orgmmmmmm
em 2012].
Gallil Maimon, Amit Roth, and Yossi Adi. A suite for acoustic language model evaluation. arXiv preprint arXiv:2409.074:3 2024.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACI., 2022.
16

---

Published as a conference paper at ICLR 2025
National Chengchi University. The necu (national chengehi university) corpus of spoken taiwan mandarin, n.d. URL 2024-10-01J.
[Accessed:
Inês Nolasco, Alessandro Terenzi, Stefania Cecchi, Simone Orcioni, Helen L. Bear, and Emmanouil Benetos. Audio-based identification of beehive states. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9256-8260 IEEE, 2019.
Lara Orlandic, Tomas Teijeiro, and David Atienza. The coughvid crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms. Scientific Data, 8(1):156, 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Yagya Raj Pandeya and Joonwhoan Lee. Domestic cat sound classification using transfer learning. International Journal of Fuzzy Logic and Intelligent Systems, 18(2):154-166 2018.
Yagya Raj Pandeya, Dongwhoon Kim, and Joonwhoan Lee. Domestic cat sound classification using learned features from deep neural nets. Applied Sciences, 8(10):1949, 2018.
Titouan Parcollet, Ha Nguyen, Solene Evain, Marcely Zanon Boito, Adrien Pupier, Salima Mdhaf- far, Hang Le, Sina Alisamir, Natalia Tomashenko, Marco Dinarelli, al. Lebenchmark 2.0: standardized, replicable and enhanced framework for sulf-suuer representations of french speech. arXiv preprint wrXiv:2309.05417 2023.
Karol Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party calli--prtt dataset for emotion recognition in conversations. In Anna Korhonen, David Traum, and Lluís Marquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 527-536, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1118665/11//1911000 URL https:
T/Acllarcholoory.ozr///1111100
Sathvik Prasad and Bradley Reaves. Robocall audio from the ftc's project point of no entry.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. In Interspeech 2020, pp.
2020.
doi: 10.2 (0.21437//nerspeech.D000 wwweechhh.2222200 2020-2826.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-28518 PMLR, 2023.
Zafar Rafii, Antoine Liutkus, Fabian Robber Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. MUSDBI8-HQ an uncompressed version of musdb18, December 2019. URL htLp:tps::/// //doi. org/10. 5281 zencdo. 3 3338373
Muhammad Mamunur Rashid, Guiqing Li, and Chengrui Du. Nonspeech7k dataset: Classification and analysis of human non-speech sound. IET Signal Processing, (7(6)ee12333 2023.
Matthew Rice, Christian J. Steinmetz, George Fazekas, and Joshua D. Reiss. General purpose audio effect removal, In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2023.
J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In 22nd ACM International Conference on Multimedia (ACM-MM' 14), pp. 1041-1044, Orlando, FL, USA, Nov. 2014.
Miguel Sarabia, Elena Menyaylenko, Alessandro Toso, Skyler Seto, Zakaria Aldeneh, Shadi Pirhos- seinloo, Luca Zappella, Barry-John Theobald, Nicholas Apostoloff, and Jonathan Sheaffer. Spa- tial librispeech: An augmented dataset for spatial audio learning. In INTERSPEECH 2023, pp- wtt.eech...222111..
3724-3728, 2023. doi: 1..2143//nte
17

---

Published as a conference paper at ICLR 2025
Jiatong Shi, Dan Berrebbi, William Chen, En-Pei Hu, Wei-Ping Huang, Ho-Lam Chung. Xuankai Chang, Shang- Wen Li, Abdelrahman Mohamed, Hung yi Lee, and Shinji Watanabe. MI-superb: Multilingual speech universal performance benchmark. In INTERSPEECH 2023, pp. 884-888, 2023. doi: 10.21445//meerspeech.2222 2023ch.322211666
Jiatong Shi, Yueqian Lin, Xinyi Bai, Keyi Zhang, Yuning Wu, Yuxun Tang, Yifeng Yu, Qin Jin, and Shinji Watanabe. Singing voice data scaling-up: An introduction ace-openepop and ace-kising. arXiv preprint arXiv:2401.17661 2024a.
Jiatong Shi, Shih-Heng Wang, William Chen, Martijn Bartelds, Vanya Bannihatti Kumar, Jinchuan Tian, Xuankai Chang. Dan Jurafsky, Karen Livescu, Hung yi Lee, and Shinji Watanabe. MI- superb 2.0: Benchmarking multilingual speech models across modeling constraints, languages, and datasets. In Interspeech 2024, pp. 1230-1234, 2024b. doi: 10221437/lmeexpeecch.02. 22448.
Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al. Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal anno- tations of sound events. Advances Neural Information Processing Systems, 36, 2024.
Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu Han. Slue: New benchmark tasks for spoken language understanding evaluation natural speech. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7927-7931. IEEE, 2022.
Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan S Sharma, Wei-Lun Wu, Hung-Y: Yi Lee, Karen Livescu, and Shinji Watanabe. Slue phase-2: A benchmark suite of diverse spoken language understanding tasks. Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8906-8937, 2023.
David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint XX1:1550000844 2015.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 24.6:2 2022.
Fabian-Rober Stöter, Soumitro Chakrabarty, Emanuël Habets, and Bernd Edler. LibriCount, dataset for speaker count estimation, 2018. URL
zenooo.
1216072
Dan Stowell, Michael D Wood, Hanna Pamula, Yannis Stylianou, and Hervé Glotin. Automatic acoustic detection of birds through deep learning: the first bird audio detection challenge. Methods in Ecology and Evolution, 10(3):368-380, 2019.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, MA Zejun, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024a.
Yuxun Tang. Jiatong Shi, Yuning Wu, and Qin Jin. Singmos: An extensive open-source singing voice dataset for mos prediction. arXiv preprint arXiv:2406. 10911, 2024b.
Mi Tian, Ajay Srinivasamurthy, Mark Sandler, and Xavier Serra. A study of instrument-wise on- detection in beijing opera percussion ensembles. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pp. 2159-2163. IEEE, 2014.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wrXiv:2307.09288. 2023b.
18

---

Published as a conference paper at ICLR 2025
Sheng-Che Chi Tsai. English lexical stress detection and sentence-basel intonation assessment based on contour shape description. Master's thesis, National Taiwan University. Taipei, Taiwan, 2015.
Available at
jjzzak
Ching Yu Tseng. An initial study on stress detection for spoken english. Master's thesis, National Tsing Hua University, Hsinchu, Taiwan, 2008. Available at https:: htttss//hdl.. hd1 handle net/
11296/ [1296/gr88s2]
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Bjöm W Schuller, Christian J Stein- metz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al. Hear: Holistic evaluation of audio representations. In NeurlPS 202/ Competitions and Demonstrations Track, pp. 125-145. PMLR, 2022.
Nicolas Turpault, Romain Serizel, Ankit Parag Shah, and Justin Salamon. Sound event detection in domestic environments with weakly labeled data and soundscape synthesis. In Workshop on Detection and Classification of Acoustic Scenes and Events, 2019.
Jörgen Valk and Tanel Alumäe, Voxlingua107: a dataset for spoken language recognition. In 202/
IEEE Spoken Language Technology Workshop (SLT), pp. 652-658. IEEE, 2021.
Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Semantic object prediction and spatial sound super-resolution with binaural sounds. In European Conference Computer Vision, pp- 638-655. Springer, 2020.
Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. Covost 2 and massively multilingual speech translation. Interspeech 2021, pp. 2247-2251, 2021. doi: 10.21457/mterspecch.000- 202222222
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. In Proceedings of EMNLP Workshop, pp. 2023.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaci, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Supermmtuuainnsmrtoosseennnliiit via declarative instructions on 1600+ tasks. In EMNLP, 2022.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. preprint 1804v:1804.0320 032009 2018.
arXiv
Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj, et al. Chime-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. In CHiME 2020-6th International Workshop on Speech Processing Everyday Environments, 2020.
WavSource. Sound effects library. https: http:://wwwwwwwwwore..... com n.d. Accessed: 2024-10- 01.
2hats2000. Human screaming detection dataset, 2023. URL https::/www.kaqqle com/ datasetstsssssse hatlaa///00//naa---eerrrrr-reeee cessed: 2024-10-02.
Ac-
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. In Interspeech 2019, pp. 1368-1372, 2019. doi: 1021497//neesjeec..01999 2019-99-ccc...122....
Julia Wilkins, Prem Seetharaman, Alison Wahl, and Bryan Pardo. Vocalset: A singing voice dataset. In ISMIR, pp. 468 474, 2018.
Zhizheng WU, Tomi Kinnunen, Nicholas Evans, Junichi Yamagishi, Cemal Hanilçi, Md. Sahidullah, and Aleksandr Sizov. Asvspoof 2015: the first automatic speaker verification spoofing and coun- termeasures challenge. In Interspeech 2015, pp. 2037-2041. 2015, doi: 10.21437/nterspeecch. 2015-462.
19

---

Published as a conference paper at ICLR 2025
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92). sound, 2019. URL
httts::/doi..rr
7488/48888 ds /265
Chih-Kai Yang, Kuan-Po Huang, Ke-Han Lu, Chun-Yi Kuan, Cui-Yuan Hsiao, and Hung-yi Lee. Investigating zero-shot generalizability on mandarin-english code-switched asr and speech-to-text translation of recent foundation models with sulf-superiision and weak supervision. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp. 540-544. IEFE, 2024a.
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. Air-bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.077299 2024b.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-War Wen Li, Shinji Watanabe, Abdelrah- man Mohamed, and Hung-yi Lee. Superb: Speech processing universal performance benchmark. In Interspeech 2021, pp. 194--198. 2021. doi: 1122445//neerspecch.205
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7163-7189, 2021.
Ruibin Yuan, Yinghao Ma Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, et al. Marble: Music audio representation benchmark for universal evaluation. Advances in Neural Information Processing Systems, 36:39626-3968 39647, 2023.
Yongyi Zang. Jiatong Shi, You Zhang, Ryuichi Yamamoto, Jionghao Han, Yuxun Tang, Shengyuan Xu, Wenxiao Zhao, Jing Guo, Tomoki Toda, and Zhiyao Duan. Cirsvdd: A benchmark dataset and baseline analysis for controlled singing voice deepfake detection. In Interspeech 2024, pp.
4783-4787, 2024. doi: 222443//miersppecch222--222
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. In Interspeech 2019, pp- 1526-1530, 2019, doi: 10.21437//teesppechh2111 2019999242...112441
Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey, and Yujun Wang. speechocean762: An open-source non-native english speech corpus for pronunciation assessment. In Interspeech 2021, pp. 3710-3714, 2021. doi: 10.21437/ Sttesppeech.221-12555
Lichao Zhang, Ruiqi Li, Shoutong Wang, Liqun Deng, Jinglin Liu, Yi Ren, Jinzheng He. Rongjie Huang. Jieming Zhu, Xiao Chen, and Zhou Zhao. M4singer: A multi-style, multi-singer and mu- sical score provided mandarin singing corpus. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Truck, 2022. URL htttps:://penrerieew net/fooum?? wtt.:/mmmmiiiiiiiimmaaaam..
Siqi Zheng, Luyao Cheng, Yafeng Chen, Hui Wang, and Qian Chen. 3d-speaker: A large-scale multi-devics multi-distance, and multi-dialect corpus for speech representation disentanglement. arXiv preprint urXiv:2306. 15354, 2023.
20

---

Published as a conference paper at ICLR 2025
A TASK TAXONOMY
Figures and 5 show the complete task taxonomy in Dynamic-SUPERB Phase-2. Additionally. Ta- cleppooearce our task taxonomy with the INTERSPEECH conference and EDICS from IEEE SPS. While not identical, our taxonomy is well-aligned with those organized by professional conferences and institutions.
La
Plase degrons
Puma Syenc
Navc Meel
Sccogettar
Phonose
Pronuedation
Shn Faloip
Audyoo
Evalsation
Plontonte Receggaation Fadks
Linguistic Strueture &
Sanaro Deneiioe
chassivatio
Arave
Fosture kaalyat
Plutolopical
Phonetics,
Geammar
lave
Maniari Stan
Phoneme
Rorapiicon
Processing
Intent Meaning
Dediger
Causticatio
Resegnnd enrignitnn
Dewertoo
Presodic Feature
Classifiuttion
Phonetics, Phonology, and Prosody
Spoken Language Understanding
Endiyy Prucessing
Naval Sinty Licaliaatinn
Prosddy New whwa
Prasody
Pravedic
Stateiing Desecion
Diorder Detecttion Classilicction
Cade Swaihing
ASR
Speech
Minerdrr
Dioeder Chaselinain
Language
Maiet
Antomatic
Speech Recognition
Kohanerarwt
Speech
Degradation Detection
Reverbradon Dusesson
ASR Post-
Speech
Safety & Security
Processing
Recognition
N-lle Conecllo
Prefucton
Spenting Sott-Sppnting
tepe Souder ASi
Speaker
Spontancous Sperch
Speeifie
Speaker & Language
Para- linguisties
Syntheric Speech Defection
Notae Dempzon
Recognition
Spool Deaecion
Communi Becoppoann
Tasks
Emation
Synduric Posmmall Aeragratio
ASR
Keywor Sprmen
Opayylyy Auumll
Speaker
Language
Vocal
Rmasin Clange Deeesto
deroponue
Depptabe Joice
Speaker
Recepation
Ematic
Recogaition
Chursclerii
Speaker
COVED Codd
Speci
Laaang
Speteer Chlrniigg
Spiier mesusio
Speiic Wetteete
Adl boce
Figure 4: Task taxonomy of speech.
21

---

Published as a conference paper at ICLR 2025
Dicnestic
Soands
Claetitation
Cal Emetion
Anmall Souni
Groural
Brehiiee
States Classficatoo
Soundssapes
Natural
Nocal Techetipor
Environmental
Clasifiation
Uibaz Nosaeo
Classification
Sound
Animal
Best Trecking
Lynk Tramsttion
Vocal Techniques
Environment
Lyric Recogition
Rhythm
Andia
Seganend
Sound Event
Lyries
Heveeeval
Singing Analysis
Exsactoo
Multi Chanued
Advanced Underslndian
Sound Event
Unlerstanding
Pitch
Puthh Clessficaum
Vocal Classfficalin
Human
Harmony Pitch
Music & Audio
Detectio
Audio
Music
Harmony
Editing Mrnaficatio
Audio Manipulation Detection
Chord Clasification
Safety
Scew Pale
Genre
Detection
Muskc Geare
Deepfake
Sigoal-Level
Class/ication
Characteristies
Music Classification
Emotion
Singing Voice Deeplake Desection
Audio Deration
Analysis
Prodiction
Enotien Clasificatto
MtulleSpp Speech
Spatial
Assessment
Classiiicaton
Audio
Quality
Percussion fostruments
Secad Ellect Dorection
Audio Sipubal Disunve! Predictio
Instrument
Clnsillution
Stroke
Sound Positicn Prediction
Percussion
Singlng
Clastllaaton
Recoptiion
Cosiliiation
MOS Prodiction
Beijonn Operc
Clacaliiation
Inttrmann
Ceonral
Sanging Autonatic
Clevvitaaion
Figure Task taxonomy of audio and music.
22

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of task laxonomy in Dynamic-SUPERB Phase-2 with INTERSPEECH and EDICS. A "v" indicates that conference session, area, EDICS category matches or closely aligns with the taxonomy.
Taxonomy
INTERSPEECH EDICS
Music/Harmony Pitch
Music/Music Classification
Music/Rhythm Analysis
Audio/Quality Assessment
Audio/Safety
hudio//iinal-Chaanctrriiiiici Analysis
Audio/Singing Analysis
Audio/Sound Event
Audio/Spatial Audio Analysis
Opeech/Paralinguie
ppeech/Phonetinn Phonology, Prosody
Speech/ /Saett and Security
Speech/Speatee & Language
Speech/ Enhancement
Speech/Spece Recognition
Speech/Speeeee Voice, Hearing Disorder
Speech/Spoken Language Understanding
B DYNAMIC-SUPERY TASKS
Table 4: The list of all tasks in Synamic-SUPERY Phase-2. Tasks are ordered by the taxonomy.
Core tasks are highlighted in green (SUPERB), blue (HEAR), or orange (MARBLE).
Domain
Task
Dataset
Audio Harmony & Pitch / Harmony
Chord Classification
Musical Instrument Chord Classification (Audio)
(DeepContractor 2024
Audio Harmony & Pitch Harmony
MARBLE Key Detection
GiantSteps (key) Knees et al.
2015)
Audio Harmony & Pitch / Pitch
HEAR Music Transcription
MAESTRO (Hawthorne et al.
2019
Audio/ Harmony &
HEAR Percussion Instruments Tonic Classification
Mridangam Stroke
Pitch / Pitch
Anantapadmaanahan et
2013
Audio Harmony & Pitch Pitch
Instrument Pitch Classification
Nsynth (Engel al.
2017
Audio Harmony & Pitch Pitch
Pitch Extraction By Lyrics
CSD CCoi et al. 2020
Audio Music Classification Emotion
Emotion
EMOTIFY Allanaki et al.
2016
Classificaiton In Songs
Audio / Music
MARBLE Emotion Detection
MTG-Jamendo
(Bogdanov et al.
Classification/ Emotion
2019
Audio Music
HEAR Music Genre
ISMIR04
Cano et al. 2018
Classification / Genre
Classification
Continued next page
23

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Audio Music
MARBLE Genre Classification
MTG-Jamendo (Bogdanov et al.
Classification / Genre
2019)
MTG-Genre
Audio Music
MARBLE Music
Magna ATune (Law et al.
2009)
Classification Genre
Tagging
Audio Music
MARBLE Music
MTG-Jamendo (Bogdanov al.
Classification / Genre
Tagging
2019)
Audio Music
Music Genre
FMA (Defferrard al. 2017
Classification Genre
Classification
Audio / Music
HEAR Percussion
Beijing Opera Percussion Instrument
Classification /
Instruments
Dataset (Tian et al. 2014)
Instrument
Classification
Audio Music Classification
HEAR Percussion
Mridangam Stroke
Instruments Stroke
Anantapadmamabbann et al. 2013
Instrument
Classification
Audio Music
Instrument
Nsynth (Engel al. 2017)
Classification /
Classification
Instrument
Audio Music Classification
Instrument
OpenMIC-2018 (Humphrey et al.
Combination Recognition
2018
Instrument
Audio / Music Classification
Instrument Source Classification
Nsynth (Engel et al. 2017)
Instrument
Audio Music
MARBLE Instrument
MTG-Jamendo (Bogdanov et al. 2019)
Classification
Instrument
Classification
Audio / Quality Assessment Singing
Singing Automatic
SingMOS TTng et 2024b)
MOS Prediction
Audio Rhythm
MARBLE Beat Tracking ASAP
ASAP Foscarin et al.
2020
Analysis
Audio Rhythm
Music Beat Tracking ASAP
ASAP Foscarin et al.
2020
Analysis
Audio Safety Audio Integrity
Audio Editing Identification
People's 's Speech (Galvez et al.
2021
Audio / Safety Audio
Scene Fake Detection
ASPIRE
Gogate et al.
2020
Integrity
Audio / Safety/ Deepfake
Audio Deep Fake Detection
LJSpeech (Ito & Johnson, 2017
WaveFake (Frank & Schönherr)
MUSDB 18HQ (Rafii et al. 2019)
Audio / Safety / Deepfake
Singing Voice Deepfake Detection
CtSVDD (Zang et al. 2024)
ACEKiSing (Shi et al.
2024al
M4Singer (Zhang et al.
[2022)
Audio / Signal
Audio Duration Prediction
NTUML22021 Yang al.
2024ab
Characteristics Analysis
Continued on next page
24

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Audio Signal
HEAR Music
MAESTRO (Hawthorne et al. 2019)
Characteristics Analysis
Speech
Librispeech (Panayotov al. 2015
Classification
Audio / Signal
Sound Effect
RemFx Rice et al. 2023
Characteristics Analysis
Detection
Audio Signal
Speech Detection
LibriSpeech-
(Panayotov
Characteristics Analysis
ct al. 2015)
Audio / Signal
Speech Detection
TesttOher (Panayotov
Characteristics Analysis
at
2015)
Audio / Signal
Speech Detection
LJSpeech (Ito & Johnson 2017
Characteristics Analysis
Audio Singing
Children Song
CSD Choi et al. 2020
Analysis / Lyrics
Transcript
Verification
Audio / Singing
Lyric Translation
SingSet (self-recorded)
Analysis/ / Lyrics
Audio Singing
Song Lyric Recognition
SingSet (self-recorded)
Analysis / Lyrics
Audio / Singing
MARBLE Vocal
VocalSet (Wilkins et al. 2018)
Analysis / Vocal
Technique Detection
Techniques
Audio Sound Event Advanced
Audio Segment Retrieval
Clotho Drossos et al. 2020)
Understanding
Audio / Sound Event
HEAR Sound Event Detection
DCASF20161ask2 (IEEE DCASE 2016 Challenge 2016)
Advanced
Understanding
Audio / Sound Event
Multi-channel Sound
STARSS23 (Shimada et al. 2024)
Advanced
Event Understanding
Understanding
Audio / Sound Event /
Animal
WaveSource Test (WavSource n.d.
Animal
Classification
Audio / Sound Event Animal
Bird Sound
Warblrb10k (Stowell et al. 2019
Detection
Audio Sound Event Animal
Cat Emotion
CuSSondClasstiiaiionDDaaadd V2 (Pandeya et al.
Classification
2018 Pandeya & Lee
2018)
Audio / Sound Event / Animal
Cornell Birdcall
Cornell Birdcall Identification (Howard al.
Identification
2020
Audio Sound Event Animal
Environmental
Ancma-Animals
Piczak 2015)
Sound Classification
Audio / Sound Event / Animal
HEAR Beehive
Beehive States Nolasco al. 2019)
States Classification
Audio / Sound Event
Domestic
DESED-PublieEval (Turpault et al. 2019
Environment
Environment Sound
Event Detection
Continued on next page
25

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Audio Sound Event Environment
Emergency Traffic Detection
Large-Salle-Audii-tttneee (Asif et al.
2022
Audio / Sound Event Environment
Environmental
SSC50-Etteior/nnnUUaaNNNissss
Sound Classification
Piczak 2013
Audio / Sound Event Environment
Environmental
ESC50-
Sound Classification
Piczak 2013)
Audio Sound Event Environment
Environmental
ESC50-
Sound Classification
Vauna//onddsaag///haaeeeeeesuue Piczak 2013)
Audio / Sound Event Environment
Environmental
UrbanSound8K SrrrKK--rbnnnNsss
Sound Classification
Sallmonn et al.
2014
Audio Sound Event Environment
Environment
ESC50 Piczak 2015)
Recognition
Audio / Sound Event Environment
HEAR
ESC50 Piczak 2015)
Environmental
Sound Classification
Audio Sound Event Human
HEAR Vocal
Vocal Imitations (Kim et al. 2018)
Imitation
Classification
Audio / Spatial Audio
Audio Spatial
Spatial LibriSpeech (Sarabia et al.
Distance Prediction
2023
Audio Spatial Audio
How Far Are You
3DSpeaker (Zheng et al.
2023)
Audio Spatial Audio
Sound Position Prediction
BinauralSounPeercpptoo Vasudevan
ct al. 2020)
Speech / Paralinguistics
Dialogue Emotion Classification
Daily Talk (Lee al, 2023
/ Emotion Analysis
Speech / Paralinguistics
Emoji Grounded Speech Emotion
RAVDESS Livingstone & Russo 2018
/ Emotion Analysis
Recognition
Speech / Paralinguistics
Emotion Change
RAVDESS Livingstone & Russo
/ Emotion Analysis
Detection
2018
Speech / Paralinguistics
Emotion Recognition
MELD
Poria et al. 2019
/ Emotion Analysis
Speech Paralinguistics
HEAR Emotion
CREMAD
Cao et al.
2014
/ Emotion Analysis
Recognition
Speech / Paralinguistics
SUPERB Emotion Recognition RAVDESS
RAVDESS (Livingstone Russo 2018)
/ Emotion Analysis
Speech / Paralinguistics
Covid19 Cough
Cough Vid Orlandic et al. 2021
/ Vocal Event Detection
Audio Classification
Speech / Paralinguistics / Vocal Event Detection
Environmental
Sound Classification
Piczak 2015)
Speech / Paralinguistics / Vocal Event Detection
Human Non-Speech
Nonspeech7k-tess (Rashid et al.
2023
Sound Recognition
Common Voice (Ardila et al. 2020
Speech / Paralinguistics / Vocal Event Detection
Human Screaming Detection
Environmentdh (whats2000
2023
Continued on next page
26

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset 2022
Speech / Paralinguistics
Vocal Sound Recognition
VocalSound (Gogg et al,
/ Vocal Event Detection
Speech / Phonetics,
Phoneme Segment Counting
Librispeech-words (lib)
Phonology, Prosody
Phonetics and Phoneme Processing Phoneme
Recognition Tasks
Speech / Phonetics,
Phone Segment
VoxAngeles Chodroff et al. 2024)
Phonology, Prosody
Counting
Phonetics and Phoneme
Processing / Phoneme
Recognition Tasks
Speech / Phonetics,
SUPERB Phoneme Recognition
PeriSpeech-TestCleee (Panayotov et al.
Phonology, Prosody
2015)
Phonetics and Phoneme Processing Phoneme Recognition Tasks
Speech / Phonetics, Phonology, Prosody Phonetics and Phoneme
Phoneme Recognition
LibriSpeech-
(Panayotov
et al.
2015
Processing / Phoneme
Recognition Tasks
Speech / Phonetics, Phonology, Prosody
Phonological Feature Classification
Vox Angeles- ConsonaatPaaceOOAAriulltaii
Phonetics and Phoneme
(Chodrotf et al. 2024)
Processing / Phonological Feature Analysis
Speech / Phonetics, Phonology, Prosody
Phonological Feature Classification
VoxAngeles- ManneOOfArtiiulation Chodrofff et al. 2024)
Phonetics and Phoneme
Processing / Phonological Feature Analysis
Speech / Phonetics, Phonology, Prosody Phonetics and Phoneme
Phonological Feature Classification
VoxAngeles-Phoot (Chodroff al.
2024
Processing / Phonological Feature Analysis
Speech / Phonetics, Phonology, Prosody Phonetics and Phoneme
Phonological Feature Classification
VoxAngeles- Vooneerrseoowerrrrnneeesss CChorooff et al. 2024)
Processing / Phonological Feature Analysis
Continued on next page
27

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Phonetics,
Phonological Feature Classification
YoxAngeles-Vooeeleeighh (Chodroff at 2024)
Phonology, Prosody / Phonetics and Phoneme
Processing / Phonological Feature Analysis
Speech / Phonetics, Phonology, Prosody /
Phonological Feature
VoxAngeees- VowelRoundedness Chodroff et al. 2024
Classification
Phonetics and Phoneme
Processing / Phonological Feature Analysis
Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme
Heteronym Differentiation
HeteronymEn (self-constructed)
Processing / Pronounciation
Evaluation
Speech / Phonetics, Phonology, Prosody
L2 English Accuracy
speechocean762-Raan Rankkin (Zhang et 2021)
Phonetics and Phoneme
Processing / Pronounciation
Evaluation
Speech / Phonetics, Phonology, Prosody
L. English Accuracy
speechocean7622ssooingg (Zhang et al. 2021
Phonetics and Phoneme
Processing / Pronounciation
Evaluation
Speech / Phonetics, Phonology, Prosody Phonetics and Phoneme Processing/ / Pronounciation
Multilingual Pronunciation Similarity
VoxAngeles Chodroff et al. 2024)
Evaluation
Speech / Phonetics, Phonology, Prosody
Accent Classification
Accentdb Extended (Ahamad et al.
2020
Prosody Prosodic
Feature Classification
Speech / Phonetics,
Stress Detection
MIRSD CHEN & JANG 2012
Phonology, Prosody
Tseng 2008
Tsai 2015)
Prosody Prosodic
Feature Classification
Speech / Phonetics,
Third Tone Sandhi Recognition
NCCU Corpus of Spoken Taiwan Mandarin (National Chengchi University n.d.)
Phonology, Prosody
Prosody Prosodic
Feature Classification
Continued on next page
28

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Phonetics,
L2 English Fluency
Frecchocean762RRnnkie Zhang
Phonology, Prosody
at
2021
Prosody Prosodic
Quality Assessment
Speech / Phonetics, Phonology, Prosody
L2 English Fluency
specchocea7762SSoornng Zhang et al.
2021
Prosody Prosodic
Quality Assessment
Speech / Phonetics, Phonology, Prosody
L22 English Prosodic
speechocean762-Rain Ranking Zhang
et al. 2021
Prosody Prosodic
Quality Assessment
Speech / Phonetics, Phonology, Prosody
L2 English Prosodic
speechocean762-Scorine (hhag et al.
2021
Prosody Prosodic
Quality Assessment
Speech Phonetics, Phonology. Prosody
Prosody Naturalness
ProsAudit-Lexical (de Seyssel et al.
2023
Prosody Prosodic
Quality Assessment
Speech Phonetics,
Prosody Naturalness
ProsAudit- Protossyttt (de Seyssel et al. 2023)
Phonology, Prosody Prosody Prosodic
Quality Assessment
Speech / Safety & Security/ Spoofing and Anti-Spoofing
Spoof Detection
ASVspoof2015 (Wu al. 2015
Speech / Safety
Spoof Detection
ASVspoof2017 Delgado al.
2018
Security Spoofing and Anti-Spoofing
Speech / Safety Security Synthetic
Deep Fake Voice Recognition
DEEP-VOICE (Bird & Lotfi
2023)
Speech Detection
Speech / Safety & Security Synthetic
Enhancement Detection
LibrilTS-TestClean (Zen et al. WHAM (Wichern et al. 2019
2019
Speech Detection
Speech / Safety& &
Fraud Robocall Recognition
CallHome Katerenchuk al.
2018
Security Synthetic
Speech Detection
Speech / Safety&
Fraud Robocall Recognition
Promo (self-constructed)
Security Synthetic
Speech Detection
Speech / Safety
Fraud Robocall Recognition
Robocall Prasad & Reaves
Security Synthetic
Speech Detection
Speech / Speaker &
HEAR Language Identification
VoxLingua 107 Top10 (Valk & Alumäe 202T)
Language/ Language
Language Identification
Continued on next page
29

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Speaker
Language Identification
VoxForge MacLean 2018
Language Language /
Language Identification
Speech / Speaker &
Age Classification
VoimmonniiccCCopps---lee Ardila et 2020
Language Speaker
Speaker Characteristics
Speech / Speaker & Language Speaker /
Gender Recognition by Voice
ConmonnVice--elllSeeeennnn/ (Ardila et al. 2020
Speaker Characteristics
Speech / Speaker &
HEAR Speaker Count Identification
LibriCount (Stöter et al. 2018
Language Speaker /
Speaker Recognition
Speech / Speaker
Multi Speaker
VCTK Yamagishi et al.
2019)
Language Speaker /
Detection
Speaker Recognition
Speech / Speaker &
Speaker Counting
LibriTTSS-TestClean (Zen et al.
2019)
Language Speaker
Speaker Recognition
Speech / Speaker &
Speaker Verification
LibriSpeech-TesiClee et al. 2015)
Paaaayotov
Language Speaker
Speaker Recognition
Speech / Speaker &
Speaker Verification
LibriSpeech-Testtoee et al. 2015)
(Panayotov
Language Speaker /
Speaker Recognition
Speech / Speaker
Speaker Verification
VCTK Yamagishi et al. 2019)
Language Speaker /
Speaker Recognition
Speech / Speaker
SUPERB Speaker
Libri2Mix- Test (Cosentino et al. 2020
Language Speaker /
Diarization
Speaker Recognition
Speech / Speaker &
SUPERB Speaker Verification
SUPERB Hidden Set (self-reconstruuteed)
Language Speaker
Speaker Recognition
Speech / Speech Enhancement / Degradation Detection
Noise Detection
LJSpeech (Ito & Johnson 2017 MUSAN-Gausiia (Snyder et al.
2015
Speech / Speech Enhancement / Degradation Detection
Noise Detection
LJSpeech (Ito & Johnson 2017
MUSAN-Music (Snyder et al. 2015)
Speech / Speech Enhancement / Degradation Detection
Noise Detection
LJSpeech (Ito & Johnson 2017
MUSAN-Noise (Snyder et al.
2015
Speech / Speech Enhancement / Degradation Detection
Noise Detection
LJSpeech (Ito & Johnson 2017
MUSAN-Speech (Snyder et al. 2015)
Speech / Speech
Noise Detection
VCTK (Yamagishi et al. 2019)
Enhancement
MUSAN- (Snyder et al. 2015)
Degradation Detection
Continued next page
30

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Speech
Noise Detection
VCTK (Yamagishi et al., 2019)
Enhancement / Degradation Detection
MUSAN-Noise (Snyder et al.
2015)
Speech / Speech Enhancement/ / Degradation Detection
Noise Detection
VCTK (Yamagishi et al., 2019)
MUSAN-Speech (Snyder et al.
2015
Speech / Speech Enhancement
Noise Detection
VCTK Yamagishi et al. 2019)
MUSAN-Gausian (Snyder et al.
Degradation Detection
2015
Speech / Speech Enhancement / Degradation Detection
Noise SNR Level Prediction
VCTK YYamaiishi et al.
2019)
MUSAN-Gaussian (Snyder et al.
2015
Speech / Speech Enhancement / Degradation Detection
Noise SNR Level Prediction
VCTK (Yamagishi et al. MUSAN-Music (Snyder et al. 2015
2019)
Speech / Speech Enhancement / Degradation Detection
Noise SNR Level Prediction
VCTK (Yamagishi et al. 2019)
MUSAN-Noise (Snyder et al. 2015
Speech / Speech Enhancement
Noise SNR Level Prediction
VCTK (Yamagishi et al., 2019)
MUSAN-Speech (Snyder et al. 2015)
Degradation Detection
Speech / Speech Enhancement / Degradation Detection
Reverberation Detection
LJSpeech (Ito & Johnson 2017
RirsVoises-TLargeRoom (Ko et al.
2017)
Speech / Speech Enhancement / Degradation Detection
Reverberation Detection
LJSpeech (Ito & Johnson 2017 RirsNoises-MediuRRoon (Ko et al.
2017
Speech / Speech Enhancement / Degradation Detection
Reverberation Detection
LJSpeech (Ito & Johnson 2017 RirsNoises-SmallRRoon (Ko al.
2017)
Speech / Speech Enhancement / Degradation Detection
Reverberation Detection
VCTK YYmaagishi et al., 2019)
LeaNooses--LargeRoooe (Ko et al.
2017)
Speech / Speech Enhancement / Degradation Detection
Reverberation Detection
VCTK (Yamagishi et al. 2019)
(Ko et al.
2017
Speech / Speech Enhancement / Degradation Detection
Reverberation Detection
VCTK (Yamagishi et al.,
2019)
RirSNoises-SmallRoom (Ko et al.
2017
Speech / Speech Recognition/ ASR Post-Processing
N-Best Correction
Librispeech-TesttOee al. 2015)
(Panayotov
Speech / Speech Recognition/ Language
AAVE Speech
CORAAL Kendall & Farrington
Recognition
2023)
Speech / Speech
Code-switch Speech
NTUML2021
Yang et al.
2024a)
Recognition Language
Recognition
Speech / Speech
Code-Switching Speech Recognition
ASCEND Lovenia et al.
2022
Recognition/ Language
Continued on next page
31

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Speech
Multi-Lingual
MLS-de Prrttpp et al.
2020
Recognition/ Language
Speech Recognition
Speech / Speech
Multi-Lingual
MLS-en (Pratap et al.
2020
Recognition / Language
Speech Recognition
Speech / Speech
Multi-LLngual
MI.S-ss Pratap et al.
2020
Recognition/ Language
Speech Recognition
Speech / Speech Recognition Language
Multi-Lingual
MLSSfr (Pratap et al.
2020
Speech Recognition
Speech / Speech
Multi-Lingual
MLS-it Pratap al.
2020
Recognition/ / Language
Speech Recognition
Speech / Speech
Multi-Lingual
MLS-nl
Pratap et al.
2020
Recognition/ Language
Speech Recognition
Speech / Speech
Multi-Lingual
MLS-pl Pratap et al.
2020
Recognition/ Language
Speech Recognition
Speech / Speech
Multi-Lingual
MI.S-pt Pratap et al.
2020
Recognition Language
Speech Recognition
Speech / Speech
PTBR Speech
TommonVoiccl17-Test Ardila et al.
Recognition/ Language
Recognition
2020
Speech / Speech
SUPERB ASR
LibriSpeech-TesttCeem (Panayotov 2015)
Recognition/ Language
al.
Speech / Speech
ASR
LibriSpeech- Pannayotov
Recognition Language
al 2015)
Speech / Speech
SUPERB OOD ASR
Common Voice--Tet (Ardila et al.
Recognition/ Language
(Ar)
2020
Speech / Speech
SUPERB OOD ASR (Es)
CommonVoice77115 (Ardila et al. 2020
Recognition/ Language
Speech / Speech
SUPERB OOD ASR
CHIME6 Test (Barker et al. 2018 Watanabe al. 2020)
Recognition/ Language
(Spontaneous)
Speech / Speech
SUPERB OOD ASR (Zh)
Voice-nVoice7-Tess (Ardila al. 2020
Recognition/ Language
Speech / Speech
Target Speaker ASR
AMI-test (Carletta et al.
2005
Recognition/ Speaker
Speech / Speech
Multi-Speaker Detection
LibriSpeech- (Panayotov al. 2015)
Recognition/ Specific
Recognition Tasks
Speech / Speech
Speech Command Recognition
AudioMNIST (Becker et al. 2024)
Recognition/ Specific Recognition Tasks
Speech/ / Speech Recognition/ Specific Recognition Tasks
Speech Text Matching
TennSpeech--TetCleen (Panayotov et al. 2015)
Speech / Speech
Speech Text Matching
LibriSpecch-Testtee
(Panayotov
Recognition/ Specific
al. 20155
Recognition Tasks
Continued on next page
32

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Speech
Speech Text Matching
LJSpeech (Ito & Johnson 2017
Recognition Specific
Recognition Tasks
Speech / Speech
Spoken Term Detection
ChbriSpeech-TestCCCe Paanayotov
Recognition Specific Recognition Tasks
ct 2015)
Speech / Speech
Spoken Term Detection
Librippecch-eestttheer TestOther Paanayotov
Recognition Specific Recognition Tasks
ct al. 2015)
Speech / Speech
Spoken Term Detection
LJSpeech (Ito & Johnson, 2017
Recognition Specific Recognition Tasks
Speech Speech
SUPERB Keyword Spotting
SpeechCommandSV (Warden 2018
Recognition Specific
Recognition Tasks
Speech Speech
SUPERB Query by Example
Quesst14-Fvva (Anguera et al. 2015)
Recognition Specific Recognition Tasks
Speech / Speech, Voice, Hearing Disorder
Stuttering Detection
SEP-28K (Lea et al.
2021
Disorder Detection and Classification
Speech / Speech, Voice, Hearing Disorder
Voice Disorder Classification
VOICED (Goldberger et al.
2000)
Disorder Detection and Classification
Speech / Spoken
Conversation Matching
EnShoriConversation CEEC
Language
Understanding Intent & Meaning
Speech / Spoken
Dialogue Act Classification
SLUE-HVB Shon al. 2022
Language Understanding Intent & Meaning
Speech / Spoken
Dialogue Act Classification
DailyTalk (Lee al. 2023
Language Understanding Intent & Meaning
Speech / Spoken
Dialogue Act Pairing
DailyTalk (Lee et al.
2023
Language
Understanding / Intent
Meaning
Speech / Spoken
SUPERB Intent
SLURP (Bastianelli et al. 2020)
Language Understanding Intent & Meaning
Classification
Continued next page
33

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Spoken
Intent Classification (Action)
SLURP-Action (Bastianelli et al.
Language Understanding Intent & Meaning
2020
Speech / Spoken
Intent Classification (Intent)
SLURP-Intent (Bastianelli et 2020
Language Understanding Intent & Meaning
Speech / Spoken
Named Entity Localization
SLUE-Vox LLUE-VoxPopuli (Shon et al.
2022
Language Understanding Intent & Meaning
Speech / Spoken
Named Entity Recognition
SLUE- SLUE-VoxPopuli (Shon et al.
2022)
Language Understanding / Intent & Meaning
Speech / Spoken
Sarcasm Detection
Mustard (Castro et al. 2019)
Language Understanding Intent & Meaning
Speech / Spoken
Semantic Textual Similarity
SpokenSTS (Agirre et al. 2012)
Language Understanding Intent & Meaning
Speech / Spoken
Speech Sentiment Analysis
MELD Poria et al.
2019
Language Understanding Intent & Meaning
Speech / Spoken Language Understanding Intent & Mcaning
Spoken Digit Arithmetic
AudioMNIST (Becker et al.
2024
Speech / Spoken
SUPERB Slot Filling
AudioSnips-Test Yang et al.
2021
Language Understanding Intent & Meaning
Speech / Spoken
Code-Switching Semantic Grammar Acceptability Comparison
CSZS-zh-en (Huang et al. 2024b)
Language
Understanding! /
Linguistic Structure & Grammar
Speech / Spoken
Nonce Word Detection
sWUGGY (Dunbar et al.)
Language
Understanding /
Linguistic Structure & Grammar
Continued on next page
34

---

Published as a conference paper at ICLR 2025
Domain
Task
Dataset
Speech / Spoken
PoS Estimation LibriTTS PoS
LibriTTS Zen et al. 2019)
Language Understanding /
Linguistic Structure &
Grammar
Speech / Spoken
PoS Estimation LibriTTS PoS with transcription
LibriTTS (enn et al. 2019)
Language
Understanding /
Linguistic Structure & Grammar
Speech / Spoken
Sentence Grammar Acceptability
sBLIMP
Dunbar al.)
Language
Understanding/ /
Linguistic Structure & Grammar
Speech / Spoken
SUPERB Speech Translation
CoVoST2-Test (Wang et al. 2021
Language Understanding Speech Translation
C TASK-LEVEL EVALUATION RESULTS
Table 5: Task-level Evaluation Results: GPT4o
Task
Metrie
LEaMA
13B
Andio Chai
Andio-
LLMMA
IT
Churd Classifcation
LLM Ct
4.00%
10.00%
0.50%
9.00%
41.00%
28.50%
44.50%
$1.00%
0.00%%
MARBLE Key
LLMCT
3.00%
0.50%
1.50%
1.00%
3,00%
17.00%
2.00%
0.50%
0.00%S
Detection
(GimmStepss- Key)
HEAR Music Transccitiinn
NAR.
92.43%
1000005
99.40RS
100.005
96.7696
96,7616
96.76%
100.00%
100,009E
(MAESTRO)
TERI
Trimscrptiin (MAESTRO)
HEAR Poncession Totie Classffcation (Mridingam Tonic)
2.009
2,50%
12,70%
13,409
11.30%
11.80%
12,805%
12,500
5.7005
0.11%
0.78%
(Nsynth)
Pitch Exuaction
NARL
37.50%
94.00%
47.90%
73.50%
65.50%
84.50%
76.00%
98.00%
68.50%
By Lyrics (CSD)
Pitch
639.24%
165.96%
136.119
Ry Iyries (CSD)
Continued on next page
35

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMOONN
SALMOON
Quee- Chaf
Cheen2- 78-tnet
Worl.M E.M
ML-
GAMAX
Emocion
LLMCT
10.0%%
0.00%
1.57%
3.33%
13.339
8.33%
8.33%
L677%
0.00%
Classffiaiton
Songs
(EMOTIFY)
MARBLE
LLM.CT
0L00PS
1.50%
11.00%
0.0005
2.70%
0.80%
0.80%
0.80%
0.10%
Ditersion (MTGG)
TEEAR Music
LLMCTT
15.58%
7,048
31.16%
46.22%
33.67%
37.69%
16.58%
27.649
2.51%
Genre Classification
(IMIRII4)
MARBLE Genre
LLM-CT
0.00%
1.60%
0.10%
0.20%
3.20%
0.10%
0,20%
0.20%
0.00%
Classification
(MTG)
Music
3.50%
1.50%
1.00%
Tagging (MagnaTag ATlane)
MARBLE Music Tagging (MTG)
LLMCt
0.10%
0.30%
0.00%
0.00%
2.901
0.301
0.10%
0.505
0.10%
10.71%
Classfication
(FMA)
HEAR Percession
LLM.CT
3.399
4.24%
10.17%
20.7666
16.10%
27.1256
24.15%
22,03%
11.44%
(Beijing Opera
Percussion)
Percussion Instruments Classification
LLMcC
3.80%
7.60%%
4.90%
11.50%
13.201
10.30%
(Mridangum
Stroke)
14,589%
17,899
Classificaiion (Nsynih)
Iestmument Combination Recogaition
1.M-CT
4.20%
13.51%
17.7296
24.0025
25.839
24.929
20.425
19,529
8.11%
(0pesMIC-20188
Instrment Source Classification (Nsynt)
LLMCT
7.22%
28.33%
37.33%
43.56%
32.67%
35.22%
30,891
38.00%
35.89%
1EM-CI
2.30%
0.00%
1.40%
0.205
Enstrument Classification
Singing Automeetic MOS Prediction (SingMOS)
NAR
50.50%%
17.30%
0.00%
97.349
Singing Automatic
23872
10.0F77
11,2308
14,8895
1.4085
2.08
(SingMOS)
Singing Automatic
KTAU
-03333
0.0126
10.1105
0.0166
0.0712
0,0099
0.0103
0.1705
0.3539
MOS Prediction (SingM11)
Singing Autousatic
LOCT
MUS Prediction (SingMOS)
Singing Automatic MOS Prediction (SingMOS)
SRCC1
0.5
0,0133
0.1325
0.0231
0.0872
0.0124
0.0118
0.2281
Q.1247
Continued on next page
36

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LITEAS
SALMOON
SALMOON
Quer- Chof
Cheen2- 78-Insl.
E.
ML-
GAMAX
MARBLE Beat
NARL
100,00%
100,005
100.00%
100.00%
100,00SS
100,00S
100,005
100,00%
100.00%
FRR.
Tracking (ASAP)
Music Beat Trncking (ASAP)
NARL
54.55%
74.55%
$1,82%
63.64%
62.73%
40.91%
47.271
80.00%
$4.55%
135.6077
18366786
77,9687
157.9917
Tracking (ASAP)
Time
Andio Editiig Idemiiication
NARA
84 38%
50%8
62,50S
75.00%
75,000
68.75%
96.88%
65.62%
$7.50%
iPoople's Speech)
Editing Idemiificaioon (Poople's Speech)
Somne Fake Detection (ASPIRE)
LLM-C1
22.00%
46 000%
48.00%
37,009
49.00%
54,00%
49.00%
20,00%
41.00%
Audio Deep Pake Detection
LLM.CT
8.25%
24.506%
22.799
38.31%
31.4396
27.9016
30.84%
39.10%
25.54%
(1.5peach,
MUSDBISHQ)
Singing Voice Deeplake
LLMCC
5.56%
22.00%
22.00%
26.19%
24.44%
21.43%
9.211
20.95%
9.345
Detection (CrSSDD. ACEKiSSing Mr-inger)
Dunation (NTUML2001))
33.00%
0.50%
Audio Dunation Prodiction
MSEL
13.3657
39.1111
31652.877
22167111
28.985
62,515
3581
1527.785
02,7674
(NE1
[EEAR Music Speech Classification (MAESTRO,
LLMCT
72.50%
77.50%
91.50%
99.50S
98.00%
89.50%
59.50%
$1.50%
$8.00%
Librispecch)
Sound Fffect Detection (RemFix)
LLM-CT
2,00%
6.50%
14.17%
17.67%
15.839
19.839
17.50%
17.00%
15.889
Speech Detection (LibriSpeechs TeoeClaa))
LLMCT
Sperch Detection (LibriSpeoch. TestOtber)
LLMct
52.00%
31.50%
46.00%
66. 66.50%
41.50%
55.90%
37.00%
45.50%
36 00%%
Speech Detection (LISpeech)
LLM-T
$4.0%%
44.50%
60,50S
74.00%
$7.00%
45.000%
$3.00%
40.00%
55.550%
Children Transcripe Verification (CSD)
58.8%%
102.085
104.225
100.999
128.679
94.931
155.4856
100.00%
Lyric Transation SeingSet)
Sacre Bleut
2,1093
1.0357
2,8126
0.994]
3.3975
3.9155
1.8969
0.1593
0.0739
Song
115.62%
195
Recogatton SSingSet)
Continued on next page
37

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMOONN
SALMOON
Quer- Chof
Cheen2- 78-tnel.
E.M
ML-
GAMAX
MARBLE Vocal Technique
LLM-T
8,00%
L,0%
11,00%
6.50%
6,50%
15.90%
8.00%
8.50%
2.50%
Detection (NocalSei)
Alsdo Segnent
NARL
S0.00%
78.00%
2,00%6
4.00F%
12,00%6
4.00%
14,00%E
82.00%
GUO0SS
Retrieaal (Cloohho)
Rettieval (Clotho)
ITEAR Sound
LLMC
0.00%
0.00%
21.43%
35.71%
7.14%
14.29%
21.43%
21.43%
7.144
Event Detection
Task2)
Multi-chranel Sound Fvent Understandin
LLMECT
58.73%
4.23%
37.5288
26.0096
39.0159
41.55%
21.85%8
4.93%
11.27%
(SLARS523)
Anamal
LLMECT
9.2596
40..00
58.2598
67.0095
73.7596
34.0016
12,001
4.25%
15.25%
Classification
(WaneSuurc--
Test)
Bird Detection
1LM-CT
28.
75.009
78.500
75.
40,50%
(WarlehhOk)
Einotion
0.00%
2.0056
4,001
14,0016
8.00%
14.00%
(Car
Sound Classifleation
Idenification
Environmental
LLMCT
14.50%
8.00%
14.00%
3.00%
82.00%
36.00%
5.00%
0.00%
2,00%
Sound
Clasiffiation
(ESCSO-Anine) Animuls)
HEAR Bechive
LLMECT
43.00%
35.0085
54,00%
42.50%
19.0096
42,501
37.00%
37.50%
19.00%
States
Clasiffiction
States)
Doasetti
NARL
91.39%
44%
99.72%
9907296
99.17%
99.72%
99.721
100.00%
99.72%
Environment
Sound Event
(DESED
Publickvvll)
Domestic Sound Evveeiettttion
Event- based FIt
0,01
(DESFD-
Emergency Traffic
LLM-CT
Detection
(Tanpo-Saale-
Audio-dataee))
Enviromeental
6.50%
6.50%
Sound
Classification
Nooses)
Envirommental Sound
LLMC
4.50%
3.50%
0.50%
0.00%
59.00%
2.50%
4.00%
0.00%
0.00%
Clasification
(ESCS0 Inserior And Domestic
Sounds)
Continued on next page
38

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SAL-MONN
SALMOON
Quern Chaf
Cheen2- 78-tned
Worl.M EM
ML-
GAMAX
Envirumentaa
LLM-CT
3,00%
7.50%
6.50%
1.00%
78.00S
5.00%
7.00%
0.00%
7.00%
Classification
(ESC50-Natre) Natural
Soundeupes And
Environmental Sound
LLM-CT
Clasilfiation
(UrbanSoundKK-
Noises)
Environment Recognitio (ESCS0))
LLMECT
7.00%
38.0098
42.11%
19.30%
56.14%
63.1616
40.351E
10.53%
31.50%
35.30%
Environmental
Sound
Classificttion
1EM-TT
18.675
Emitition
Classificaiion
Imittaionns)
Sputial
Distance
Prodiction (Spatial
LibnSpeechh
Sputial Distance Prediction (Spatial
LibrSpeecch)
How (3DSpesker)
26.00%
4.50%
30,005
18.00%
Sound Position Prodiction
NARL
$1.25%
1000006
1000005
100,00S
100,000S
25.0016
93.7518
100,00%
100,001E
Sound Position Prediction
Angle DiTT
1,4006
I,1063
0,8681
Dialogue Emation
33.50%
33.000%
Classfication (Daily Talk)
Emerii Grounded Speech Recognition (RAVDESS))
LLMcT
1.10%
0.00%
0.00%
0.10%
1.30%
1.20%
0.10%
1.90%
48. 18.50%
Emosion Change
LLM-C1
0.08%
0.42%
0.58%
0.96%
0.045
2.96%
0.08%
0.58%
0.04YS
Detection
(Ravdess)
Emocion Recognition
LLM-CT
22.50%
14.50%
19,009
9.50%
43.500
36.50%
40.50%
0.50%
2.50%
Recugnitin (CREMAD)
SUPERR Fmustinn (RAVDESS)
1EM-T
12.50%
25.83%
12,08%
12,505
70-42%
75.83%
12.08%
12.929
0.00%
Covid19 Cough
LLMECT
0.933%
1J0356
0.00%
0.00%
4.12%
0.72%
4.22%
21.91%
0.21%
Audio
Classification (Cough
Continued on next page
39

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMOONN
SALMOON
Quern Char
Cheen2- 78-tnel.
Worl.M E.
ML-
GAMAX
Envirumentaa
LLM-CT
17.30%
11.00%
8.50%
3.00%
$2,00%
29.00%
16.50%
0.00%
2,00%
Classification
(ESC50-Ulumun And Non Speech
Human
LLM-CT
27.145
17.145
Speect Sound
Recognition (Nonspecch7k. CommonNbioe)
Human Screaming Detection
LLMECT
52.50%
25.0096
52,50%
65.0055
62.50%
87.5016
47,50%
42.50%
00.00%
29
75.56%
Recognttin (NocalSound)
Phoneme Segment (1 (Iiiiipeoch. words)
NARA
16.29%
48 64%6
39,0995
11,6256
3.99%
1.73%
0.50%
2.67%
72.22%
Phoneme Segment
ACC?
13.56%
11.77%
11.84%
8.25%
5.42%
13,46%
9.55%
0.47%
2.29%
(Librispeech- Words)
Phoneme Segmens Couning (Librispeech- wortk)
Abs TiMrq
2.5378
1012.8103
6.5285
50.7724
7.1857
3.0622
7.9109
15.6777
5.0491
Phone Segment Couning (VoxAugeles)
NARL
19.19%
57.24%
41,88%
12,5096
0.411
16.11%
15.04%
41.47%
18.79%
Segnnent Counting
ACCT
(VuxAngeles)
Phone Segment
Abs DiTT.
41.236
3759.3878
5,4857
4,9331
3.1286
287066
45754.9952
5.4242
2.5694
(NoxAngoles)
SLIPERR Phoneme
PER L
100
102.75%
25.369
24.605
100.589:
100.9695
99.995
1.0.275
100.37%
TeseClean)
Phoneme Recogattin
PER.
100L17%
101.88%
22.79%
22.9156
100,62%
100.88%
99.9958
181.18%
100.615
(LibriSpeech-
Teattter)
Phonologial Feature
LLM-C
25.77%
1.39%
1.88%
1.37%
1,54%
3.24%
1.88%
1.54%
0.51%
Classification
(NuxAngeles-
Consonant Place Of Articulaioon
17.42%
9,36%
2.0698
1.50%
7.96%
8.99%
Feature
Clasification
(NexAngeles-
Manner
Asticulation)
Phonolugical Feature
LLMct
5.73%
0.09%
1.78%
3.60%
3.97%
0.00%
0.00%
0.09%
Classification
(NoxAngeles-
Phone)
Continued on next page
40

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SAL-MONN
SALMOON
Quee- Chaf
Cheen2- 78-tnel.
Warl E.
ML-
GAMAX
Phonological
LLM-CT
50.71%
17.52%
41.44
38 .9P
48.47%
57.23%
36.46%
42.97%
10.39%
Feature
Classification
(NuxAngeles-
Vowel Frontnes))
Phonological
24.44%
35.236
Feature
Classiftcation
(NoxAngoles-
Phonological
LLM.TT
Feature
Classification
(NoxAugelss-
Roundelness)
lltteroym
LLMCt
55.00%
26.00%
37,00%
45.00%
$2.00%
44.00%
51.00%
20.00%
20.00%
Differentiaion
(eseronymmin)
Fuglish Accuracy
1.M-CT
49.
48.06%
(gpeschoeeu766..
Ronking)
English Accuracy
XARL
tspeschoccun/62-
English Accuracy
Multi-liguua
38.40%
48.
23.709
44.201%
40.60%
Promunciation
Sinillrrtty (NoxAngeles)
17.50%
3,0056
26.50%
14,0016
7.00%
27.00%
4.00%
(Aecenidb Extended)
Streee Desection (MIRST)
LLM-C1
15.30%
3.00%
2.00%
13,009
16.50%
23.50%
25.50%
0.00%
1. 1.00%5
NAR
38%
96.885
Recogaition
(NCCU Corpus
Taiaan Mundaci)
Thind Tone Sandhi
loUit
0.2179
0.75
D
0.1364
Recognition (NCCU Corpus
Spoken Mandarin)
1.2 Fnglish
1LM-C
51 30%
94%
50,00%
50,009:
40.28%
50.565
27.78%
49.445
45.000
Fluency
(gporchooeen766 Ranking)
1.2 Euglish Fluency
NARL
14.67%
1696
0.000S
0.67%
8.75%
0.81%
41.86%
63.53%
55.72%
(eperchoeean766. Sooring)
L2 English Fluency
PCC1
0.0055
0.0332
0,0292
0.0183
-0.022
-0,0588
0.0222
0.0505
0.0532
lspeschoccun702 Scoring)
Continued on next page
41

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMMONN
SALMOON
Quer- Chaf
Cheen2- 78-tnol
Worl.M EM
ML-
GAMAX
La English Prosodic
LLM-CT
26 39%
35,00%%
46.67%
$2.50%
34,449
SLIIS
34.44%
49.72%
42.22%
lspecchocea7622
Ranking)
L.2 English Provadic
NARA
39.43%
94.75%
11.27%
1.21%
71.47%
3.10%
11.98%
94.621
82.91%
(sperchoeesn762
Scoring)
L2 English Prosodic
PCCT
0.0435
-0.1747
110422
0,00775
0,1446
0.0201
0.0461
+0.1555
CLO973
(specchoccun Scoring)
Prosody
LLM-CT
49.81%
32,8308
48.269
47. 10%
21.62%
51,74%
54.85%
47.10%
5:41%
Naturalness
Lexica))
Prosody
LEM-CT
53.44%
31,6888
46.56%
46. 1899
25.959
50.76%
43.51
40.249%
4.30%
Naturalness
(ProsAudit-
Protosyetak)
Spoof Desection
LLMCCT
41.00%
19.00%
55,00%
14,00%
13.00%
23.00%
19.50%
69.50%
0.50%
(ASVspoof2015))
Spoof Desection
LLMct
49.00%
4,50%
63.00%
38,009
27.00%
32.50%
21.00%
64.00%
0.00%S
(ASVxpoof2017)
Deep Recogattion
11M-CT
40,00%
45.00%
(DELP-VOICE)
Enhancement (LibriTS
LLMET
50,30%
46.50%
55.00%6
56,3016
20.00%
11.00%%
TestClean, WHAMM)
Robocall
73.33%
0.009
26.67%
100.00%
Recugnitio
(CallHome)
Frand Robocall
1.M-CT
47.37%
10.53%
D.00%
73,689
$7.895
63.169
63.169
5.269
100LD0SS
Fraud Robocall
LLM.CT
94.87%
51.28%
1000005
20.50%6
51.28%
61.5416
30.7718
20.721
CLOOPSS
Recognition (Robocall)
Language
92.18%
6.28%
Mdeiiiication
(Noxt Nnxuiingua10r)
Language Idemitication (NoxForgee
LLM-CT
95.50%
13.50%
22.00%
8.505
$4.50%
93.000%
18.0095
6.009%
CLOCKSS
Age Classification (Common Viice)
LLMC
0.50%
19.75%
22.75%
21.00%
26.00%
35.50%
21.00%
23.50%
17.75%
Gender Recagnition by (Common Voice)
LLM-CT
1.00%
81.50%
63.00%
$7,008
$3.50%
97.90%
35.00%
$1.00%
63.00%
HEAR Speaker
NARL
50.40%
42.10%
35.90%
42.50%
3.500
0.40%
22.00%8
23.20%
26.20%
(LibxiCount)
HEAR Speaker
ACC1
17.53%
16.93%
13.20%
17.22%%
10.26%
13.26%
5.51%
13.80%
17.48%
Count
Idemiffiction
(LibeiCount)
Continued on next page
42

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMMONN
SALMOON
Quer- Chaf
Cheen2- 78-tnel.
Warl EM
ML-
GAMAX
Mulii Speaker
LLMcT
46.30%
23.50%
36.00%
30,50%
57,00%
51.00%
32,50%
39.00%
48.50%
Speaker Counting (LIbiTTS--
1EM-CT
50%
6.009
17,509
TeaCClean)
LLMET
48000%
54,50%6
43.50%
52.0016
19.0095
24.00%
13.00%
(LibriSpeech- TessClean)
Speaker (LibriSpeech-
LLM-C1
46.00%
36.00
00.9
19.00%
46,009
47.00%
54.00%
21.00%
33.00%
23.00%
TeatOher)
Speaker Verification
TEM-C
48.00%
38,509
54,00%
58. 58.509
32,500%
51.00%
24.00%
19,50%
7.00%5
SUPERB Speaker Diarization (Libri2Mix: Test)
NARL
95.00%
99.50 30.5
1000005S
100,00S
99.50%6
98.0016
97.5016
100,00%
00.50%
Speaker
74.81%
Diarization
(Libri2Mix: Test)
SUPERR Speaker Verificatoon
ILM.CT
45.00%
44.00%
45.00%
$1,00%5
49.0095
51.00%
49.009
44.00%
45, 0.0%
(SUPERR
Set)
Noise Detection ILJSpeechh Caussian)
LLMCT
45.50%
18.50%
50.00S
47.50%
42.00%
49.00%
39.00%
41.00%
21.50%
Norie Detection (LJSpeonch, MUSAN-Musie)
LLMCT
52.00%
8.50%
$2,00%
47.00%
50,50%
47.00%
44.50%
33.00%
6 6.00%
LLM-CT
47.00%
49.00%
50,50%
(LISpezch. Musse-Noise)
Noise Detection
LLMCCT
48.00%
13.50%
$3,00%
36.00%
50.50%
47.00%
45.00%
50,50%
51.00%
Specc))
Neise Detection
LLM-CT
46.00%
32.50%
45.50%
57. 50%
42.00%
$4.50%
$2.005
37,50%
3.50%
(VCTK.
MUSAN-Musio)
LLM.CT
(VCIK. MUSAN-Noise)
Noise Detection
LLMcT
45.00%
51.50%
43.50%
52.50%
46.50%
57.00%
44.50%
48.50%
56.50%
MUSAN-Speedh
Nosse Detection
LLM-C
$3.00%
15.505
49,008
56.90%
45.50%
54,50%
46.00%
38.50%
11.50%
(VCTK. MUSAN-
Prodiction
(VCTK, MUSAN-
Caussian)
SNR
Prediction
(VCIK.
MUSAN Music)
Continued on next page
43

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTT-AS
SALMOONN
SALMOON
Quee- Chaf
Cheen2- 78-tnel.
Warl EM
ML-
GAMAX
Noise SNR Level Prediction
LLMCT
23.00%
12.50%
9.50%
16.00%
7,50%
19.00%
14.00%
11.00%
0.50%
(VCTK. MUSAN-Nuise)
Nose SNR Level
LLM.CT
23.50%
11.50%
15,00%5
1,50%6
14,00%
20.500
15.0095
10,500%
0L500
Prediction
(VCTK, SpSAN)Speach)
Reverheration
LLMECT
44.30%
11.00%
19,005
34,0096
28.00%6
48.000%
42.501
17.500
CLOOKS
Detection
(LJSpezeh,
RirsNoses-
LargeRoom)
Reverberation
LLMCT
40.00%
4,50%
14.50%
25.00%
31,00%
48.00%
41.00%
10,50%
5.00%
(L.Sperch,
RirsNotaes
MediumRuum)
Reverberation
45.00%
4.50%
9.00%6
19.50%
24.0016
48.0016
14.00%
6.50%
(LJSpezzh,
RirsNoises-
SmallRoom)
Detection (VCT8,
RirsNoises-
LargeRoom)
Detection (VCTK. RireNoises-
MediumRoum)
8.005
17.500
46.001%
10.50%
Detection (VCTK, RiseNoiset-
SutallRoom)
Conection (Librispeech- TestOtber)
31.80%
29.2096
32.80%
20.80%
30.80%
AAVE Speech Recogniton
WERL
21.73%
97.34%
23.99%
31.58%
96.81%
38.55%
34.915
136.50%
102.9214
(CORAAT)
Cado-wwitch
MFR.
215.51%
185:279
130.185
Speech
Recognition
(NTUME
Switching
Speech
Recognition
(ASCEND)
Switching Sperch
28.30%
L3.73%
Resignition
(ASCEND)
Multi-Lingee
73,80%
132.355
25.2196
79.6096
24.969%
99.37%
Spoech Recagnition (MLSS-d)
Multi Speech Recugniton (MLS-cs) (83
65.03%
10012996
9.57%
9.47%
27.9979
17.4716
96.039
57.09%
Mulii- Lingull Recugniion (MLS-es)
WERL
72.49%
333.269
23.45%
16.109
58.829
18.19%
44.905
103.199 19%
101.75%
Continued on next page
44

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMOONN
SALMOON
Quee- Chaf
Cheen2- 78-tnet
Warl EM
ML-
GAMAX
Mulii Lingual Speech Recogniton (MLS- Ei
WER
71.09%
133,86%
26.52%
21.27%
46.27%
19.01%
42.35%
102.25%
102.01%
Mutti Mingi-Lingual Spoech
WER.
85. 85.38%
125.3996
39.1196
31,5456
56.35%
33.169%
46.14%
102.72%
108.84%
Rocogaition
(MLS-I)
Mutti Lingual Speech Rocogaition IMLS- ml)
WERL
744.9%%
120.2966
32.11%
29.57%6
140,0195
42.994%
35.5015
102.0556
101.22%
Mulli-Lingual Speech
WERL
76 79%6
139.049
73,834
$1.539
278.199
101,415
90.115
100.235
105.499L
Recognison (MI-S-p))
Mulii-I Lingual Speech Recogattin (MI--pp)
WERL
73.6%%
136 666
31.889
25.07%
117.33%
24.72%
46.26%
99.71
103.48%
PTBR Speech Recegnntin
WERL
45.75%
248.08%
69.88%
73.911%
233.85%
38.57%
62.931%
639.12%
270.66%
(Common Voice)
SUPERRB
33.96%
69.35%
(LibriSpeech-
TesiCleum)
ASR Teatiher)
WER.
42.14%
91.89%
11.44%
4.31%
79.53%
40.28%
9.17%
111.33
130,00PS
SUPERB OOD
WERL
51.04%
245.85%
216.51%
178.02%
289.25%
178.215
149.15%
225.28%
504.53%
ASR (M))
(Commoo Voice)
SLIPRRB OOD
10.9%%
150,005:
99.2256
99.
141.68%
303
ASR (Ex) (Commoo Voice)
SLPPRR OOD
WFR.
65.71%
122.865
61.11%
62,809%
92.17%
80.21
136.245
137.255
266 72%6
(Spostanecus) (CHIMF6-T(s))
SUPERB OOD ASR (Zh) Vouce)
CERL
29.67%
508,0246
609.28%
310.33%
449.32%
270.635
445.36%
435.33%
1097.411
Target Speaker
WERL
207.019
132.1368
266.266
ASR (AM1-1e41)
Multi-Speaker Detection
LLM-C
46.00%
29.00%
16.50%
22,0055
58.009
52.000%
22.0095
42.50%
$3.009%
(1.hiiSpeoch TesCCiann
Speech Commund
NARL
4.50%
76.40%
3.07%
0.90%
0.67%
0.00%
0.675
L.60%
76 -0%4
Recogaitin
(AudioMNIST)
Speech Command
ACCT
96.2455
Recugniton
(AudiMNISST)
Speech Text Mutching
LLM
86.50%
43.50%
$7.00%
59.50%
64.00%
92.001
50.50%
50.50%
42.00%
ILibxiSpeects Too(lean)
Speech Text Matching
LLM
80.50%
42.50%
56.50%
60.00%
66.00%
92.90%
$1.50%
47.50%
40.00%
(LibriSpeechs TestOtber)
Continued on next page
45

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMMONN
SALMOON
Quer- Chof
Cheen2- 78-tnel.
Ward E.
ML-
GAMAX
Speech Text Matching (LJSpeech)
LLM-CT
83.30%
44.50%
S7 57.00%
60.00%
67.00%
90.00%
60.00%
49.50%
41.00%
Spoken Tomn Detection (1ahiippadss TeaClean)
LLM-CT
76.50%
28.0099
50,008
54,50%
77,00%5
61.50%
$1.00%
30,50%
57.50%
Spoken Term Detection (LibriSpeech TeatOherr)
LLMCT
75.50%
23.00%
$4.50%
50.50%
72.50%
64.00%
59.50%
41.00%
34.00%
Spoken Term Detection (LJSpeech)
LLM-CT
83.50%
33.50%
$7.00S
$3.50%
74.50%
79.50%
56.50%
38.50%
23.00%
SUPERR Keyword Spotting
56.50%
1.0%
2.00%
60.505
(Speach
Commands
V1-Iest)
Query
by Example 4vval
Stutiering
LLMcC1
49.10%
51,00%
50.50%
$0.30%
50,50%
52,40%
55.00%
49.60%
47.40%
Detection (SEP28k)
Viice Disorder Classiflcation
LLM-CT
13.46%
1.2%
13,40R
17.30
13.469
16.35%
18.279
21.15%
6.73%
(VDICED)
Comversation
77.78%
Mauching
Dialegue Act Classifization (SLUE--HVB)
LLMCT
11.08%
0.00%
36,00%
0.92%
5.00%
15.83%
13.175
L833
1.33%
Dialogue Classiication (Tmily Talk)
1LM-CT
29.00%
42,00%
Dialogue Act Patring (Daily
LLMCT
50.00%
3.50%
$1.00%
48.00%
43.50%
46.00%
37.00%
40.00%
25.00%
Talki
SUPERR Enient Classification
1LM-CT
56.50%
0.06%
21.008
5.50%
26.509
26.00%
28.005
3.50%
1.0095
27.00%
Classifiction
(SLURO)AAciON)
Intent
LLMcT
45.30%
11.50%
27.00%
19,00S
68.50%
56.00%
56.50%
4.00%
0.50%
(SLURP Imeent)
Named Emity
NARL
X1.25%
1000005
100,00S
100.00S:
100,00S:
25.000%
93.25%
100.005
100L0095
Localization
(SLIIE-
VocPopuli)
Named Emity
F11
1.4006
1.166S
0.8681
Localization
ISLUE: VexPopuli)
Named Emity
NARL
90.76%
98.37%
95.63%
98.375
95.11%
91.30%
91.85%
97.83%
96.74%
Recognition ISLUE-
VoxPopoli)
Continued on next page
46

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTTEAS
SALMOON
SALMOON
Quee- Chof
Cheen2- 78-tnel.
Warl E.
ML-
GAMAX
Named Enity Recognitin
loUt
0.3333
0,0625
0,0222
ISLUE-
VoxPopuli)
Sareasm Detection (Mustand)
LLM.CT
44.50%
15.00%%
38.50%
46,00%6
42.50%
48.000%
46.50%
43.00%
5,00%5
Similarity (SpokenSTS) (S)
Sporch Sentimett Analysis
LLM-C
58.50%
37.50%
33.50%
35.099
38.50%
49.00%
44.00%
27.50%
27.000
Spoken Dight Anthmmetic (AnfioNNIST)
LLM-CT
43.50%
15,005
SUPERB Siot
XARL
38.00%
88.50%
34,00%
68.50%
51.00%
45.00%
44.0015
99.00%
97.00%
(AudioSnips:Test
SLPPRR Siot
Slot
0.8637
0,8864
0.9325
0.9396
0.8616
0,7704
0.907
0,8939
Filling
Type
(AudioSnips-Tets
Filling
Value
(AndioSnipt-Tes
CER.
Cude Switching
LLMcTT
51.50%
45 45.00%
50.00%
69.50%
27.50%
49.50%
27.50%
28.00%
19.00%
Grummar
Acceprability
Compazison
1.M-CT
48 48.43%
21.08%
35.04%
Detection (WWUGGY)
PoS Estimation
POSL
2.5722
1,792
2,9675
1.199
3.062
2.3091
3.1126
L.5327
1.9635
(LINiTTS))
Estimation with transcription (LibriTTS)
POS
0.9081
1.3507
1,7583
1,0601
2.158
1,7514
2.0463
L2394
1.3911
$2.78%
6.75%
42.0669
28-975
40.80%
50,79R
Grmmmr Acceptability (xBI (sBLIMP)
SUPERB (Co (CoV Testi
18.7884
16,7835
21.7423
Bleu?
Table 6: Task-level evaluation results using LLaMA-3.1-88 -nstutt
Task
Metric
Whippr-
ETU-AS
SALMONN
SALMONN
Quee-
QweaZ: 7B0 Tast.
WorLEM
GAMMN
Chot
Chard
LLM.CT
4.00%
5,00%
7.50%
13,0056
43.00%
16.00%
45.50%
50.00%
0.50%
Clasiffiction
Key
Detection
(GiantSteps Key)
Continued on next page
47

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LITLAS
SALMOONN
SALMOON
Quer- Chaf
Cheen2- 78-tnel.
Warl E.
ML-
GAMAX
[TEAR Music Transcription (MAESTRO)
NARL
95.14%
100,00S
100.00%
100.00%
97.30%
96.22%
$4.86%
100,00%
100.00%
HEAR Music Traascrigtoon (MAESTRO)
TER
99 45%
99.5449
100.0095
99.48%
Instruments Tonic
Classiftcation
(Mridangam Tonic)
Inserument Pitch Clavsification (Nsynt)
LLM-CT
0.70%
0.90%
1.70%
5.40%
1,60%
18.90%
1.20%
0.40%
1.009%
Pitch Exnracion By Lyrics (USD)
S0.009S
73.500%
85.50%
76.50%
97.50%
53.00
Picch Extraction
TEK1
1.36
2,82
4.52
7.38
2.33
2.42
2.71
1.00
3.87
By Lyrics (CSD)
Enocion
L.M-CT
10.00%
0.00%
1.70%
3,3098
21,70%
8.30%
8.30%
1.705
1,70%5
Classificaiion
Songs
(EMOTIPY)
MARBLE
LLM-CT
3,609
8.70%
D.805S
3.709E
9.909%
3.30%
1.80%
7.20%
1.10%
Detection (MTC))
HEAR Music Genr
LLMcCT
13.07%
10.05%
26.136
45.73%
31.66%
37.19%
15.589E
26.131
1.51%
Classification
LLM-CT
0.50%
1.805
Classification
(MTG)
Music Tagging (MagnalagAATiuee)
LLMcCT
3,00%
7.00%
16.00%6
5.0016
3.5016
14.50%
9.00%
75.0%%
MARBLE Music Tagging (MTG)
LLMcCT
5.20%
3,00%
1.40%
0.9096
4,500%
4.801
2.60%
3.80%
1.1055
13.40%
5.40%
Classifiation
(FMA)
HEAR Percussion
LLMCT
2.98%
3,82%
10.00%
20.7445
16.48%
29.26%
25.4216
22.02%
11.84%6
[Beijing Opera
Percussion)
[EEAR Percussion Instruments
LLMCT
0.10%
5.00%
7.90%
4.90%
12.00%
11.50%
14.40%
10.60%
0.90%
Classifictioo
[Mridangam
Stroke)
15.30%
21.30%
10.90%:
(Nsynih)
Instrument Combination Recogaition
LLMECT
8.10%
12.00%
21.905
27.90%6
33.6006
25.50%
24.90%8
22.80%
10.20%
(UpeaMMC-2018)
Instrument Source Classifictiio (Nsynth)
LLMICT
9.90%
36.20%
39.20%
48.00%
37.10%
37.99%
38.0015
41.40%
47.70%
Continued on next page
48

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMMONN
SALMOON
Queen Chof
Cheen2- 78-tnet.
Warl EM
ML- LLMMA
GAMAX
MARBLE
LLM-CT
1.50%
6.00%
0.20%
0.00%
7.70%
2.60%
1,30%
3.00%
7 7,00%
Classification (MTG)
Sunging Autonsatic
NARA
99.50%
68.22%
10.3%%
16.14%
17.50%
0.833
23.13%
97.84%
58.30%
MOS Prediction (SingMOS)
Singing Automatic MUS Prediction
MIEL
6.5670
24110
5.3810
33,0060
14,8900
1,3810
11,2760
4.3720
2,3070
(SingMOS)
Singing
MOS Prediction
(SingMOS)
Singing Automsatic
LOCT
0.6150
-0.0780
0,0240
0.0140
0.0820
0,0400
-0.0050
0.0980
0L3330
MOS Prediction
(SingMUS)
Singing Autocnatic
SRCCT
0.5000
0,0100
0.2210
0.0330
0.0870
0.0250
0.0380
0.1870
0.2790
MOS Prediction (SingMOS)
Tricking (ASAP)
MARBLE Beat Tracking (ASAP)
EXRR
Music Beat Tracking (ASAP)
NARL
53.64%
91,82%
50.00%
65.45%
51.82%
40,91%
43.64%
78.18%
95.45%
76.7
98,6640
56, 3060
131,7640
31,3990
81.3490
Tracking (ASAP)
Time
Audio Editing
NAR
57.50%
93.75%
84.38%
84,38
46.88
31,25%
81.25%
87.50%
37.50%
Idemiñication
(Poople's Speech)
Audio Editing Idemification (Poople's Speech)
ACC1
153.6330
283,9050
82,8400
123.3700
165.0880
154,1090
147.6570
650.0000
49.7720
Fale (ASPIRE)
29
07.00%
Audio Deep Pake Detection
LLM.CT
18.50%
34.20%
29.10S
42,00%5
36.00%6
39.7016
42.009E
40.30%
37.90%%
(1J5peech,
WaveFake. MUSDBINEE
Singing Voice Deepdake
LLMECT
10.00%
35.20%
29.80%
29.80
38.60%
29.70%
31.90%
27.10%
40,80%
22.20%
Detection
(ChSVDD.
ACEKiSing, McSinger)
Dunation Prodiction
NAR
$3.00%%
0.50%
49
0.00%
0.005:
46.00%
(2TU
Andio Danation
13:3660
35,6460
2,00S
3330
28.9400
3,245.7400
L.779,4200
62,7500
(NTUML2201)
[EEAR Music Sperch (MAESTRO,
LLMC
65.00%
68.50%
87.50%
99.50%
91.00%
89.50%
22.00%
53,50%
$8.00%
Librispecch)
Sound Elfect Detection (RemFs)
1EM-CT
4.00%
7.67%
14.67%
18,009
17.00%
19.679
17.339
17.00%
16.339
Continued on next page
49

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMOONN
SALMOON
Quer- Chof
Cheen2- 78-tned
Wart E.
ML-
GAMAX
Sperch Detection (LibriSpochh- TestClean)
LLMcT
48.00%
32.00%
46 0075
66 50
45.50%
56.90%
36.50%
46.50%
56 00%
Speech Detection (LibxiSpeectl- TeaDther)
LLM-CT
51 100%
33.009%
46,00%
66. 66.50%
45.000%
55.50%
38.50%
45.00%
$1.00%%
Speech
(LJSpeech)
Children Song
WERL
58.70%
102.70%
104.00%
101.0%%
128.70%
94.90%
155.50%
100.00%
100.10%
Trinseript
Lyric Translation (SingSet)
Sacre Blaur?
2,1090
3,9160
Song Lyne
MER.
1.1993
1.453
3.5036
2,60000
1,1841
1,1502
4,8133
L.175
1.0852
Recogaitio
(SingSet)
MARBLE Vocal Tochniqqe Detection (NucalSet)
LLM-CT
8.50%
2,00%
12,00%
11.00%
7.50%
15.50%
10.00%
9:00
4.00%
Segnnent Rettieval (Clotho)
NARL
Audio Segment Rettieval (Clotho)
loU't
0.3700
0.2970
0.1900
0.2460
0.0570
0.1930
0,1710
0.0000
0.1590
Detection
(DCASF206
Task2)
Multi-hhand Ulderstanding (STARSS23)
36.00%
9.20%
38.0096
43.0016
8.50%
15.50%
Animal
LLM-C1
9.80%
30%
58.50%
66. 50..009
75.80%
34.50%
13.30%
4.30%
15.30%
(WaneSource-
Test)
Bird Sound Detection (WarlirbIkk)
1.M-CC
32.50%
41.50%
75.50%
77,505
78.509
82.50%
75.00%
45.00%
35,50%
Car Emsoton Classification (Cat
LLMECT
10000%
2,00%6
2.00%6
4.00%
4.00%E
16.00%
8.00%
2.00%
S.00%S
Sound
Dalaset V2)
Cornell Dirdcall
LLMCT
10.00%
3.30%
0.00%
3.30%
10.00%
10.00%
13.301
3.30%
13.30%
Ideniiiiction
Environmental
14.00%
8.00%
78.0096
34,0016
5.50%
2.50%
Clasiffcction
(ESCS0-Animala)
Bechive
LLMCC
92.50%
92,0096
56.0096
92.0006
61.00%
70.00%
Classifecatio
(Beehive States)
Domestic
NARL
97.22%
100.00%:
99.72%
100.00%
99.445
99,725
100.005
98.339
99.72%
Sound Event
Detection
IDESED
Continued on next page
50

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whiegee
LIU-AS
SALMOON
SALMOON
Quer- Chof
Cheen2- 78-tned
Warl E.M
ML-
GAMA-
Domestic
Event- baved FIt
0,4800
0,0000
0,0000
0,0000
0,1680
L,0000
0,0000
L.0000
0.0000
Sound
EvvenDetecctinn
(DESED-
Emergencc Traffic
LLM-CT
50%
66.0095
Detection
(1 (lango-Scale-
Audio-dataser)
LLMCCT
67.500%
Sound
Classiffcation
[ESC5O-Exttriar
Nosses)
Environmental Sound
LLMCT
13.50%
7.50%
1.00%
0.50%
56.50%
15.00%
10.00%
0.00CE
6.50%
(ESC50 Inserior And Domestic Sounds)
Environmental
7,009
68.509
8.50%:
Classification
(ESCSI-Namral
Sounds)
Environmental Sound
LLMCT
13.00%
5,80%
2.30%
5.30%
35.10%
7,9016
10.901E
0.50%
4.70%
Classification
(UrbanSoundSK- Utbaa Naises)
Environment Recogniton
LLMCT
7.00%
15.60%
40,40%
21.10%
68.40%
66.70%
38.60%
10,50%
$2.60%
1LM-CT
10.70%
76.60%
73,0005
4.305
43.40%
Environmental
Sound
Classificaion
(ESCC0)
HEAR Vocal Iimiiaiion
LLMET
15.33%
9.00%6
2.00%6
2.17%
17.8896
11.3316
22.854
4.17%
7.67%
Classification
Lmitations)
Spotial Distance
NARL
45.00%
64.00%
81.00 $1.00%
95.50%
80.00%
41.50%
39.50%
92.50%
55.50%
Prediction (Sputial
LibriSpeacho
Spatial Distance
MEDAEL
0,7110
0.6800
1.3150
0,7840
0.6340
0.5910
Prodiction (Spuiial
LiirrSpeoch))
(5DSpeaker)
Sound Pusition Prodiction
NARL
100,00%
100.00%
100.00%
100.00%
100.00%
100.00S
100.00%
100.00%
100.00%
Dimi
Dialogue Emotion
LLM-tt
51.00%
18.00%
28.50%
13.00%
33.50%
51.00%
22.50%
42.00%
41.00%
Classification
Talk)
Grounded
2.205
Speech Emotion
Recogniton (RAVDESS)
Continued on next page
51

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMOONN
SALMOON
Quern Chaf
Cheen2- 78-tnel
Warl EM
ML-
GAMAX
Emocion Change
LLMcTT
5,00%
10.30%
10,00%
11.20%
L,00%
10.30%
3.90%
11.30%
2.90%
(Ravdess)
Emocion Recagnition
LLM-CT
$3,50%
14.00%
20,008
10.0095
43,500
36.50%
40.500
1.50%
8,50%
Recognitio (CREMAD)
SUPERB Emotion
LLM-C1
12.92%
24.58%
13.75%
13.339
70,42%
76.25%
12,50%
13.75%
5. 5,00%5
(RAVDESS)
Covid19 Cough
LLMECT
40000%
18,90%
4,90%
86.40%
9.301E
8.30%
18.60NE
42.10%
GLOORS
Audio
Clasiffication
(Cough
Environmental Sound Classification (ESCCO-Human And Non Speech Sounds)
LLMECT
17.00%
11.0088
9,50%6
3.00%
77.0006
29.0016
19.501
0.00K6
11.00%
Theman Non Spoech Recugnitio (Nonspecch7L. CommonNbioe)
LLM-C1
3.60%
5.70%
45,00%
30.709
27.90%
23.80%
18.60%
17.10%
28.60%
Human Sereaming
$2.50%
62.50%
47.50%
47.50%
67.50%%
(biviroumessd)
Vocal Sound Recogniton (VocalSound)
LLMcCT
32.50%
2.50%
11.70%
4.00%
75.70%6
38.301
22.1018
1.00%6
3.90%
Phoneme Sugmens Counting (Librispeech- wordks
NARL
23.29%
60.40%
37.69%
18.675
9.52%
13.44%
26.81%
4.97%
7.0%
Segment Counting (Libwispeech- words)
ACCT
Segmens Counting (Librispeech- words)
Difli
Segment Counting (Nux.Angoles)
23
14,075
12.30%
46.88%
Phone Segment
ACC1
0.1510
0.1180
0.1900
0.2500
0.0730
0.0950
0.1610
0.1100
QL0430
(VoxAngeles)
Phone Segment
Abs Didf.)
2.2150
5.532.7270
8.1900
6.2270
2,8590
2.7220
51.811 11881186600001 3.9190
2,8570
Counting
(NoxAngeles)
SUPERR
100.12%
102.75%
24.60%
100.5895
100.9665
1.00.755
100.375
Phoneme
Recagnbtin
(LibriSpeech-
TestCleam)
PERLL
Recagnition (LibxiSpeecll-
Teettterr)
Continued on next page
52

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SAL-MONN
SALMOON
Quer- Chof
Cheen2- 78-tnel.
Worl.M E.
ML-
GAMAX
Phonological
LLM-CT
25 25.90%
2.20%
2,00%
5.10%
4,40%
9,90%
3.60%
3.10%
1. 1.00%
Feature
Classificttion
(NuxAngeles-
Consonant Place
Phonolagical
LLMCT
9,509
Feature
Clasiliiation
Manner
Articalation)
Phonolagicl
LLMCT
5.90%
0.00%
2.80%
9.501
5.201
79.901
0.605E
0.10%
Feature
(NoxAngeles
Phone)
Phonologial Feature
LLM-11
50.30%
14,90%
41.30%
38.30%
50,50%
38.20%
47.000%
42,20%
16.10%
Classification
(NuxAngeles- Vowel Frontoess)
21.20%
15.10%
35.209
10.305
25.705
Classificaiion
(NexAngeles-
Phonological
LLM.CT
Feature
Clasiffiation
(VoxAugeles-
Roundedness)
fleteroym
LLMCC
55.00%
25.00%
42,00%
$3.00%
45.00%
44.00%
49.00%
29.00%
20.00%
Differeniation
(etterooymmn))
English Accurncy
1LM-CT
$1.90%
(6petchoeem76..
Ranking)
English Accurncy
NARL
0.94%
tspeschoccun/62.
scoring)
English Accuracy
(speschoccun162
Multi- Multi-liguul
37.90%
46.20%
14.8096
43.201%
20.20%:
Sinillrity
(NuxAngeles)
LLMEC
17.50%
8.50%
6,0058
7.00%6
24.00%6
14.0016
7.50%
37.00%
4.00%
[Accentdb Extended)
Stresk Detection
LLM-CT
15.00%
6.50%
7.00%
18.509
9,50%
20.00%
29.00%
5.00%
5.00%
NARL
28.
Recogattion
(NCCU Corpos Taiwan Mandarin)
Continued on next page
53

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LITEAS
SALMMONN
SALMOON
Quer- Chof
Cheen2- 78-tned
E.
ML-
GAMAX
Thind Tone Sundhi
loUt
0.0420
0,0530
0,0910
1,1460
0.0K30
0,0630
0.1740
0L1430
0.0630
Recogntton
INCCU Corpus
Spoken Taiwan
Mandarin)
Faglish
27
50.60%
$1,905%
49.400%
Fluency
tsperchoocan7102-
Ronking)
L2 English Fluency
NARL
13.59%
74.02%
0.00%
19.52%
2.4216
0.4015
41.18%
14.27%
2.02%
1specchoccan7002
Scoring)
L2 English Fluency (sperchoceon76)
PCC1
0,0210
0.2580
0,0290
0,0370
0.0460
-0,0860
0,0540
0,3110
0.2920
Scoring)
1.2 English Prosodic
ILM.CT
23 500%
36,70%
46.70%
$0.80S
36.10%
47.220%
45.30%
52,50%
24. 20.
(sperchoceun766)
Ranking)
L.2 English Prosodic
XARA
27.86%
92.73%%
01275%
10.77%
26.92%%
2.699%
13.869
82.10%
17.77%
tspeechoccun7662
Scoring)
L2 English Prosodic
PCC1
0.0900
0.3550
0,0430
0.0900
0,4930
-0.0170
0.0940
-0.0670
0.1330
(speschoccun766.
Scoring)
Prosody
LLMECT
57.10%
40.90%
51.40%
52,9096
32,00%6
38.7016
68.2018
74.90%
36.30%
Naturalness
(ProsAndit- Lexicl)
Prosody
LLMECT
61.50%
41.00%
53,80%
50.40%
36.30%
39.2016
39,2018
75.20%
36.30%
Naturalness
(ProsAndit-
Protosystak)
Spoof Detection
11M-C1
19.009
13,00%
73.00%
(ASVspuo(2015)
Speof Detection
LLM-CT
48.50%
11.00%
68.50%
42,50%
28.0096
35.500%
27.500%
66.00%
10,50%
(ASVspoof2117)
Deep Recognitio
(DEEF-YOICE)
Enhancement Detection
LLM-11
52.00%
33.00%
30.00%
50.509
$9.00%
58.00%
35.00%
54,50%
45.50%
(LibriTSS TesiClem. WHAM)
73 30.30%
D.0R%
Recogaition (CallHome)
Fraud Robocall
LLMcCT
47.40%
0.00%
0.00%
23.700%
42.10%
63.20%
47.0015
47.40%
100L0015
Recogniton
Fraud Robocall Recagnition (Robocall)
LLM-CT
94.90%
46.209 20.0
100.00%
20.50%
74.40%
61.50%
33.30%
38.50%
0.00%
Language (ostiiggu107)
10.48%
17.50%
26.025%
54.325
4.70%
12.34%
Continued on next page
54

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMMONN
SALMOON
Quer- Chof
Cheen2- 78-tnel.
Warl EM
ML-
GAMAX
LLM-CT
96.00%
12.00%
21,00%
8.50%
$4.50S
93.00%
17,50%
5.50%
0.00%%
Mdnntfication (NoxForge)
Age Classitication (Commoo Vaice)
LLM-CT
12.80%
24.50%
23.30S
20,805
24.50%
36.50%
21,80%
23.50%
15.50%
Gender Recognition by Voice (Cummon Vosce)
LLMECT
82.005
57,00%6
53.50%
97.501%
51.00%
64.50%
HEAR Speaker
NARA
32.80%
56.70
45.30%
47.00%
10,70%
9.60%
37,90%
36.70%
24.80%
Ideniification (LibriCuunt)
HEARR Speaker
ACC1
20 39%
24. 949
16.64%
20.9455
12.215
15.60%
11.115
17.69%
17.42%
(1 (1ibiCCmnt)
Multi Speaker Detection (VCTK)
LLMCt
49.50%
32.50%
36.00%
30.50%
50.00%
51.00%
30.5015
22,50%
42.00%
18.00%
13.50%
17.50%
(LibwiTTS-
TesiCleun)
Speaker
LLM.CT
56.50%
28.50%
45.00%
58.0095
43.500
63.00%
16.505
36.00%
34.550%
Verificttion
TeahClaa))
Speaker Verifictioon (LibriSpeechh TestOtber)
LLMCT
48.00%
29.00%
19.00%
60.00%
48.00%
70.00%
19.00%
41.00%
33.00%
Speaker Verificaiion
LLM-CT
55.00%
29.50%
$4.00%
59.50%
44.00%
60.50%
23.00%
33.50%
36.50%
Speaker Durrization (1 (Tili)MissTeel)
NARL
100L00SS
SUPERB Speaker (Libxi2Mis Test)
DERL
SUPERR Spesker Verification
LLM-CT
$2.00%
38.00 00.9
45.00%
$2,00%
$2.00%
59.00%
43.005
$2.00%
00%%
(SUPERB
Nosse Detection (LJSpeoch,,
LLMeT
45.50%
18.50%
50,00%
40.00%
42.00%
49.00%
45.00%
26.50%
31.50%
MUSAN- Caussian)
Nosse Detection (LISpeech, MUSAN
LLMC
$3.00%
23.00%
$2.00%
47.00%
50.50%
47.50%
50.00%
31.50%
37.00%
Noise (1JSpeach, MUSAN-Muise)
50,50%
$7.00%
50,50%
48.00%
Nose Detection
LLM.CT
45.00%
22.00%
53.005
36,0095
50.50%
47.00%
46.50%
50.00%
51.00%
ILJSpesch. MUSAN-Speach)
Noise Detection
LLM-Ct
47.50%
39.00%
45.50%
$7.50%
42.00%
58.50%
56.00%
37.50%
39.000
(VCIK.
MUSAN
Noise Detection
45.50%
56.50%
44.505%
33.50%
(NCTK.
MUSAN-Noist
Continued on next page
55

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTT-AS
SALMOONN
SALMOON
Quer- Chaf
Cheen2- 78-tnel
Warl E.
ML-
GAMAX
Noise Detection
LLMCT
46.30%
50.00%
43.50%
$2.50%
45.50%
57.00%
46.50%
48.50%
56,5 50%
(NCTK.
MUSAN-Speeod
Nose Detection
LLM-CT
$3.00%
24.50%
49,008
51.50%
45.50%
54.50%
52,50%
36.00%
21.00%%
(VCTK, MUSAN-
SNR
Prediction
(VCTK, MUSAN-
Caussian)
21.50%
Prediction
(VCTK.
MUSAN Music)
Noise SNR
50,009
10.50%
15.50%
15.00%%
(VCTK, MUSAN-Noke)
Noise SNR Level (VCTK. SpscN-Spesch)
LLMCT
24.00%
15.00%
16.00%
10.00%
17.50%
21.90%
18.00%
15.50%
13.50%
Reverberation
LLMc1
46.30%
38.50%
45.50%
$7,008
$4.50%
48.00%
47.000
49,00%
27.50%
(LISpeech.
RirsNoisgs-
TargeRoom)
40.50%
$3.500
49.505
20.005
ILJSpezch,
RirsNoiset-
MediumRoom)
LLM.CT
Detection
(LISpeech.
RissNosses-
SmallRoom)
11M-CT
63.0005
45.00%
Detection (VCTK. RirsNoises-
TargeRoam)
1EM-CT
$3,0005
47.00%
27.000%
Detection (VCTK, RirsNoises-
MediumRoom)
Detection (VCTK, RirsNoises-
SmallRoom)
(Librisprech- ToatCher)
AJVE Speach (CORAAT.)
WER.
21.70%
97.30%
24.00%
31,60%
96.80%
38.609
34.905
136.5098
102.9055
Code-switch Speech
MERI
424.58%
215.51%
243.50%
185.29%
165.54%
116.88%
172,0096
130.18%
193.47%
Recogaition
(NTUML2021)
Cedo-Switching Speech
NAR
88
0.00%
69.40%
Rocognitin (ASCEND))
Continued on next page
56

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMMONN
SALMOON
Quee- Chaf
Cheen2- 78-tnel.
Warl EM
ML-
GAMAX
Cude-S Swiiching
ACC1
28.30%
50.57%
58.14%
14.29%
44.31%
60.80%
10.92%
13.73%
9.84%
Recognition
(ASCEND)
Mutti- Mingi-Lingual
WER.
73.80%
132.35%6
34.40%S
25.2116
79.6096
24.966%
49.566
99.3719
105.8156
Spoech Recogaition
(MLS-4e)
Mutti Lingual Speech Recogaition IMLS on3
WERL
65 65.03%
10012946
9.57%
9.47%
27.97%%
17.47%
17.23%
96.0316
57.09%%
Mulli-Lingual Speech
WERL
72.49%
123.269
23.45%
16.10%
58 829
18.19%
44.90%
103.195
101,758
Recugnison (MIS-es)
Mulli-I inguul
WERL
109%
133.866
26.508
21.279
46.279
19.015
42.35%
102. 102.2555
102.01
Speech
Recogattin
(MI)-FF)
Mulli Lingual
WERL
85.38%
125.29%
39.11%
31.54%
56.55%
33.16%
46.14%
102.72%
108.84%
Speech Recogation
Mulii Lingual
WERL
74.96%
120.29%
32.11%
29.37%
140.01%
42.96%
55.56%
102.05%
JOL.22%
Sperch Recognition
Mutti-Liegual Speech
WER.
139.045
101,4195
Recognition
(MLS-pl)
Lingu-Li Speech Recognition
WERL
99.71%
IMLS-p0
45.75%
639,
Recognttin (C (Common Vaice)
SUPERB ASR iLibxiSpecet- TeoClean)
WER.
33.96%
96.28%
15.119
2.79%
69.35%
36,70%
6.87%
103.67%
116.6695
ASR (LibxiSpeechh TestOtber)
WERL
42.14%
91.89%
11.44%
4.31%
79.53%
40.28%
9.17%
101.03%
130.00%
SLIPERR
178.025
289
149.158
504,
ASR (Ar) (Common Voice)
SUPERB OOD
WER.
10.9%
150000SS
98.90%
99.2256
100.839
75.08%
99.38%
141.68%
303 28%6
(Cominon Volce)
SUPERB OOD
WERL
65.71%
122.86%
61.11%
62 62.80%
92.17%
80.21
136.249
137.25%
266. 72%
ASR
(Spontaneous) (CTIIME6-TT Text)
SUPERB OOD
CERL
29.67%
508.02%
609.28%
310.33%
449.30%
270.639
445.368
435.339
1097.419
ASR (7hm (Common Voice)
187.15%
132.1388
266.2698
32%
(AMl-test)
Continued on next page
57

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMMONN
SALMOON
Quer- Chaf
Cheen2- 78-tnet
Warl EM
ML-
GAMAX
Mulii Speaker
LLM-T
49.00%
31.50%
16.50%
21.50%
50,50S
52.00%
20.50%
33.50%
48.50%
(LibriSpeech-
TesiCleum)
Speech Command
NARA
4.93%
76.30%
2.95%
0.00FE
0.933%
0.00KS
0.53%
5.20%
75.87%
Recogotiio (AudioMNIST)
Speech Command
ACC1
0.5890
0,8300
0,9670
0,7410
0,9620
0.7710
0,9340
0.1000
0.2430
Recognttion
(AudioMNIST)
Speech
Matching (LibriSperch- TesCCleun)
Speech Text Matching (1 (1ihiippaddh Teatitirr
82.0095
56.50%
72.009%
45.00%
18.50%
Speech Text Matching (LISpezch)
LLMcT
82.30%
32.00%
59.50%
55.50%
64.50%
76.00%
49.00%
47.50%
16.00%
Spoken Term Detection (1ibrippechh- TestCleun)
LLMCT
77.00%
29.50 50%50%
$4.90%
53.50%
$2.50%
62.00%
$7.00%
48.50%
39.50%
Spoken Tema Detection (patiippadds- Teatiher)
LLMET
77.00%
27.50%
50,008
50,0095
48.5009
66.001%
55,50%
40.00%
39.50%
Spoken Term Detection (LJSpeech)
LLM-TT
83.50%
28.50%
$3.00%
$1.50%
47.00%
79.50%
50.00%
40.00%
31.00%
SUPERR
75.50%
4.50%
30.508
31.50%
30.005
47.50%
27.50%
Keywurd Spotting
(Speach
Commands
Query by Example (Qusst14-Eval)
37.550%
Stutiering
LLMcCT
49.30%
45.40%
50.50%
$0.30%
50.20%
$2.60%
58.90%
49,80%
48.50%
(SEP28K)
Viice Disorder
LLM-C
16.30%
13.50%
13.50%
17.30%
19.70%
17.30%
20.209
22, 10%
13.500%
Classification (VDICED)
Comversation
13M-CT
77.80%
9.30%
$1.90%
1.90%
Matching
Dialegue Act Classification
LLMCt
21.80%
18.90%
22,80%
3.30%
9.40%
20.80%
17.30%
4.90%
4.10%%
Dialogue Classifiction (Tuily Talk)
1LM-CT
43.50%
Act Paiting (Daily Talki
LLM
51.00%
16.00%
$3.50%
48.00%
43.00%
48.501
39.501
50.00%
39.0016
SUPERB Insent Classification (SLURP)
LLMICT
33.00%
7.50%
13.50%
3.00%
31.00%
26.50%
20.5015
4.00%
2.00%
Continued on next page
58

---

Published as a conference paper at ICLR 2025
Task
Metrie
Whilgee
LTU-AS
SALMOONN
SALMOON
Quee- Chaf
Cheen2- 78-tnel.
Warl E.
ML-
GAMAX
Intent
LLM-CT
24.30%
12.50%
16.50%
16.00%
27.50%
67.50%
33.00%
7.00%
10.00%
Classffiation
(SLURP Action)
Insent
LLM-CT
56.50%
12.005
21,008
15,50%
48.500%
43.00%
41.50%
6.00%
6.50%
Classiftatiion
Named Emity
Localization
ISLUE- VoxPopuli
Named Emity
Localization
(SLUEE- VoxPopuli
Namad
43.48%
84.785%
44.57%
64.13%
89.13%
Recogaition
(SLI1E- VotPopula
Named Emity Recogaition ISLUE:
loUt
0.0190
0.2640
0.2120
0,1180
0.0290
0,1190
0.6930
0,0500
0.0000
VotPopula)
Surzasm Detection (Mustand)
LLMct
38.50%
36.50%
41.50%
46,005
44.00%
48.00%
47.50%
45,50%
36.00%
Semantic Sinillaity (SpokenSTS)
11M-
40%
50
22,00%
$4.00%
Speech Sentimest
LLMcT
39.00%
31.00%%
33.50%
36.00%6
38.50%
49.0016
42.501
28.50%
29.50%
Analysis
Spoken Digit Arithmeti (AulioMNIST)
1LM-CT
43.50%
$1,505
26,50%
SUPERB
40.50%
82.50%
70.00%6
41.50%
58,0016
63.501
95.00%
92.00%
Filling
(AudieSnips-Test
SUPERB Size
Slot
0.8570
0,5000
0.9230
0:9210
0,8120
0.9270
0.9960
0:7770
0U9020
Filling
(AudioSnips-Tee
SUPERB
0.3540
0,8500
0.2370
0.2970
0,4600
0,2390
0,5510
0.6690
0.4160
Filling
Value
(AudisSnipseTes
CER.
Cade-Swicching
51
49.50%
45.550%
45.50%
32.00%
Semantic
Grummar
Acceptabiliny
Campuzion
(C57S-hhen))
Noace Word Detection
LLM
63.00%
25.60%
55.00%
58.10%
32.20%
61.80%
63.50%
42.50%
19.90%
(WLUGGY)
Ratimation (LJIIITTS)
POSL
1.2700
1.5327
PuS Estimation with transcription (LibriTTS)
POS
0.9081
1.3507
1.7583
1.0601
2,1580
1,7514
2.0468
1.2994
1.3911
Grammr Acceprability (sBLIMP)
SUPERR
17.3720
0.1077
18.7884
16.7835
21.7423
Translation
Bleut
(CWMAST2-Tex))
59

---

Published as a conference paper at ICLR 2025
D HEAR & MARBLE EVALUATION RESULTS
Tables and show the evaluation results for the HEAR and MARBLE tasks, respectively, as collected in Dynamic-SUPERE Phase-2. Importantl, we replaced the datasets for some tasks to address licensing issues and reduced their size to enable more efficient inference. Furthermore, we removed certain tasks because they overlap with existing SUPERB tasks. Consequently, these results not directly comparable to those evaluated the original HEAR and MARBLE.
Table HEAR (audio) Results. LLM-C stands for LLM Classification, NAR stands for Not Appli- cable Rate, TER stands for Token Error Rate, and ACC stands for Accuracy.
Task
Metric
Whisper- LLaMA
LTU-AS
SALMONN
7B
SALMONN 13B
Owen- Andio-
Qwen2- Audio BB-Inst.
WarLIM
ML- LLaMA
GAMA-IT
Boshive States Clasifiction
Brahion Stores
LLM-C1
43,00%
35,00%
54.0%%
42.0%%
19.00%
42.50%
32.00%
37.50%
19.00%
Ensiies Recogaition
CREMAD
LLM-CT
6.70%
9.30%
15,00%
18.10
62,50%
61.10%
25.009
12:40%
13.20%
Claxiifestion FSC-50
Envirenmental Sound
L1MC1
480%
25.30%
61.00%
73,00%
68,80%
43.90%
9.80%
0.40%
35,30%
Lungugg Meniification
Noxtinguz107 Topll
LIM-CT
92.18%
1995
6.28%
11.11
18.00%
36.119
1,58
8008
0.106
Gamee Clasilication
15MIRM
LIM-CT
33
Speech Claseficntin
59.50%
Music
1000005
96.76
96.76%
100,009
MAESTRO
8.3438
6,2500
Percussion Instuments
Classifcatioo Beijing
LIMET
10.17%
20.766
16.10%
27.12%
24.15R
22.05%
11.
Percussio Instnument
Percussias Inteuments
Stokk Clavilication
LLM-C1
0.10%
3.80%
7.60%
4,90%6
11,50%
10.90%
83,00%
10.00%
0.10%
Mildangam Strok
Percussion Instrments
Toonic Clascticttion
LLM-C1
2000
2.50%
12.70%
13-40%
11.30%
11,80%
12,80%
12.50%
S. 5.70%
-Midangam Strike
Sound Event Detection
DCASE2001 Task2
LLM-CT
0.0%
0.005S
21.43%
35.71%
7.1498
14.29%
21.495
21.45%
7.144
Speaker Cours Identification
NAR ACC
30.00%
42. 1.90%%
35.90%
42.50%
3,50%
0.40%
22,00%5
23.200
26 26.20%
LLibeiCount
17.53%
16.90%
13,20%
17.22%
10.26%
13,76%
5,51%
13.80%
17.48%
Claveiicatios
L1M-CT
18667
17.505
Table 8: MARBLE (music) Results. LLM-C stands for LLM Classification, NAR stands for Not Applicable Rate, and MSE stands for Mean Squared Error.
Tisk
Metric
Whisper- LLaMA
LTU-AS
SALMONN
7B
SALMONN 13B
Ques- Audio-Chaat
Quen?- Audio-7B-Last.
WavLLM
MU- LLaMA
GAMA-IT
Key Detection .Giuntsteps (Key)
LIMcC1
3,00%
0.50%
1.50%
LO0S
3.00%
17.00%
2,00%
0.50%
0.00%
Music Tagging
Junee-J
LIM-C1
0.18
0.30%
0.0%%
0.005
2.08%
0.305
0.10%
0.50%
0.10%
Music Tagging
MiggnaaagATun
LIM-C1
5.509
1.00%
3.50%
9.00%
4 4.50%
1.50%
1.00%
3.50%
7.50%
Geune Clavalication
MMGG- Jumendo
LIM-C1
0.08%
1.60%
1.00%
0.20%
3 1.20%
0.105
0.20%
0.30%
0.00%
Emation Detection
Junendo
LIMC1
0.08%
1.50%
0.00%
0.005
2 2.70%
0.805
0.805
0.80%
0.10%
Beat Tracking
NAR MSB
100,00S
10010S
100,005S
100000S
100,005
100,000
180.009
100:00
ASAP
Detection VocalSet
Vocal Teclnique
LIMC1
8.00%
1.00%
11. 0.00
6.50%
6. 5.50%
15.90%
8.00%
8.50%
2,50%
Instament Classileation
June-do
LIM-C
0.40%
2.30%
0.0%
0.005S
6 6.40%
1.40%
0.209%
0.60%
0.30%
60

---

Published as a conference paper at ICLR 2025
E LLM-BASED EVALUATION
Here, we list the prompt templates used in the evaluation. Figure iillustrates the prompt for classi- fication task evaluation, while Figure shows the prompt for regression task evaluation. In addition to GPT-4o, we also employed open-source L.MMs to ensure better reproducibility. Tables and 10 present the performance evaluations of the proposed prompts classification and regression tasks, respectively, comparing their correlation with a human-labeled evaluation set. We found that GPT-4o and 3.LMMA-3.1-705 I-ntut-nstt achieved very high accuracy in both classification and regres- sion tasks. This demonstrates that our evaluation using GPT-4o is reliable and that researchers can leverage LLaMA-3. I-stuc- to conduct fully reproducible evaluations.
Performance evaluation of the proposed post- post-processing prompt on classification tasks using numan-labeled evaluation set. Accuracy determined by agreement between the model output and the corresponding human label.
Table 9:
Model Name
Accuracy
Correct Count
gpt-4o
98.67%
148/150
gpt-4o-mini
98.00%
147/150
spt-ol-preview
98.67%
148/150
gpt-ol-mini
99.33%
149/150
Ilama-31170B-Inntruue
96.67%
145/150
1lama-3.1-8B-Intruus
92.67%
139/150
llama-3.-3B-IIntrruc
86.67 %
130/150
Vlama-3.2-1B-Insttee
61.33%
92/150
Table 10: Performance evaluation of the proposed procesrocessing prompt regression tasks ing a human-labeled evaluation set. Accuracy determined by exact matches between the post- processed model output and the corresponding human-generated post-processed output.
Model Name
Accuracy Correct Count
got-4o
92.00%
138/150
gpi-4o-mini
86.67%
130/150
gpt-ol-preview
86.00%
129/150
gpt-ol-mini
89.33%
134/150
3lama-3t1-0B-Instruut
90.00%
135/150
1lama-3.1-BB-nnstuuct
82.00%
123/150
llama-3.2-3B-Instrud
61.33%
92/150
1amma-3.2-IB-Insttee
39.33%
59/150
61

---

Published as a conference paper at ICLR 2025
Classification Post-procesiing Prompt:
You will be given question, corresponding correct answer(s), and response from model. The model's response a reply the question. Your task to judge the "Model's Response" aligns with the "Ground Truth Answer" based on the "Question."
Please strictly follow the guidelines below:
Briefly explain the reasons for your judgment.
Answer with the format "Result: <YES or NO>" at the end.
Output "YES" the response aligns with the ground truth answer; output "NO" the response does not match the ground truth answer, selects incorrect or irrelevant options, provides more answers than required.
questions would single-choice or multi-choice:
For single-choice questions, the model's response should contain one and only one answer. If the model's response selects more than one answer or does not clearly indicate single answer, you should mark it as incorrect and output "NO." For multi-choiec questions, the model's response must exactly match all applicable correct choices. If the model's response selects too many, too few, or any incorrect answers, you should mark as incorrect
output "NO."
Since the question is short answer, the model's response does not need to mention the content the question. You only need to check if the model's response has the same meaning the ground truth answer(s).
Input Format:
Question: finstruction)
Ground Truth Answer: {label}
Model's Response: (response)
Figure 6: The prompt used for post-processing the classification tasks inference outputs in the study.
62

---

Published as a conference paper at ICLR 2025
Regression Post- post-procsssing Prompt:
You will be provided with "instruction," "ground--ruth label," and "model output." Your job is to extract the value from the given "model output" and return post-processed "model output". Provide your output with just the post-proe posteproeesee output without an explanation.
[Task Description]
Analyze the "Instruction" and "ground-truff" label" to understand the desired output format. Post-process the "model output" so that it matches format of the "ground-truth label" and is numerically comparable to it.
[Guidelines]
the "model output" can be adjusted to match the format of the "ground-truth label," return only the post-processed result.
If the "model output" is already in an appropriate format, return it as is without changing it. If it not be adjusted into the correct format, does not clearly indicate any of the candidates, is ambiguous, or implies more than one answer, return "N/A."
Do not confuse the "ground-truth label" with "model output"; you are only supposed to use "ground-truth label" to understand the desired output format.
Do not tamper with the original content of "model output": you are only supposed post-process the format of "model output" evaluated against each other directly.
matches the format of "ground-trutt truth label" and can be compared or
The post-processed output should only be a number in numeric form.
Below you have three examples. Please consider their patterns to understand your task better:
[Example Inputs]
"instruction": "Please identify the total times code-switching in this wavefile." "ground-truth label": 8
"model output": "There instances of code-switching in this wavefile; it is purely Mandarin
speech." "
Your response: 0
[Example 2- Inputs]
"instruction": ""Listen the audio and the clarity and accuracy of pronunciation considering
precise phonology, pronunciation errors, and overall comprehensibility. Use Arabic numeral from to 10 for your answer.""
"ground-trutt truth label":
"model output": "The audio is of a man speaking, in a Chinese, saying, "I has find different back door". The accuracy of the pronunciation is 90%.."
Your response:
[Example 3- Inputs]
"instruction": "Please predict the spatial distance(0.5mm-455m) based the given audio."
"ground-ue truth label": 119993333332990
"model output": "Based on the given audio, the spatial distance is likely between 0.5m and 4.5m."
Your response: N/A
[Inputs]
"instruction": (instruction)
"ground-truth label": {label}
"model output": (model output}
Your response:
Figure 7: The prompt used for post- protessocsssing the regression tasks inference outputs in the study.
F CALL FOR TASKS
Here provide material from the call for tasks. Figure shows a task proposal that ask each contributor to initiate. Task proposers need to provide a high-level description of the task, explain its importance and challenges, and list the datasets they plan to along with the dataset licenses. Figure [ presents the standard README format for each task. the README, task proposers include the task introduction, its challenges, the dataset, and the evaluation metric. Importantly,
63

---

Published as a conference paper at ICLR 2025
also ask them to provide table of tfate-of-the-ar performance the task, even if not achieved by universal spoken language models. These results can serve as a reference for researchers to better understand the task's 's difficulty and how it has developed so far. Then, Figure TO shows the JSON format we require to record task information. For each task, our evaluation pipeline uses this JSON file input automatically download data from Huggingface, formatting it for subsequent use.
Part-of-Speech Estimation
This task to tag Pare-of-Speech (POS) tags an utterance. This task significant because tests the model's profidency in understanding and processing the syntactic structure of spoken language. Accurate POS tagging from acoustic data can enhance various applications such as speech recognition (ASR), spoken language understanding (SLU), and machine translation (MT).
The task is part of the metatask #1440
This task can provide greater insights when combined with our other task, #142, which focuses on word-boundar detection. the model performs poorly in both #142 and this task, we can conclude that the model's inability perform POS tagging is due to fallure learn word boundaries. the model performs well task #142 but poorly in this task, we can infer that the model lacks syntactic ability.
Task Objective
Accurately tag word with POS tags from audio input.
Datasets and Evaluation Metric
Dataset
We plan to use LibriTTS test-other split [dataset] [paper]
chose
speech split
transcriptions.
For simplicity we select sentences with 10-15 words. The detaset is under the CC BY 4,0 license.
Preprocessing
Use Stanza attain word-POS pair from transcriptions.
We use Stanza which uses Universal POS set. [universal POS].
Tag adjective (ADJ), adposition (ADP), adverb (ADV), auxilliary (AUX), coordinating conjunction (CCON). determiner (DET), interjection (INT)) noun (NOUN),
(NUM), particle (PART), pronoun (PRON), proper
(PROPN), subordinating conjunction (SCON),
The reported performance on POS tagging is around 96.70% to 98.28% on official test sets for POS tagging
sstanza performance).
Eveluation
WER: word error rate of transcriptions
POS Accuracy: accuracy POS tags utterance
Per POS Tag Accuracy: accuracy Individual POS tag
Sample Instructions
For each word in the given audio utterance, determine and assign the corresponding Part-of-Speech (POS) tag. The
list of tags includes ADJ (adjective), ADP (adposition), ADV (adverb), AUX (auxillary). CCONJ (coordinating conjunction), DET (determiner), INT] (Interjection), NOUN (noun), NUM (numeral), PART (particle), (pronoun).
PROPN (proper noun), SCONJ (subordinating conjunction), tags followed separate example, when you listen
(verb), Please
transcribed
first, fox', please generate tags
quick
(transcription): quick
(POS):
ADJ NOUN.
For each word in the given audio clip, assign the appropriate Part-of-Speech (POS) tag. The available tags include ADJ (adjective), ADP (adposition), ADV (adverb), AUX (auxiliary). CCONJ (coordinating conjunction), DET (determiner). INT] (interjection), NOUN (noun), NUM (numeral), PART (particle), PRON (pronoun), PROPN (proper noun), SCON) (subordinating conjunction), and VERB (verb). Begin with the transcribed sentence, followed by the POS tags separated by For instance, transcription breakfast PRON
as:
Transcription:
NOUN.
Figure 8: An example of task proposal from task contributor.
64

---

Published as a conference paper at ICLR 2025
Part of Speech Estimation
The Part of Speech (PoS) Estimation task aims to accurately tag words with their respective Pa5 tags from audio Input. This task capabiiit understanding processing the syntactic spoken language. Accurate tagging from acoustic data can significanlly enhance various applications such Automatic Speech Recognition (ASR), Spoken Language Understanding (SLU), and Machine Translation (MT).
Objective
The primary objective is to assess whether benchmark models accurately identily the PoS of words speech utterances. The task
divided into two subtaska:
model generate lags
transcription:
generate transcriptions tags
Dataset
The dataset derived from the LibrITTS corpus, multi-speaker English corpus [LibriTTSoggeSLL LibiTTS-Hugginn Face). To suit the purpose
quotes Yee, he said.) (2) Remove punctuations Select sentences that include to words. We uniformly selected the sentences for each number of words. Consequently, the dataset includes 890 samples. Further details regarding the proprocessing stops, please refer to repository
task, we went through several preprocessing stops follows:
The PoS Estimation task inclucies two variants: each subtask, Pas estimation and PoS estimation with transcription, corresponds to the PoSEstimation, LibriTTS, dataset and the PoSEstimation, LibniTTS sentPoS dataset, respectively. PoSEstimation, LibriTTS, Pos, the model asked generate tags both transcriptions and Poś tags from the audio. Therefore, while the audio samples are shared between the two datasets, the instruction' 'label columns differ.
for PoSEstimation, LibriTTS senttoo, the model asked generate
For more detailed informacion, please refer the dataset directory
version: e3 (d0os00ccdd0002220000a6ee2200
Evaluation Metric
Word Error Rate (WER) PoS are evaluated using WER metric.
WER with transcriptions: the PoS estimation with transcription subtask, certain preprocessing necessary exclude the transcriptions, which are expected to be separated by Y.
State-of-the-an performance
best knowledge, LibriTTS been utilized for Part-af- tagging have been reported show performance between 96.20% and 99.1 15% with conventional Poń teggers (ttps://ttafordddg github lowstaana//errommmacc html).
tagging tasks English
Reference
[1] Zen, Dang, V. Clark, R.A, Zhang, Y. Weiss, Jia,ia, Ja, Y, Chen, Z, Wu, (2019), LibriTTS: Corpus Derived from LibriSpeech for Text-to-Speach. Interspeech.
Figure An example of the README file required in the task contribution.
2
"name": "PoSEstimation bestettiiatioo__iiiiSSS..... LibriiTSS_entPoS""
3
"description": "Generate correct part-of- tags. Also evaluate the performance transcriptions as side task. We word rate metric for both tasks.",
4
"keywords":
"linguistics",
5
"metrics": "word
(PoS)",
7
"word error rate (transcription)"
8
1.
9
"path": "DynamicSupert/
PoS Eww.iittiiiinnn.mm Lirooo__ibiirsss_.m PoS with "version": oaclaseeeeeesss0000744aaaaaaaae0
orrrrisssspssttthtttoocccpppoooo transcription"
11
Figure 10: An example of the JSON file required in the task contribution.
65

---

Published as a conference paper at ICLR 2025
G AUDIO DURATION DISTRIBUTION
Figure [hows the distribution of all audio files in Dynamic-SUPERB Phase-2. Only about 6.5% of the audio files have duration longer than 30 seconds, which reflects an inherent limitation of models that incorporate Whisper in their framework.
10000
Number of Samples
Duration 6.57%
4000
2000
10
Duration (s)
Figure 11: Duration distribution of all audios in Cynamic-SUPERRS Phase-2.
66


Reviews:
[{"review_id": "Br2cGBN3I1", "reviewer": "Reviewer_59Pj", "summary": "This paper builds on the prior work, Dynamic-Superb, by expanding its scope as an audio LLM benchmark to include a broader range of tasks centered on audio and speech understanding. It includes diverse tasks across speech, music, and general audio comprehension. The authors also evaluate several well-known audio LLMs, including SALMONN and Qwen-Audio, on this benchmark, showing that none of the current open-source audio LLMs can yet achieve satisfactory performance across all tasks.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work is comprehensive and well presented. It brings the largest benchmark by far for speech and audio evaluation. It also conducts detailed experiments to assess the performance of several popular audio LLMs on the proposed benchmark.", "weaknesses": "There're a few possible issues that may further improve this paper.\n- What is the motivation of creating this new benchmark? How will it guide the research community in advancing audio LLM research? I think the authors could emphasize this point further.\n- I understand that the number of tasks has significantly expanded in Phase 2 compared to Phase 1. However, what is the primary focus of these additional tasks? Are there specific challenges unresolved in Phase 1 that Phase 2 addresses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhlMRtWhOz", "reviewer": "Reviewer_wDFb", "summary": "This paper presents Dynamic-SUPERB Phase-2, a benchmark designed to evaluate the capabilities of instruction-based universal speech models. By expanding the task count of its predecessor to 180, it becomes the largest benchmark for speech, music, and audio, encompassing a diverse array of tasks contributed by the global research community. This work is poised to drive further advancements in developing universal spoken language models and their evaluation methodologies.", "questions": "My questions are listed above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Dynamic-SUPERB Phase-2 is the largest and most comprehensive benchmark for instruction-based universal speech models. It encompasses a wide range of tasks across speech, music, and audio, all paired with natural language instructions to evaluate models' cross-modal instruction-following abilities. This is a well-motivated and forward-looking approach that aligns with future trends in universal speech models, offering strong guidance for the field.\n2. The paper is clearly written and easy to follow.", "weaknesses": "1. The benchmark lacks tasks for audio, speech, or music generation.\n2. A primary concern is that the evaluation metrics heavily rely on large language models (LLMs) as referees. The reliability of these metrics is highly dependent on the capabilities of the LLMs themselves, which affects the benchmark's robustness and comparability. This reliance may also limit the benchmark's ability to expand to more complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QSHxEvhh1R", "reviewer": "Reviewer_59hW", "summary": "This paper introduces a \"Dynamic-SUPERB Phase-2\", a benchmark for evaluating of audio-in-capable LLMs. This benchmark contains 180 tasks that cover a a large variety of tasks, spanning from music and audio classification to speech recognition, nonce word detection, intent and age classification, etc. These tasks were collected collaboratively. The audio-LLM are assumed to take in text instruction, an audio snippet, and produce text output. The interaction with the model is performed solely by the instruction, no finetuning is assumed. Depending on the task, the output encodes either classification label, regression value, or a natural language string. The benchmark is split into (a) full version, (b) an easier to operate representative sub-sample of tasks.\n\nGiven the benchmark, the authors also run a study comparing several off-the-shelf audio-capable LLMs, such as AudioQWEN, SALMONN, WhisperLLAMA, etc. One particular issue encountered is that various audio-LLMs would produce outputs in different formats, so the authors leverage another textual LLM to \"translate/interpret\" their outputs to the desired format.", "questions": "* The paper text reports that In Figure 3, Whisper-LLaMa performs best in speech recognition and understanding. However, when we look at Table 2, it is way behind SALMONN in phone recognition. Looking at ASR performances across tables, it does seem that SALMONN mostly falls behind on a few non-English transcriptions (SUPERB OOD Asr Zh). I wonder if there is a more nuanced story here than is told in S5.1.\n\n* Table 7 has inconsistent WER reporting, ie \"Children Song Transcript\" has WER not scaled by 100% while it is scaled in other places.\n\n* It would be nice to have the \"N/A rate\" reported for the models.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper tackles perhaps one of the most important problems in the audio-LLM space: the lack of audio-text benchmarks. In contrast to text-LLMs, where sets of hundreds of tasks are readily available, audio-LLM development is hindered by the lack of standard testing suites.\n* I find the organization of the benchmarks to be thoroughly thought through.\n* The paper runs a nice study on comparing off-the-shelf models, thus validating the proposed benchmarks.\n* The paper validates referee model selection against human preferences.", "weaknesses": "* Using GPT-4o as a referee, I believe, jeopardizes reproducibility of the reported results and usefulness of the paper. Imagine that in two years someone wants to compare their model to the benchmark numbers reported in this paper. Would GPT-4o still be around? Would this particular version of it be still accessible? This is even more puzzling, since, according to Appendix E, LLaMA-3.1-70B-Instruct is extremely close to GPT-4o here. It seems to me that using an openly accessible model here would be extremely beneficial.\n\n* Regarding the model evaluation study: it is not clear how sensitive the tested models are wrt the prompts used. It would be very nice to have a small study on that.\n\n* Whenever the evaluated model doesn't follow the instruction, its score is multiplicatively penalized by (1-N/A) or 1.0/(1-N/A) (depending on the metric). This is convenient as allows to have a single number but it totally hides how good models are at instruction-following.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6R7Y3uBgHq", "reviewer": "Reviewer_j6zR", "summary": "The paper suggests a new (milestone of a) benchmark to test the audio analysis tasks of universal speech models. It presents an overview of 180 tasks, contributed by the research community and categorized into a a taxonomy, which all have in common that one or multiple audios, as well as a task prompt are presented to a to-be-evaluated model, which then provides a language response, aiming to solve the given task. The response is then evaluated by a post-processing pipeline, potentially including a \"judge LLM\". Besides providing the evaluation benchmark pipeline, the authors conducted an analysis of current universal (at times specialised) speech models. They conclude that no model excels at all tasks, but strengths and weaknesses can be observed across different models and tasks.", "questions": "What is the justification of using Whisper + LLaMA as the baseline? There should be some explanation why this particular architecture has been chosen.\n\nL.267: I think the phrase for distinguishing sources \"speech is produced by humans\" could be rephrased, as humans could also \"produce\" other non-speech sounds. Speech should rather be connected to the vocal chords.\nL.286: The example of Speech Enhancement to mitigate does not become clear. Above it is stated that the models should produce text as output (which can maybe be interpreted as lists). Under Speech Enhancement for mitigating noise (unless you mean only e..g, SNR estimation) I would expect an audio output.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper addresses an important point challenging the \"universality\" claim universal speech models. The authors claim to have gathered the largest benchmark by a large margin, improving on top of presumably their own work and provide a detailed analysis comparing several current speech models. The benchmark has the potential to have a large impact on the research community and it provides useful insights into the limitations of LLMs.", "weaknesses": "L.371: Maybe it is worth that the authors iterate why they didn't choose a more \"direct\" approach for the evaluation, in which the model is prompted to give a single word based on the fixed set of possible answers. In the case of emotion recognition, it could be \"answer with a single word out of the choices 'happy, sad, neutral,...'\", which would allow a more direct evaluation. Especially in the case of emotion recognition a non-pre-defined set of emotions will also make it difficult for the judge LLM (and humans for reference) to judge how accurate the result is. This might at least be considered as a second evaluation metric, this seems more or less as what is being done for regression, but why isn't this \"extra\" step directly asked from the model being evaluated? For sequences evaluation it is stated that the evaluation of unprocessed outputs \"guarantees consistency and objectivity\".\n\nSection 5.1. It might be difficult to define for some tasks, but a comparison to chance level and/or state-of-the-art task-specific models and/or \"perfect\" scores would seem a very interesting insight. As is, it seems that the evaluation allows relative comparison between -quite elaborate- models, which allows to see which model is better amongst two, but doesn't really allow to see if the models are still doing reasonably well in these tasks and how much room for improvement there still is. For instance in the case of of spatial audio, none of the models are performing better than the baseline, but it not clear, if none of them or all of them are reasonably good at the task. This is somewhat done in section 5.2. but seems to be missing in 5.1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)