Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks

## Abstract

Abstract Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data.
Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions.
However, the absence of a comprehensive evaluation benchmark poses a significant challenge.
We present Dynamic-SUPERB Phase-2 , an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models.
Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks , making it the largest benchmark for speech and audio evaluation.
While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio.
Evaluation results indicate that none of the models performed well universally.
SALMONN-13B excelled in English ASR, while WavLLM demonstrated high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks.
We will soon open-source all task data and the evaluation pipeline.

## 1 Introduction

Recent advancements in large language models (LLMs) have accelerated the development of natural language processing (NLP) (Touvron et al., ; Achiam et al., ; Li et al., ; Anthropic, ; Bai et al., ).
These models can follow natural language instructions, making users quickly adopt them for a variety of applications.
They have been integrated into commercial products, such as ChatGPT (Achiam et al., ) and Claude (Anthropic, ), as well as in the open-source research community, including the LLaMA series (Touvron et al., ; ; Dubey et al., ).
Yet, they are primarily text-based models, meaning they cannot process speech or audio, which are essential for more natural ways of communication and interaction with the real world.

Compared to written text, spoken language has always been a more natural and convenient way for humans to communicate.
Spoken language conveys a wealth of information, including semantics, prosody, emotion, and speaker characteristics, while text is limited to representing semantic information, which can sometimes even depend on the prosodic cues present in spoken language (Lin et al., ).
This highlights the need for universal speech models and explains why automatic speech recognition (ASR) systems using text-based language models are not optimal.
Several attempts have been made to develop instruction-based universal speech or audio models capable of performing various tasks, such as LTU-AS (Gong et al., ), SALMONN (Tang et al., ), Qwen-Audio (Chu et al., ; ), and WavLLM (Hu et al., ).

Despite significant research in universal speech models, evaluating them effectively and comprehensively remains a major challenge.
In NLP, to evaluate text LLMs, some benchmarks consist of a larger number of tasks.
CrossFit (Ye et al., ) includes 160 tasks, BIG-bench (Srivastava et al., ) comprises 204 tasks, and Natural-Instructions (Mishra et al., ; Wang et al., ) provides 1,616 tasks.
For universal speech models, several benchmarks have been proposed but with a fixed and limited set of tasks (typically around a dozen) to evaluate specific capabilities of models (Yang et al., ; Dunbar et al., ; Maimon et al., ).
This is insufficient for evaluating a universal model, as we aim to investigate whether such models can perform a broad range of tasks beyond the limited set of tasks included in these benchmarks.
There is a strong demand for a benchmark that can evaluate these models across various aspects with a wide range of speech tasks, which led to the creation of Dynamic-SUPERB (Huang et al., ).

Dynamic-SUPERB is the first benchmark designed to evaluate universal instruction-based speech and audio models.
As universal models advance, we dynamically expand the benchmark by adding new tasks to provide better guidance for researchers in developing their models.
We have designed a well-structured pipeline that harnesses the collective efforts of the entire community to gather a diverse array of challenging, novel, and creative tasks.
The first generation of Dynamic-SUPERB consists of 55 tasks, covering various aspects of speech (e.g., content, semantics, speakers, etc.), making it the speech benchmark with the most tasks.
However, this is not sufficient to pave the way for developing universal speech models, especially given the complexity and richness of the information conveyed by spoken language.

This paper presents the Dynamic-SUPERB Phase-2.
We collected 91 new tasks with contributions from the global research community, allowing the benchmark to grow more than twice as large as its first iteration.
They cover a broad range of types, and some of these tasks present novel challenges that have not been explored in any previous research.
Besides, previous speech research has typically treated music and environmental audio as background noise to be ignored.
However, these sounds contain rich information and share overlapping elements that complement each other.
Thus, in Dynamic-SUPERB, we also introduced preliminary music and audio tasks from established music and audio benchmarks (Yuan et al., ; Turian et al., ).
We also define the core tasks by reformulating the SUPERB (Yang et al., ) (speech), MARBLE (Yuan et al., ) (music), and HEAR (Turian et al., ) (audio) benchmarks to enable quick-round research experiments.
One challenge of evaluating with such a large-scale benchmark is deriving concrete and useful insights from hundreds of evaluation results.
We provide a taxonomy for every task in Dynamic-SUPERB, where tasks are clustered by the specific model capabilities they probe.
Researchers can follow this taxonomy to develop or reinforce specific capabilities of the models they build.
To our knowledge, Dynamic-SUPERB is the largest benchmark and the first to provide such a detailed task taxonomy in speech processing.

We conducted a comprehensive evaluation of several models using Dynamic-SUPERB Phase-2.
To evaluate the diverse output formats of these models, we propose an automated pipeline that uses LLMs to assess and process model outputs for general evaluation across various tasks.
The evaluation results show that these models perform well only on a very limited range of tasks.
For example, SALMONN-13B performed well on English ASR, while WavLLM achieved high accuracy in emotion recognition.
However, they do not generalize well to a much broader range of tasks in speech recognition and paralinguistics.
On the other hand, the results reveal that training on diverse data, even with significant differences in signal-level characteristics, can enhance performance across domains.
We observed that spoken language models outperformed music language models in certain music tasks.
This highlights the potential for developing unified models for speech, music, and general audio processing.
Importantly, we have open-sourced all materials, including task data and the evaluation pipeline, to facilitate reproduction and further development by researchers.
We also invite researchers to join us in continuing to enhance Dynamic-SUPERB, making it more diverse and comprehensive to advance the development of universal spoken language models.

## 2 Related works

### 2.1 Instruction-following universal speech models

LLMs have shown strong natural language processing abilities and are used in speech, audio, and music applications.
Recent frameworks integrate a pre-trained speech encoder with an LLM through fine-tuning techniques such as LoRA (Hu et al., ).
LTU-AS (Gong et al., ) combines Whisper (Radford et al., ), an ASR system, with LLaMA (Touvron et al., ), and it is fine-tuned on an open-ended speech and audio question-answering dataset.
SALMONN adopts a window-level Q-former (Li et al., ) to generate soft prompts that fuse representations from the speech and audio encoders.
Qwen-Audio (Chu et al., ) introduces task-specific tags into Qwen to encourage knowledge sharing and prevent interference across diverse tasks, and it supports multi-turn dialogues for both audio and text inputs.
WavLLM (Hu et al., ) uses curriculum learning to prevent overfitting on specific tasks while maintaining the LLM’s original capabilities.
All these models accept speech, audio, and text as input but output only text, focusing on understanding rather than generation in speech and audio.
There are also several attempts to use prompts for speech, music, and audio generation (Guo et al., ; Agostinelli et al., ; Liu et al., ).
However, to our knowledge, there is no universal model that is capable of handling both generation and understanding tasks.
Moreover, these models have not been comprehensively evaluated on a benchmark, which hinders fair comparisons between them.

### 2.2 Evaluation benchmarks

Several benchmarks have been developed to evaluate the performance of speech models.
SUPERB (Yang et al., ) is the most widely used benchmark for assessing the performance of speech foundation models across various tasks that cover different aspects of speech.
On the other hand, SLUE (Shon et al., ; ) focuses more specifically on spoken language understanding.
Yet, they are primarily limited to English.
LeBenchmark (Evain et al., ; Parcollet et al., ) evaluates speech foundation models specifically for French, while IndicSUPERB (Javed et al., ) is dedicated to languages spoken in India.
ML-SUPERB (Shi et al., ; ) offers ASR and language identification tasks in 100+ languages.
Aside from speech, MARBLE (Yuan et al., ) and HEAR (Turian et al., ) offer platforms for evaluating various music and a wide range of audio tasks.
Using these benchmarks, researchers build a specialized model for each task.
Thus, the number of tasks is limited due to the growing costs associated with adding more tasks.
Conversely, universal models are expected to do various tasks without fine-tuning for each one, allowing users to engage in far more diverse applications and enabling benchmark developers to include more tasks for comprehensive evaluation.
In NLP and CV, benchmarks have been developed to evaluate models across a much broader range of tasks (Bitton et al., ; Srivastava et al., ; Mishra et al., ).
However, in speech, we lack a benchmark of comparable scale.
Dynamic-SUPERB (Huang et al., ) is the first benchmark for instruction-based universal speech models.
While its first version includes far more tasks than all other speech benchmarks, they are all classification tasks.
AIR-bench (Yang et al., ), on the other hand, includes tasks from speech, music, and audio, and expands beyond classification, although the number of tasks available for evaluating universal models remains limited (19 tasks in foundation track).
Thus, there remains a critical need for a comprehensive benchmark that evaluates universal models across a broader range of tasks to fully assess their capabilities.
Consequently, we developed Dynamic-SUPERB Phase-2, substantially upgrading the first generation with a detailed taxonomy and establishing community contribution protocols to facilitate continual task integration.

## 3 Dynamic-SUPERB Phase-2

### 3.1 Overview

Figure 1 depicts the framework of Dynamic-SUPERB Phase-2.
Our goal is to evaluate universal models that meet the following criteria:
(1) The model can accept speech, music, or other audio as input.
(2) The model can perform specific tasks without requiring fine-tuning.
(3) The model follows natural language instructions to execute the corresponding tasks.
To comprehensively evaluate these universal models, we have collected hundreds of tasks and developed a task taxonomy to better guide benchmark users (Figure 1a).
All tasks in Dynamic-SUPERB Phase-2 are intended solely for testing purposes; we do not provide training data for two main reasons.
First, current models are trained on large-scale, open-source, or proprietary datasets, making it challenging to ensure that all models are trained under consistent conditions.
Second, we aim to evaluate universal models that do not require fine-tuning for downstream tasks.

The Dynamic-SUPERB project is designed to evolve dynamically with research advancements by incorporating novel tasks from the research community (Figure 1b).
With the first call for tasks, Dynamic-SUPERB Phase-2 has grown from its first generation, expanding from 55 to 180 tasks.
Table 1 compares several benchmarks used in speech, music, and audio research, demonstrating that Dynamic-SUPERB Phase-2 is the largest benchmark covering all three areas.
It provides a more fine-grained evaluation than any other benchmark.

Figure: Figure 1: An overview of Dynamic-SUPERB.
Refer to caption: extracted/5986474/dynamic-superb-framework.png

**Table 1: Comparison of popular benchmarks with Dynamic-SUPERB.**
| Benchmark | SUPERB | SLUE | HEAR | MARBLE | AIR-Bench | Dynamic- | Dynamic- |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SUPERB Phase-1 | SUPERB Phase-2 |  |  |  |  |  |  |
| # of tasks | 13 | 7 | 19 | 13 | 19^∗ | 55 | 180 |
| Speech | ✓ | ✓ |  |  | ✓ | ✓ | ✓ |
| Music |  |  |  | ✓ | ✓ | ✓ | ✓ |
| Audio |  |  | ✓ |  | ✓ | ✓ | ✓ |

### 3.2 Task formulation

As Figure 1c depicts, in Dynamic-SUPERB Phasse-2, each task is structured to include:
(1) Text instruction: A natural language instruction that guides the model on the task to perform.
Each task has multiple different instructions to evaluate the model’s ability to understand instructions.
(2) Audio component: At least one audio element, which can be in the input, the output, or both.
(3) (Optional) Text component: Text elements (other than instruction) that may serve as inputs or outputs.
The number of audio elements in the inputs or outputs may vary depending on the task.
For example, in speaker verification, two utterances are used to determine whether they were produced by the same speaker (green blocks in Figure 1c).
Besides, text inputs are not always required; for instance, ASR does not involve any text inputs.
We use text format for instructions instead of spoken format.
Spoken instructions involve varying content, speaker characteristics, and prosody, making them more complex.
Text instructions act as an intermediary, bridging text and spoken universal models.
These designs ensure consistency and simplify the model’s understanding and processing of diverse tasks while maintaining the benchmark’s extendibility with minimal constraints.

Another problem in evaluating universal models is the format of classification and regression tasks.
Generally, a task-specialized model generates predicted labels (soft distributions or hard labels) within a pre-defined set of labels for classification or numeric values for regression within a pre-defined range.
However, a universal model does not necessarily have access to this information, and thus it is hard to evaluate them in the same way, especially using it to perform several different tasks.
To address this, we believe that a universal speech model should produce outputs in natural language for all tasks (model outputs in Figure 1c).
Therefore, in Dynamic-SUPERB, all classification labels are represented as text.
For regression tasks with specific formats (e.g., scalars, JSON, or Python-style lists), we can parse the natural language outputs using a post-processing pipeline (Section 4.2).

### 3.3 Call for tasks

Dynamic-SUPERB fosters community collaboration, encouraging the addition of innovative tasks to remain relevant in this rapidly changing field.
In Phase-2, we initiated a call for tasks in March 2024 to invite contributions from the research community.
We established an organized and transparent submission process on our GitHub portal, where we organize members who serve as editors to guide contributors.
Contributors propose tasks by providing relevant information on GitHub.
Each proposal is assigned to an editor who checks for major issues and prevents duplicated efforts.
Then, task proposers upload their data to our Huggingface space in specified formats, using only datasets with proper licenses.
They complete submissions by opening a pull request on GitHub with the necessary files.
Afterward, editors review submissions on a rolling basis, offering suggestions for refinement rather than immediate acceptance or rejection.
After iterative improvements, accepted tasks are merged into the repository.
Between March and July 2024, we received about 145 task proposals and accepted 91 tasks.
Tasks still under review are not included here but will be featured in the next phase.
For details, please refer to Appendix F.

### 3.4 Task taxonomy

Figure: (a) Task Taxonomy for speech tasks.
Refer to caption: extracted/5986474/speech-task-taxonomy-2-layers.png

One primary challenge in building a large-scale benchmark is offering valuable insights to its users.
To address this, we developed a task taxonomy(^1^11We developed the taxonomy by referencing sessions from the INTERSPEECH conference and EDICS of IEEE SPS. Though not identical, they have a very similar structure (please refer to Appendix A for comparison).) that helps researchers interpret performance results across various tasks.
Researchers can leverage this taxonomy to select specific tasks for model development instead of evaluating every task in the benchmark.

Figures 2 illustrates the high-level task taxonomy in Dynamic-SUPERB Phase-2.
Due to the space limitation, we only show some representative tasks.
Each leaf node includes at least one task and may be further categorized into more fine-grained sub-domains, which are not shown here due to space constraints.
For example, within ‘Speaker & Language/Speaker’, we have two categories: ‘Speaker Characteristics’ and ‘Speaker Identification’, each containing several tasks.
Please refer to Appendix B for the complete list of tasks.
We first categorize tasks into two primary fields: (I) speech and (II) music & audio, which are generally distinguished by the source of the sound.
For instance, speech is produced by humans, music is created using instruments, and audio includes sounds generated by other creatures, materials, or natural phenomena.
We then split each field into several domains based on the key attributes and challenges that the tasks within them present.

#### 3.4.1 Speech

As Figure 2(a) shows, there are 8 domains within the speech field.
Speech Recognition focuses on converting spoken language into text.
This includes tackling various challenges like multilingual and code-switching ASR, as well as spontaneous ASR which is very different from audiobook-style ones.
It also contains specialized tasks such as command recognition and keyword spotting.
Speaker and Language addresses the analysis of speaker characteristics and languages, covering tasks such as speaker verification, diarization, and language identification.
Spoken Language Understanding deals with understanding and analysis of the content and semantics of spoken language.
It covers tasks like sentiment analysis and speech translation.
Phonetics, Phonology, and Prosody focuses on the sound structure of speech, including phoneme recognition, pronunciation evaluation, and prosodic features like stress and accent classification.
Paralinguistics explores non-verbal aspects of speech, such as emotion recognition and vocal event detection, which can capture nuances like screaming or coughing.
Speech Enhancement aims to improve speech quality by detecting and mitigating noise, reverberation, and other degradations.
Speech, Voice, and Hearing Disorders is dedicated to identifying and classifying disorders such as stuttering and so on.
Safety and Security focuses on detecting synthetic or manipulated speech, addressing tasks like spoof detection and recognizing deepfake voices.

#### 3.4.2 Audio & Music

The audio and music field includes a wide range of tasks that focus on various attributes of sound beyond speech, such as musical elements, environmental sounds, and advanced sound analysis.
This field is divided into 9 domains, each addressing a specific aspect of audio or music processing.
Music Classification tasks focus on categorizing musical elements such as instruments, genres, and emotions, providing a foundation for recognizing and analyzing different types of musical content.
Pitch Analysis delves into identifying the pitch and harmony within music, including tasks like pitch extraction, chord classification, and key detection.
Rhythm Analysis involves tasks such as beat tracking, which is critical for understanding the temporal structure of music.
In Singing Analysis, tasks address both lyric recognition and translation, as well as the classification of vocal techniques used in singing.
Quality Assessment evaluates the perceived quality of singing, including automated predictions of Mean Opinion Scores (MOS).
The Sound Event domain is broader, covering various sound sources such as animals, the environment, and human activities. Tasks range from animal sound classification to emergency traffic detection and even advanced tasks like multi-channel sound event understanding.
Safety domain includes detecting deepfakes in singing voices and identifying manipulated audio files, ensuring sound authenticity and integrity.
Spatial Audio covers all tasks related to understanding spatial information, such as estimating the distance or position of sounds in the real world.
Finally, Signal Characteristics Analysis domain addresses general signal-level characteristics of audio, including sound effect detection, and duration prediction.

#### 3.4.3 Core tasks

While our task taxonomy provides a comprehensive, hierarchical, and systematic approach for benchmark users to easily get started with Dynamic-SUPERB Phase-2, evaluating all tasks with limited resources remains a significant challenge.
To address this, alongside the task taxonomy, we have selected core tasks for speech, music, and audio.
These core tasks are reduced subsets of essential tasks that have been widely used or studied within the research community.
Specifically, we include tasks from three popular benchmarks: SUPERB (speech), MARBLE (music), and HEAR (audio).
The three benchmarks were originally designed for evaluating encoders, not for an instruction-following universal model, so some modifications are required.
We manually crafted instructions for each task, reduced the data size, and reformulated them into the Dynamic-SUPERB format.
Since the core task set is much smaller than the full benchmark, researchers can more efficiently evaluate their models across a reasonable range of domains.

## 4 Experimental settings

### 4.1 Models

We evaluated several publicly-available models on Dynamic-SUPERB Phase-2: SALMONN, LTU-AS, Qwen-Audio, Qwen2-Audio, WavLLM, MU-LLaMA (Liu et al., ), GAMA (Deshmukh et al., ).
SALMONN is further categorized into 7B and 13B versions based on the size of its LLM component.
All of these models are publicly available, and we utilized their official implementations for inference without any modifications.

The first five models are instruction-based speech models, each also trained with audio understanding capabilities.
However, as they were not trained on music data, we do not expect them to perform well on music-related tasks.
Hence, we further included MU-LLaMA and GAMA.
MU-LLaMA is specifically designed for various music-understanding tasks, and GAMA is trained with both general audio data and a music corpus.
Similar to the speech models, MU-LLaMA and GAMA adopt an LLM-based framework.
They use a music or audio encoder, such as MERT (LI et al., ) or Q-former, to convert music or audio into features, which the LLM then uses as prompts for subsequent reasoning.
The sampling strategies used for generating outputs from the LLMs were retained as defined in their official implementations.
Finally, we implemented a cascaded system baseline called Whisper-LLaMA.
The system first transcribes audio using Whisper-v3-large, and then LLaMA3.1-8B processes the transcriptions to perform various tasks based on the provided instructions.

When evaluating models on these diverse tasks, several challenges inevitably arise.
One challenge was testing the baseline models on tasks that involve multiple audio inputs.
For example, in speaker verification, a model determines whether two utterances are produced by the same speaker.
Among the evaluated models, only Qwen-Audio and Qwen2-Audio provide interfaces that allow inputting multiple audio files.
For the remaining models, we concatenated all audio files, separated by 0.5 seconds of silence, and used the concatenated file as input.

Last, different models have their own maximum supported audio durations.
Models using Whisper to encode speech face an inherent limitation: Whisper can process audio files of up to 30 seconds, and some models are restricted to handling even shorter durations.
Consequently, we retained the original model settings and did not modify the model architecture to accommodate longer audio clips.
A preliminary analysis of all audio (Appendix G) in Dynamic-SUPERB Phase-2 shows that only a small proportion (around 7%) exceeds 30 seconds in length.
Hence, we believe our settings do not largely impact the evaluation results and ensure reasonable inference efficiency.

### 4.2 Evaluation metrics

The outputs of universal speech models are natural language sentences, which makes it difficult to directly assess their correctness using conventional metrics.
For classification tasks, such as emotion classification, a conventional task-specific model outputs a label from predefined emotions in the dataset, while a universal model generates sentences like “The speaker sounds happy.”
In this case, we can easily assess the correctness of the former by comparing labels, but this approach is not applicable to the latter.
Similarly, in regression tasks, a natural language response like “Using MOS scoring criteria, I give this audio a score of 3 out of 5” makes it difficult to directly use previous evaluation metrics such as mean square error.

In evaluation, we categorize tasks into three types: (1) classification, (2) regression, and (3) sequence generation.
For each type, we use different pipelines to evaluate the models’ outputs.
For classification tasks, we employ external LLMs (GPT-4o) as referees, with the temperature set to be zero for evaluation consistency, to evaluate whether the outputs from speech or music models(^2^22To avoid ambiguity, unless specified otherwise, the terms ‘output” or model output” in this section refer to the results generated by the models in Sec. 4.1.) match the ground truth.
This approach has been widely adopted in the NLP community (Wang et al., ; Liu et al., ; Chiang & Lee, ), and we extend its application to speech research.
We design a prompt that includes the task instructions, the model’s output to be evaluated, and the corresponding ground truth.
The LLM judge gets this prompt and determines if the output aligns with the ground truth using a chain-of-thought reasoning process.
By leveraging the strong instruction-following capabilities of these LLMs, we constrain them to provide the final decision in a fixed format, allowing us to extract the answer using simple methods such as regular expressions(^3^33We have tried to ask the speech universal models to directly output their decisions in a fixed format, but they do not always follow the instructions. Therefore, the evaluation pipeline described here is necessary.).
We then define accuracy as the percentage of outputs that the LLM judge considers aligned with the ground truth.
Please refer to Appendix E for details on the prompts and the alignment between LLMs and humans.

For regression tasks, the aforementioned method cannot be directly applied because performance is not assessed using hard labels.
Instead, we employ LLMs (GPT-4o) as a post-processor to transform natural language responses into a format compatible with the original metrics used in these tasks.
Specifically, we developed an additional prompt that directs the LLMs to extract essential information from the output and convert it into the same format as the ground truth.
If the output lacks any correct or relevant information that can be converted to match the ground truth format, the LLM returns “N/A” to mark it as invalid.
Since conventional metrics cannot accommodate “N/A” and setting a default value is unreasonable due to the variability of different metrics, we also calculate the N/A rate for each regression task, defined as the percentage of invalid outputs within a task.
A higher N/A rate indicates that the model struggles with following instructions, which may indirectly affect its performance on the task.

For sequence generation tasks such as ASR and captioning, we apply their original metrics directly to the raw outputs from baseline models.
This is because identifying redundant prefixes or sections unrelated to the task objectives is highly challenging in sequence generation, even with human involvement.
Besides, our review of the original evaluation results reported by these baseline models revealed no explicit mention of post-processing procedures.
Therefore, we base our evaluations solely on the unprocessed outputs, ensuring consistency and objectivity across all models.

## 5 Results

### 5.1 Domain-Level Performance Comparison in Taxonomy

Here, we analyze the results based on the taxonomy. Due to space limitations, we can only report and compare the results of different models at the domain level in this subsection.
Since different metrics are used across tasks, we adopted a relative-score-based method to summarize task performance.
For each task, we calculated the relative improvement of each model compared to the cascaded system baseline (Whisper + LLaMA) and then obtained the domain-level scores by averaging all its improvements across the tasks within each domain.
For regression tasks, we introduced the N/A rate to measure how well a model meets the task requirements.
The original task metric was only computed on instances that followed the task format (after post-processing by the LLM).
To account for whether a model can follow instructions, we incorporated the N/A rate into the reported scores.
Accordingly, we calculated the improvement based on scaled values.
For metrics where a higher value indicates better performance (such as the F1 score), we multiplied the metric value by (1 – N/A rate). Conversely, for metrics where a lower value indicates better performance (such as word error rate), we divided the metric value by (1 – N/A rate).

Figure 3 presents a domain-level comparison of relative scores across different models.
Whisper-LLaMA consistently has zero scores throughout all domains, serving as the reference baseline for evaluating other models.
Starting with speech domains, no model outperforms Whisper-LLaMA in speech recognition and spoken language understanding.
This highlights that using ASR such as Whisper remains a strong baseline for language understanding, as text more explicitly represents semantic information than speech.

Conversely, in the speaker and paralinguistics domains, all models surpass the baseline.
This improvement occurs because the ASR process in the cascaded system tends to discard critical information from the speech signal, such as speaker characteristics, pitch, and emotion.
In contrast, other models utilize soft representations that retain more speech information, giving them the potential to learn beyond ASR, and some of these models were also trained to perform specific tasks within certain domains
(^4^44Although the models have been trained on some tasks in the same domains, the training tasks of these models generally do not use the same instructions as Dynamic-SUPERB and do not offer as broad coverage.).
Additionally, music-specific large language models like GAMA and Mu-LLaMA perform poorly on speech recognition and understanding tasks, which is expected since they were primarily designed for music understanding.
Nevertheless, they surprisingly achieved scores comparable to the speech models in speaker and paralinguistics.

Turning to audio and music domains, where ASR models cannot transcribe non-speech information into text, most models outperform the baseline in several areas, such as music classification and sound event detection.
Surprisingly, spoken language models primarily trained on speech data (Qwen, SALMONN, WavLLM)(^5^55Qwen and SALMONN are trained on a mixture of speech, music, and audio data, with speech data being the dominant component.) outperform the two music models (GAMA and Mu-LLaMA) in various music-related domains such as harmony and pitch, music classification, and rhythm analysis.
We speculate that training on diverse data, even with significant differences in signal-level characteristics, enhances performance across domains, emphasizing the importance of developing unified models for speech, music, and general audio processing.

Lastly, we observe some outliers with significantly higher or lower values in the figure, typically resulting from unbounded evaluation metrics.
For instance, in the phonetics and prosody domain, WavLLM and LTU-AS exhibit poor scores due to their erroneous outputs in phone/phoneme segment counting tasks, predicting thousands of segments in utterances lasting only seconds and resulting in an excessively high mean square error.
While the relative score-based approach effectively summarizes model performance, domain-level scores can be distorted by specific tasks within a domain.
Thus, we encourage researchers aiming to develop model performance in specific domains to comprehensively report task-level scores to more accurately reflect their models’ capabilities.

Figure: Figure 3: Domain-level relative-score-based performance comparison across different models. We exclude tasks where the models have a 100% NA rate or where Whisper-LLaMA scores zero, as these make it impossible to average their scores with other tasks.
Refer to caption: extracted/5986474/domain-level-rel-scores.png

### 5.2 Core tasks results

**Table 2: Evaluation on SUPERB tasks. A “*” indicates that the metric is scaled with NA rate, and a “-” means a model has a 100% NA rate.**
|  | PR | KS | IC | ER | ASR | QbE | SF | SV | SD |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PER $\downarrow$ | Acc $\uparrow$ | Acc $\uparrow$ | Acc $\uparrow$ | WER $\downarrow$ | Acc $\uparrow$ | F1^∗ $\uparrow$ | CER^∗ $\downarrow$ | Acc $\uparrow$ | DER^∗ $\downarrow$ |
| Whisper-LLaMA | 100.1 | 36.5 | 55.5 | 28.7 | 34.0 | 46.5 | 53.6 | 51.3 | 53.0 | 1068.7 |
| SALMONN-7B | 25.4 | 30.5 | 25.5 | 39.8 | 15.1 | 49.0 | 61.6 | 61.5 | 65.5 | - |
| SALMONN-13B | 24.6 | 2.0 | 9.0 | 30.9 | 2.8 | 51.5 | 29.6 | 106.3 | 93.5 | - |
| Qwen-Audio | 100.6 | 60.5 | 15.5 | 33.7 | 69.4 | 48.0 | 42.2 | 90.1 | 47.5 | 8852.2 |
| Qwen2-Audio | 101.0 | 47.0 | 11.5 | 68.1 | 36.7 | 53.5 | 42.4 | 55.7 | 55.5 | 4664.4 |
| WavLLM | 100.0 | 43.0 | 4.5 | 79.1 | 6.9 | 49.5 | 50.8 | 84.9 | 66.5 | 3621.2 |
| LTU-AS | 102.8 | 1.0 | 0.0 | 37.2 | 96.3 | 45.0 | 10.2 | 559.3 | 40.0 | 14923.3 |
| GAMA | 100.4 | 2.0 | 0.0 | 17.7 | 116.7 | 2.5 | 2.7 | 4750.0 | 33.5 | 187.6 |
| Mu-LLaMA | 110.3 | 4.0 | 0.0 | 25.0 | 103.7 | 51.0 | 1.0 | 15869.6 | 49.0 | - |

We tested the performance of the models on core tasks that are widely studied and commonly used.
Table 2 presents the detailed results of the models on SUPERB tasks (please refer to Appendix D for results on HEAR and MARBLE), displaying each task’s performance in its own metric rather than relative scores.
Due to space limitations, please refer to Appendix D for results on HEAR and MARBLE.
It is worth noting that these tasks have been adapted to the Dynamic-SUPERB framework.
Therefore, the results are not directly comparable to previous studies that tested on the original SUPERB.
However, they indicate current performance levels on these tasks.

No single model excels across all tasks.
In phoneme recognition (PR), the SALMONN models were the only ones to achieve a relatively lower phoneme error rate (PER) compared to the others.
For keyword spotting (KS), Qwen-Audio was the only model to perform slightly better than a random guess (50%).
Whisper-LLaMA surpassed all other models in intent classification (IC).
Regarding emotion recognition (ER), Qwen2-Audio and WavLLM stood out, with the former even achieving almost 80% accuracy.
In ASR, SALMONN-13B and WavLLM are the only two models that achieved a word error rate (WER) lower than 10%.
Query-by-example (QbE) appears to be very challenging for all models, as none performed better than a random guess (50%).
Slot-filling (SF) is evaluated using two metrics: F1 for slot type and CER for slot value.
While Whisper-LLaMA and SALMONN-7B are the top two in this task, their results are still far from ideal.
In speaker verification, SALMONN-13B was the only model to significantly outperform the others, achieving over 90% accuracy.
In speaker diarization, all models performed poorly, with some even reaching a 100% N/A rate.
Comparing Figure 3 and Table 2, we observed that performance on core tasks sometimes deviates from trends observed at the domain level.
For example, although WavLLM excels in ER, it performs poorly in paralinguistics, revealing limited capabilities and highlighting the need to enhance its generalizability.
In summary, among tested models, some excel in specific tasks, while others perform poorly across the board, and none dominate across all tasks.
We believe these findings provide valuable insights for researchers aiming to improve models, starting from core tasks and progressively tackling each domain in the benchmark for broader applicability.

## 6 Conclusions

This paper presents Dynamic-SUPERB Phase-2, the largest benchmark for evaluating instruction-based universal spoken language models, building upon collaborative efforts across the research community.
Dynamic-SUPERB covers a wide range of diverse tasks and offers a fine-grained task taxonomy.
The recent models show good performance on specific tasks but poor generalization across common tasks like those in SUPERB, highlighting the need for further research on universal models.
All materials will be made openly available to facilitate reproduction and benchmarking.
We sincerely invite researchers to join our vibrant community and collaborate to advance the field.

Limitations:
Although Dynamic-SUPERB Phase-2 is the largest and most comprehensive benchmark, we acknowledge its limitations.
It lacks comprehensive speech-generation tasks, as Phase-2 focused on understanding tasks due to the few universal generation models.
Despite our efforts to develop the task taxonomy scientifically, new domains may emerge as the benchmark grows, and tasks can be categorized in various ways.
While our automatic evaluation pipeline using LLMs correlates well with human evaluations (Appendix E) for current tasks, it may not generalize to all future tasks.
Upholding the core spirit of the Dynamic-SUPERB project, we are striving to address these issues and enhance the benchmark for the next phase.

## 7 Ethics Statement

Dynamic-SUPERB Phase-2 includes several datasets, and we asked contributors to describe how they utilized these datasets.
Most contributors derived task data by uniformly sampling from the original datasets without applying any special processing.
As a result, Dynamic-SUPERB Phase-2 inevitably inherits biases present in these datasets across the tasks.

## 8 Reproducibility Statement

As mentioned in the main text, we will open-source all task data and the evaluation pipeline to ensure reproducibility.
Additionally, we set the GPT-4o temperature to 0 to stabilize the evaluation process.
In Appendix E, we also present a preliminary study on using open-source LLMs for evaluation, showing promising correlations with human annotators.

## Appendix A Task taxonomy

Figures 4 and 5 show the complete task taxonomy in Dynamic-SUPERB Phase-2.
Additionally, Table 3 compares our task taxonomy with the INTERSPEECH conference and EDICS from IEEE SPS.
While not identical, our taxonomy is well-aligned with those organized by professional conferences and institutions.

Figure: Figure 4: Task taxonomy of speech.
Refer to caption: extracted/5986474/speech-task-taxonomy.png

Figure: Figure 5: Task taxonomy of audio and music.
Refer to caption: extracted/5986474/audio-task-taxonomy.png

**Table 3: Comparison of task taxonomy in Dynamic-SUPERB Phase-2 with INTERSPEECH and EDICS. A “✓” indicates that a conference session, area, or EDICS category matches or closely aligns with the taxonomy.**
| Taxonomy | INTERSPEECH | EDICS |
| --- | --- | --- |
| Music/Harmony & Pitch |  | ✓ |
| Music/Music Classification |  |  |
| Music/Rhythm Analysis |  | ✓ |
| Audio/Quality Assessment |  | ✓ |
| Audio/Safety |  | ✓ |
| Audio/Signal-Characteristics Analysis |  |  |
| Audio/Singing Analysis | ✓ |  |
| Audio/Sound Event | ✓ | ✓ |
| Audio/Spatial Audio Analysis | ✓ | ✓ |
| Speech/Paralinguistics | ✓ |  |
| Speech/Phonetics, Phonology, Prosody | ✓ |  |
| Speech/Safety and Security | ✓ |  |
| Speech/Speaker & Language | ✓ |  |
| Speech/Speech Enhancement | ✓ | ✓ |
| Speech/Speech Recognition | ✓ | ✓ |
| Speech/Speech, Voice, Hearing Disorder | ✓ |  |
| Speech/Spoken Language Understanding | ✓ | ✓ |

## Appendix B Dynamic-SUPERB tasks

**Table 4: The list of all tasks in Dynamic-SUPERB. Tasks are ordered by the taxonomy.^†**
|  |  |  |
| --- | --- | --- |
| Domain | Task | Dataset |
| Audio / Singing Analysis / Vocal Techniques | Vocal Technique Classification | VocalSet (Wilkins et al., ) |
| Audio / Singing Analysis / Vocal Techniques | MARBLE Vocal Technique Detection | VocalSet (Wilkins et al., ) |
| Audio / Singing Analysis / Lyrics | Lyric Translation | SingSet (recorded by the task contributor) |
| Audio / Singing Analysis / Lyrics | Song Lyric Recognition | SingSet (recorded by the task contributor) |
| Audio / Singing Analysis / Lyrics | Children Song Transcript Verification | CSD (Choi et al., ) |
| Audio / Spatial Audio Analysis | Sound Position Prediction | BinauralSoundPerception (Vasudevan et al., ) |
| Audio / Spatial Audio Analysis | Audio Spatial Distance Prediction | Spatial LibriSpeech (Sarabia et al., ) |
| Audio / Spatial Audio Analysis | HowFarAreYou | 3DSpeaker (Zheng et al., ) |
| Audio / Sound Event / Human | HEAR Vocal Imitation Classification | Vocal Imitations (Kim et al., ) |
| Audio / Safety / Audio Integrity | Scene Fake Detection | SceneFake (Yi et al., ) |
| Audio / Safety / Audio Integrity | Audio Editing Identification | Peoples Speech (Galvez et al., ) |
| Audio / Harmony & Pitch/Harmony | MARBLE Key Detection | Giantsteps Key (Knees et al., ) |
| Audio / Harmony & Pitch/Harmony | Chord Classification | Acoustic Guitar and Piano (DeepContractor, ) |
| Audio / Safety / Deepfake | Singing Voice Deepfake Detection | CtrSVDD ACEKiSing M4Singer (Zang et al., ) |
| Audio / Safety / Deepfake | Audio Deep Fake Detection | LJSpeech WaveFake MUSDB18HQ () |
| Audio / Sound Event / Advanced Understanding | Multichannel Sound Event Understanding | STARSS23 (Shimada et al., ) |
| Audio / Sound Event / Advanced Understanding | Audio Segment Retrieval | Clotho (Drossos et al., ) |
| Audio / Sound Event / Advanced Understanding | HEAR Sound Event Detection | DCASE2016 Task2 (Dohi et al., ) |
| Audio / Quality Assessment / Singing | Singing Automatic MOS Prediction | SingMOS (Tang et al., ) |
| Audio / Music Classification / Emotion | Emotion Classification in Songs | EMOTIFY (Aljanaki et al., ) |
| Audio / Music Classification / Emotion | MARBLE Emotion Detection | MTG MoodTheme (Bogdanov et al., ) |
| Audio / Rhythm Analysis | Music Beat Tracking | ASAP (Foscarin et al., ) |
| Audio / Rhythm Analysis | MARBLE Beat Tracking | GTZAN-Rhythm (Tzanetakis & Cook, ) |
| Audio / Harmony & Pitch / Pitch | Pitch Extraction by Lyrics | Children’s Song Dataset (CSD) (Choi et al., ) |
| Audio / Harmony & Pitch / Pitch | Instrument Pitch Classification | Nsynth (Engel et al., ) |
| Audio / Harmony & Pitch / Pitch | HEAR Music Transcription | MAESTRO Hawthorne et al. () |
| Audio / Harmony & Pitch / Pitch | HEAR Percussion Instruments Tonic Classification | Mridangam Tonic (Anantapadmanabhan et al., ) |
| Audio / Sound Event / Environment | Environment Recognition | ESC50 (Piczak, ) |
| Audio / Sound Event / Environment | Emergency Traffic Detection | ETD (tabarka rajab, ) |
| Audio / Sound Event / Environment | Domestic Environment Sound Event Detection | DESED-PublicEval (Turpault et al., ) |
| Audio / Sound Event / Environment | Environmental Sound Classification | UrbanSound8K-UrbanNoises (Salamon et al., ) |
| Audio / Sound Event / Environment | HEAR Environmental Sound Classification | ESC50 (Piczak, ) |
| Audio / Sound Event / Environment | Environmental Sound Classification | ESC50 Natural Soundscapes And Water Sounds (Piczak, ) |
| Audio / Sound Event / Environment | Environmental Sound Classification | ESC50-ExteriorAndUrbanNoises (Piczak, ) |
| Audio / Sound Event / Environment | Environmental Sound Classification | ESC50-InteriorAndDomesticSounds (Piczak, ) |
| Audio / Music Classification / Instrument | Instrument Source Classification | Nsynth (Engel et al., ) |
| Audio / Music Classification / Instrument | Instrument Classification | Beijing Opera Instrument (Tian et al., ) |
| Audio / Music Classification / Instrument | Instrument Combination Recognition | OpenMIC-2018 (Humphrey et al., ) |
| Audio / Music Classification / Instrument | Instrument Classification | Nsynth (Engel et al., ) |
| Audio / Music Classification / Instrument | HEAR Percussion Instruments Classification | Beijing Opera Percussion (Tian et al., ) |
| Audio / Music Classification / Instrument | HEAR Percussion Instruments Stroke Classification | Mridangam Stroke (Anantapadmanabhan et al., ) |
| Audio / Music Classification / Instrument | MARBLE Instrument Classification | MTG Instrument (Bogdanov et al., ) |
| Audio / Sound Event / Animal | Animal Classification | Wave Source Test (WavSource, ) |
| Audio / Sound Event / Animal | Cornell Birdcall Identification | Cornell Birdcall Identification (Howard et al., ) |
| Audio / Sound Event / Animal | Cat Emotion Classification | Cat Sound Classification Dataset V2 (Pandeya et al., ; Pandeya & Lee, ) |
| Audio / Sound Event / Animal | HEAR Beehive States Classification | Beehive States (Nolasco et al., ) |
| Audio / Sound Event / Animal | Environmental Sound Classification | ESC50 Animals (Piczak, ) |
| Audio / Sound Event / Animal | Bird Sound Detection | Warblrb10k (Stowell et al., ) |
| Audio / Music Classification / Genre | Music Genre Classification | FMA (Defferrard et al., ) |
| Audio / Music Classification / Genre | HEAR Music Genre Classification | GTZAN Genre (Tzanetakis & Cook, ) |
| Audio / Music Classification / Genre | MARBLE Music Tagging | MTG Top50 (Bogdanov et al., ), MagnaTagATune (Law et al., ) |
| Audio / Music Classification / Genre | MARBLE Genre Classification | MTG-Genre (Bogdanov et al., ), GTZAN (Tzanetakis & Cook, ) |
| Audio / Signal-Characteristics Analysis | Sound Effect Detection | IDMT-SMT-Audio-Effects (Stein, ) |
| Audio / Signal-Characteristics Analysis | Audio Duration Prediction | NTUML2021 (Yang et al., ) |
| Audio / Signal-Characteristics Analysis | HEAR Music Speech Classification | GTZAN Music Speech (Tzanetakis, ) |
| Audio / Signal-Characteristics Analysis | Speech Detection | LJSpeech (Ito & Johnson, ) |
| Audio / Signal-Characteristics Analysis | Speech Detection | LibriSpeech (Panayotov et al., ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phonological Feature Analysis | Phonological Feature Classification | VoxAngeles-ConsonantPlaceOfArticulation (Chodroff et al., ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phonological Feature Analysis | Phonological Feature Classification | VoxAngeles-MannerOfArticulation (Chodroff et al., ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phonological Feature Analysis | Phonological Feature Classification | VoxAngeles-Phone (Chodroff et al., ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phonological Feature Analysis | Phonological Feature Classification | VoxAngeles-VowelFrontness (Chodroff et al., ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phonological Feature Analysis | Phonological Feature Classification | VoxAngeles-VowelHeight (Chodroff et al., ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phonological Feature Analysis | Phonological Feature Classification | VoxAngeles-VowelRoundedness (Chodroff et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | L2 English Prosodic | speechocean762-Ranking (Zhang et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | L2 English Fluency | speechocean762-Ranking (Zhang et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | L2 English Fluency | speechocean762-Scoring (Zhang et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | L2 English Prosodic | speechocean762-Scoring (Zhang et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | Prosody Naturalness | ProsAudit-Lexical (de Seyssel et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | Prosody Naturalness | ProsAudit-Protosyntax (de Seyssel et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Recognition | SuperbSD | Libri2Mix Test (Cosentino et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Recognition | SuperbSV | VoxCeleb1 Test (Nagrani et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Recognition | HEAR Speaker Count Identification | LibriCount (Stöter et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Recognition | Speaker Counting | LibriTTS-TestClean (Zen et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Recognition | MultiSpeaker Detection | VCTK (Yamagishi et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Recognition | Speaker Verification | LibriSpeech-TestClean (Panayotov et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Recognition | Speaker Verification | VCTK (Yamagishi et al., ) |
| Speech / Spoken Language Understanding / Speech Translation | SuperbST | CoVoST2 Test (Wang et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody/Prosodic Feature Classification | Third Tone Sandhi Recognition | NCCU Corpus of Spoken Taiwan Mandarin (National Chengchi University, ) |
| Speech / Phonetics, Phonology, Prosody / Prosody/Prosodic Feature Classification | Accent Classification | Accentdb Extended (Ahamad et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody/Prosodic Feature Classification | Stress Detection | MIRSD (CHEN & JANG, ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phoneme Recognition Tasks | Phone Segment Counting | VoxAngeles (Chodroff et al., ) |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phoneme Recognition Tasks | Phoneme Segment Counting | Librispeech-words () |
| Speech / Phonetics, Phonology, Prosody / Phonetics and Phoneme Processing / Phoneme Recognition Tasks | Superb PR | LibriSpeech-TestClean (Panayotov et al., ) |
| Speech / Safety and Security / Spoofing and Anti-Spoofing | Spoof Detection | ASVspoof2015 (Wu et al., ) |
| Speech / Safety and Security / Spoofing and Anti-Spoofing | Spoof Detection | ASVspoof2017 (Delgado et al., ) |
| Speech / Spoken Language Understanding / Linguistic Structure & Grammar | Sentence Grammar Acceptability | sBLIMP () |
| Speech / Spoken Language Understanding / Linguistic Structure & Grammar | Nonce Word Detection | sWUGGY () |
| Speech / Spoken Language Understanding / Linguistic Structure & Grammar | PoS Estimation | LibriTTS_PoS (Zen et al., ) |
| Speech / Spoken Language Understanding / Linguistic Structure & Grammar | PoS Estimation | LibriTTS_PoS_with_transcription (Zen et al., ) |
| Speech / Spoken Language Understanding / Linguistic Structure & Grammar | Code Switching Semantic Grammar Acceptability Comparison | CSZS-zh-en (Huang et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-de (Pratap et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-en (Pratap et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-es (Pratap et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-fr (Pratap et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-it (Pratap et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-nl (Pratap et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-pl (Pratap et al., ) |
| Speech / Speech Recognition / Language | Multi-lingual Speech Recognition | MLS-pt (Pratap et al., ) |
| Speech / Speech Recognition / Language | Code-switching Speech Recognition | ASCEND (Lovenia et al., ) |
| Speech / Speech Recognition / Language | Code-switching Speech Recognition | NTUML2021 (Yang et al., ) |
| Speech / Speech Recognition / Language | PTBR Speech Recognition | CORAA (Candido Junior et al., ) |
| Speech / Speech Recognition / Language | AAVE Speech Recognition | CORAAL (Kendall & Farrington, ) |
| Speech / Speech Recognition / Language | Superb ASR | LibriSpeech-TestClean (Panayotov et al., ) |
| Speech / Speech Recognition / Language | Superb OODAsr Ar | CommonVoice7-Test (Ardila et al., ) |
| Speech / Speech Recognition / Language | Superb OODAsr Es | CommonVoice7-Test (Ardila et al., ) |
| Speech / Speech Recognition / Language | Superb OODAsr Zh | CommonVoice7-Test (Ardila et al., ) |
| Speech / Speech Recognition / Speaker | Target Speaker ASR | AMI-test (Carletta et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Reverberation Detection | LJSpeech (Ito & Johnson, ), RirsNoises-SmallRoom (Ko et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Reverberation Detection | LJSpeech (Ito & Johnson, ), RirsNoises-MediumRoom (Ko et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Reverberation Detection | LJSpeech (Ito & Johnson, ), RirsNoises-LargeRoom (Ko et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Reverberation Detection | VCTK (Yamagishi et al., ), RirsNoises-SmallRoom (Ko et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Reverberation Detection | VCTK (Yamagishi et al., ), RirsNoises-MediumRoom (Ko et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Reverberation Detection | VCTK (Yamagishi et al., ), RirsNoises-LargeRoom (Ko et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise SNR Level Prediction | VCTK (Yamagishi et al., ), MUSAN-Music (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise SNR Level Prediction | VCTK (Yamagishi et al., ), MUSAN-Speech (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise SNR Level Prediction | VCTK (Yamagishi et al., ), MUSAN-Gaussian (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise SNR Level Prediction | VCTK (Yamagishi et al., ), MUSAN-Noise (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | VCTK (Yamagishi et al., ), MUSAN-Music (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | VCTK (Yamagishi et al., ), MUSAN-Speech (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | VCTK (Yamagishi et al., ), MUSAN-Gaussian (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | VCTK (Yamagishi et al., ), MUSAN-Noise (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | LJSpeech (Ito & Johnson, ), MUSAN-Music (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | LJSpeech (Ito & Johnson, ), MUSAN-Speech (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | LJSpeech (Ito & Johnson, ), MUSAN-Gaussian (Snyder et al., ) |
| Speech / Speech Enhancement / Degradation Detection | Noise Detection | LJSpeech (Ito & Johnson, ), MUSAN-Noise (Snyder et al., ) |
| Speech / Speaker & Language / Language / Language Identification | HEAR Language Identification | VoxLingua107 Top10 (Valk & Alumäe, ) |
| Speech / Speaker & Language / Language / Language Identification | Language Identification | VoxForge (MacLean, ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Named Entity Localization | SLUE-VoxPopuli (Shon et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Named Entity Recognition | SLUE-VoxPopuli (Shon et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Sentiment Analysis | SLUE-VoxCeleb (Shon et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Semantic Textual Similarity | SpokenSTS (Agirre et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Conversation Matching | EnShortConversation () |
| Speech / Spoken Language Understanding / Intent & Meaning | Spoken Digit Arithmetic | AudioMNIST (Becker et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Superb IC | FluentSpeechCommands-Test (Lugosch et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Superb SF | AudioSnips-Test (Yang et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Dialogue Act Pairing | DailyTalk (Lee et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Dialogue Act Classification | DailyTalk (Lee et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Dialogue Act Classification | SLUE-HVB (Shon et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Intent Classification | FluentSpeechCommands-Action (Lugosch et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Intent Classification | FluentSpeechCommands-Location (Lugosch et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Intent Classification | FluentSpeechCommands-Object (Lugosch et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Sarcasm Detection | Mustard (Castro et al., ) |
| Speech / Spoken Language Understanding / Intent & Meaning | Sentence Grammar Acceptability | sBLIMP (Nguyen et al., ) |
| Speech / Speech, Voice, Hearing Disorder / Disorder Detection and Classification | Stuttering Detection | SEP-28k (Lea et al., ) |
| Speech / Speech, Voice, Hearing Disorder / Disorder Detection and Classification | Voice Disorder Classification | VOICED (Goldberger et al., ) |
| Speech / Paralinguistics / Emotion Analysis | Emoji Grounded Speech Emotion Recognition | RAVDESS (Livingstone & Russo, ) |
| Speech / Paralinguistics / Emotion Analysis | Emotion Change Detection | RAVDESS (Livingstone & Russo, ) |
| Speech / Paralinguistics / Emotion Analysis | Superb ER | Iemocap (Busso et al., ) |
| Speech / Paralinguistics / Emotion Analysis | HEAR Emotion Recognition | CREMAD (Cao et al., ) |
| Speech / Paralinguistics / Emotion Analysis | Emotion Recognition | Multimodal EmotionLines Dataset (Poria et al., ) |
| Speech / Paralinguistics / Emotion Analysis | Dialogue Emotion Classification | DailyTalk (Lee et al., ) |
| Speech / Speech Recognition / ASR Post-Processing | N-Best Correction | Librispeech-TestOther (Panayotov et al., ) |
| Speech / Paralinguistics / Vocal Event Detection | Human Screaming Detection | Environmentdb (whats2000, ) |
| Speech / Paralinguistics / Vocal Event Detection | Human Non-Speech Sound Recognition | Nonspeech7k-test (Rashid et al., ), CommonVoice-DeltaSegment-15 (Ardila et al., ) |
| Speech / Paralinguistics / Vocal Event Detection | Covid19 Cough Audio Classification | CoughVid (Orlandic et al., ) |
| Speech / Paralinguistics / Vocal Event Detection | Vocal Sound Recognition | VocalSound (Gong et al., ) |
| Speech / Paralinguistics / Vocal Event Detection | Environmental Sound Classification | ESC50-HumanAndNonSpeechSounds (Piczak, ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | L2 English Accuracy | speechocean762-Ranking (Zhang et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | L2 English Accuracy | speechocean762-scoring (Zhang et al., ) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | Heteronym Differentiation | HeteronymEn (Synthesized heteronym pairs from Wikipedia with the OpenAI text-to-speech service OpenAI ()) |
| Speech / Phonetics, Phonology, Prosody / Prosody / Prosodic Quality Assessment | Multilingual Pronunciation Similarity | VoxAngeles (Chodroff et al., ) |
| Speech / Safety and Security / Synthetic Speech Detection | DeepFake Voice Recognition | DEEP-VOICE (Bird & Lotfi, ) |
| Speech / Safety and Security / Synthetic Speech Detection | Fraud Robocall Recognition | CallHome (Katerenchuk et al., ) |
| Speech / Safety and Security / Synthetic Speech Detection | Fraud Robocall Recognition | Promo (Generated using XTTS-v1 Casanova et al. () on the Robocall Audio Dataset ) |
| Speech / Safety and Security / Synthetic Speech Detection | Fraud Robocall Recognition | Robocall () |
| Speech / Safety and Security / Synthetic Speech Detection | Enhancement Detection | LibriTTS-TestClean (Zen et al., ), WHAM (Wichern et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Speech Command Recognition | AudioMNIST (Becker et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Superb KS | SpeechCommandsV1 Test (Warden, ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Superb QbE | Quesst14 Eval (Anguera et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | HEAR Spoken Commands Classification | SpeechCommands-5hr (Warden, ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Spoken Term Detection | LJSpeech (Ito & Johnson, ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Spoken Term Detection | LibriSpeech TestOther (Panayotov et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Spoken Term Detection | LibriSpeech TestClean (Panayotov et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Multi Speaker Detection | LibriSpeech TestClean (Panayotov et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Speech Text Matching | LJSpeech (Ito & Johnson, ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Speech Text Matching | LibriSpeech TestClean (Panayotov et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Speech Text Matching | LibriSpeech TestOther (Panayotov et al., ) |
| Speech / Speech Recognition / Specific Recognition Tasks | Speech Command Recognition | GoogleSpeechCommandsV1 (Warden, ) |
| Speech / Speech Recognition / Spontaneous Speech | Superb OOD Asr Spon | SBCSAE Test (Du Bois et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Characteristics | Age Classification | CommonVoiceCorpus Test (Ardila et al., ) |
| Speech / Speaker & Language / Speaker / Speaker Characteristics | Gender Recognition by Voice | CommonVoice-DeltaSegment-15 (Ardila et al., ) |

## Appendix C Task-level evaluation results

**Table 5: Task-level Evaluation Results: Part 1^†**
| Task | Metric | Whisper- | LTU-AS | SALMONN | SALMONN | Qwen- | Qwen2- | WavLLM | MU- | GAMA |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA | 7B | 13B | Audio | Audio | LLaMA |  |  |  |  |  |  |
| Vocal Technique Classification | LLM-C$\uparrow$ | 7.50% | 0.50% | 3.50% | 5.50% | 5.00% | 13.00% | 7.00% | 2.50% | 2.00% |  |
| MARBLE Vocal Technique Detection | LLM-C $\uparrow$ | 8.00% | 1.00% | 11.00% | 6.50% | 6.50% | 15.50% | 8.00% | 8.50% | 2.50% |  |
| Lyric Translation | Sacre Bleu $\uparrow$ | 2.1093 | 1.0357 | 2.8126 | 0.9941 | 3.3975 | 3.9155 | 1.8969 | 0.1593 | 0.0739 |  |
| Children Song Transcript | WER $\downarrow$ | 0.5868 | 1.0268 | 1.0402 | 1.0099 | 1.2867 | 0.9493 | 1.5548 | 1 | 1.0012 |  |
| Song Lyric Recognition | MER $\downarrow$ | 1.0193 | 1.6453 | 3.5636 | 2.6600 | 1.1841 | 1.1562 | 4.8113 | 1.1753 | 1.6852 |  |
| Sound Position Prediction | NAR $\downarrow$ | 81.25% | 100.00% | 100.00% | 100.00% | 100.00% | 25.00% | 93.75% | 100.00% | 100.00% |  |
| Angle diff $\downarrow$ | 1.4006 | - | - | - | - | 1.1063 | 0.8681 | - | - |  |  |
| Audio Spatial Distance Prediction | NAR $\downarrow$ | 50.50% | 59.00% | 62.50% | 100.00% | 85.50% | 58.00% | 44.00% | 97.00% | 71.00% |  |
| medae $\downarrow$ | 0.6651 | 0.9330 | 0.4490 | - | 1.5011 | 0.6915 | 0.9011 | 0.9212 | 0.5000 |  |  |
| How Far Are You | LLM-C $\uparrow$ | 26.00% | 13.00% | 4.50% | 8.00% | 30.00% | 26.00% | 32.00% | 18.00% | 2.50% |  |
| HEAR Vocal Imitation Classification | LLM-C $\uparrow$ | 15.17% | 5.50% | 1.17% | 1.83% | 18.67% | 11.17% | 17.50% | 3.83% | 5.83% |  |
| Scene Fake Detection | LLM-C $\uparrow$ | 1.98% | 9.41% | 38.12% | 45.54% | 33.17% | 47.03% | 26.73% | 54.46% | 48.51% |  |
| Audio Editing Identification | NAR $\downarrow$ | 84.38% | 87.50% | 62.50% | 75.00% | 75.00% | 68.75% | 96.88% | 65.62% | 37.50% |  |
| Acc $\uparrow$ | 204.3094 | 53.6938 | 165.0617 | 43.1693 | 184.695 | 154.6202 | 100 | 164.0558 | 154.8122 |  |  |
| Pitch Extraction by Lyrics | NAR $\downarrow$ | 37.50% | 94.00% | 47.50% | 73.50% | 65.50% | 84.50% | 76.00% | 98.00% | 68.50% |  |
| Token Error Rate $\downarrow$ | 1.4628% | 1.1667% | 2.2621% | 6.3924% | 1.0645% | 1.6596% | 1.3611% | 1% | 1.9518% |  |  |
| Instrument Pitch Classification | LLM-C $\uparrow$ | 0.56% | 0.56% | 0.11% | 0.22% | 0.67% | 18.11% | 0.78% | 0.33% | 0.78% |  |
| HEAR Music Transcription | NAR $\downarrow$ | 92.43% | 100.00% | 99.46% | 100.00% | 96.76% | 96.76% | 96.76% | 100.00% | 100.00% |  |
| Token Error Rate $\downarrow$ | 13.0% | - | 36.0% | - | 16.3% | 8.4% | 24.2% | - | - |  |  |
| HEAR Percussion Instruments Tonic Classification | LLM-C $\uparrow$ | 2.00% | 2.50% | 12.70% | 13.40% | 11.30% | 11.80% | 12.80% | 12.50% | 5.70% |  |
| Audio Deep Fake Detection | LLM-C $\uparrow$ | 5.56% | 22.06% | 22.06% | 26.19% | 24.44% | 21.43% | 9.21% | 20.95% | 9.84% |  |
| Singing Voice Deepfake Detection | LLM-C $\uparrow$ | 8.25% | 24.56% | 22.79% | 38.31% | 31.43% | 27.90% | 30.84% | 39.10% | 25.54% |  |
| Multichannel Sound Event Understanding | LLM-C $\uparrow$ | 38.73% | 4.23% | 37.32% | 26.06% | 38.03% | 41.55% | 21.83% | 4.93% | 11.27% |  |
| Audio Segment Retrieval | NAR $\downarrow$ | 96.88% | 100.00% | 87.50% | 90.62% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |  |
| IoU $\uparrow$ | 0.0000 | 0.0000 | 0.1238 | 0.0691 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |  |  |
| HEAR Sound Event Detection | LLM-C $\uparrow$ | 0.00% | 0.00% | 21.43% | 35.71% | 7.14% | 14.29% | 21.43% | 21.43% | 7.14% |  |
| Singing Automatic MOS Prediction | NAR $\downarrow$ | 96.88% | 100.00% | 87.50% | 90.62% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |  |
| MSE $\downarrow$ | 12.1700 | 2.3872 | 10.0777 | 11.2308 | 14.8895 | 1.4485 | 11.4210 | 4.5300 | 2.0800 |  |  |
| KTAU $\uparrow$ | -0.3333 | 0.0126 | -0.1105 | 0.0166 | 0.0712 | 0.0099 | 0.0103 | 0.1705 | 0.3539 |  |  |
| LCC $\uparrow$ | -0.6747 | -0.0706 | -0.1354 | 0.0360 | 0.0824 | 0.0255 | -0.0216 | 0.1866 | 0.3846 |  |  |
| SRCC $\uparrow$ | -0.5000 | 0.0183 | -0.1325 | 0.0231 | 0.0872 | 0.0124 | 0.0118 | 0.2281 | 0.4247 |  |  |
| Emotion Classification in Songs | LLM-C $\uparrow$ | 10.00% | 0.00% | 1.67% | 3.33% | 13.33% | 8.33% | 8.33% | 1.67% | 0.00% |  |
| MARBLE Emotion Detection | LLM-C $\uparrow$ | 0.10% | 1.50% | 0.00% | 0.00% | 1.30% | 0.50% | 0.10% | 0.00% | 0.10% |  |
| Music Beat Tracking<br>(ASAP) | NAR $\downarrow$ | 54.55% | 74.55% | 51.82% | 63.64% | 62.73% | 40.91% | 47.27% | 80.00% | 84.55% |  |
| Miss Time $\downarrow$ | 135.6077 | 183.6786 | 118.4047 | 77.9687 | 157.5917 | 76.0904 | 111.7759 | 240.0000 | 93.8751 |  |  |
| Music Beat Tracking<br>(GTZAN-Rhythm) | NAR $\downarrow$ | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |  |
| MSE $\downarrow$ | - | - | - | - | - | - | - | - | - |  |  |
| Pitch Extraction by Lyrics | NAR $\downarrow$ | 37.50% | 94.00% | 47.50% | 73.50% | 65.50% | 84.50% | 76.00% | 98.00% | 68.50% |  |
| Token Error Rate $\downarrow$ | 146.28% | 116.67% | 226.21% | 639.24% | 106.45% | 165.96% | 136.11% | 100% | 195.18% |  |  |
| Instrument Pitch Classification | LLM-C $\uparrow$ | 0.56% | 0.56% | 0.11% | 0.22% | 0.67% | 18.11% | 0.78% | 0.33% | 0.78% |  |
| HEAR Music Transcription | NAR $\downarrow$ | 92.43% | 100.00% | 99.46% | 100.00% | 96.76% | 96.76% | 96.76% | 100.00% | 100.00% |  |
| HEAR Music Transcription | Token Error Rate $\downarrow$ | 1.01% | - | 36.0% | - | 8.34% | 1.69% | 6.25% | - | - |  |
| HEAR Percussion Instruments Tonic Classification | LLM-C $\uparrow$ | 2.00% | 2.50% | 12.70% | 13.40% | 11.30% | 11.80% | 12.80% | 12.50% | 5.70% |  |

**Table 6: Task-level Evaluation Results: Part 2^†**
| Task | Metric | Whisper- | LTU-AS | SALMONN | SALMONN | Qwen- | Qwen2- | WavLLM | MU- | GAMA |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA | 7B | 13B | Audio | Audio | LLaMA |  |  |  |  |  |  |
| Environment Recognition | LLM-C $\uparrow$ | 7.02% | 38.60% | 42.11% | 19.30% | 56.14% | 63.16% | 40.35% | 10.53% | 31.58% |  |
| Emergency Traffic Detection | LLM-C $\uparrow$ | 9.67% | 25.33% | 23.33% | 27.00% | 28.33% | 27.33% | 25.96% | 7.67% | 27.33% |  |
| Domestic Environment Sound Event Detection | NAR $\downarrow$ | 91.39% | 99.44% | 99.72% | 99.72% | 99.17% | 99.72% | 99.72% | 100.00% | 99.72% |  |
| Event Based F1 $\uparrow$ | 0.0000 | 0.6000 | 0.0000 | 0.0000 | 0.0000 | 0.0100 | 0.0000 | 0.0000 | 0.0000 |  |  |
| Environmental Sound Classification<br>(UrbanSound8K UrbanNoises) | LLM-C $\uparrow$ | 10.00% | 3.26% | 1.40% | 0.00% | 39.53% | 4.88% | 10.47% | 0.00% | 0.70% |  |
| HEAR Environmental Sound Classification<br>(ESC-50) | LLM-C $\uparrow$ | 4.80% | 25.30% | 61.00% | 73.40% | 68.80% | 43.90% | 9.80% | 0.40% | 35.30% |  |
| Environmental Sound Classification<br>(ESC50 Natural Soundscapes And Water Sound) | LLM-C $\uparrow$ | 3.00% | 7.50% | 6.50% | 1.00% | 78.00% | 5.00% | 7.00% | 0.00% | 7.00% |  |
| Environmental Sound Classification<br>(ESC50 Exterior And UrbanNoises) | LLM-C $\uparrow$ | 4.50% | 14.50% | 0.50% | 0.50% | 77.00% | 25.00% | 6.50% | 0.00% | 6.50% |  |
| Environmental Sound Classification<br>(ESC50 Interior And Domestic Sounds) | LLM-C $\uparrow$ | 4.50% | 3.50% | 0.50% | 0.00% | 59.00% | 2.50% | 4.00% | 0.00% | 0.00% |  |
| Instrument Source Classification | LLM-C $\uparrow$ | 7.22% | 28.33% | 37.33% | 43.56% | 32.67% | 35.22% | 30.89% | 38.00% | 35.89% |  |
| Instrument Classification<br>(Beijing Opera Instrument) | LLM-C $\uparrow$ | 3.39% | 10.59% | 6.36% | 23.73% | 30.93% | 28.39% | 22.03% | 25.85% | 10.17% |  |
| Instrument Combination Recognition | LLM-C $\uparrow$ | 4.20% | 13.51% | 17.72% | 24.02% | 25.83% | 24.92% | 20.42% | 19.52% | 8.11% |  |
| Instrument Classification<br>(Nsynth) | LLM-C $\uparrow$ | 1.23% | 9.56% | 13.97% | 14.58% | 17.89% | 54.41% | 11.89% | 6.99% | 10.17% |  |
| HEAR Percussion Instruments Classification | LLM-C $\uparrow$ | 3.39% | 4.24% | 10.17% | 20.76% | 16.10% | 27.12% | 24.15% | 22.03% | 11.44% |  |
| HEAR Percussion Instruments Stroke Classification | LLM-C $\uparrow$ | 0.10% | 3.80% | 7.60% | 4.90% | 11.50% | 10.80% | 13.70% | 10.30% | 0.10% |  |
| MARBLE Instrument Classification | LLM-C $\uparrow$ | 1.00% | 1.00% | 0.90% | 0.00% | 9.70% | 3.30% | 1.10% | 1.00% | 0.60% |  |
| Animal Classification | LLM-C $\uparrow$ | 9.25% | 40.50% | 58.25% | 67.00% | 75.75% | 34.00% | 12.00% | 4.25% | 15.25% |  |
| Cornell Birdcall Identification | LLM-C $\uparrow$ | 0.00% | 0.00% | 0.00% | 3.33% | 10.00% | 10.00% | 13.33% | 3.33% | 0.00% |  |
| Cat Emotion Classification | LLM-C $\uparrow$ | 6.00% | 0.00% | 2.00% | 4.00% | 6.00% | 14.00% | 8.00% | 2.00% | 14.00% |  |
| HEAR Beehive States Classification | LLM-C $\uparrow$ | 43.00% | 35.00% | 54.00% | 42.50% | 19.00% | 42.50% | 37.00% | 37.50% | 19.00% |  |
| Environmental Sound Classification | LLM-C $\uparrow$ | 14.50% | 8.00% | 14.00% | 3.00% | 82.00% | 36.00% | 5.00% | 0.00% | 2.00% |  |
| Bird Sound Detection | LLM-C $\uparrow$ | 28.50% | 43.00% | 74.50% | 75.00% | 78.50% | 79.00% | 75.00% | 40.50% | 33.00% |  |
| Music Genre Classification | LLM-C $\uparrow$ | 10.71% | 1.79% | 9.82% | 16.07% | 8.04% | 15.18% | 9.82% | 9.82% | 4.46% |  |
| HEAR Music Genre Classification | LLM-C $\uparrow$ | 7.30% | 9.70% | 37.50% | 42.50% | 29.00 % | 29.40 % | 9.50% | 26.90% | 7.00% |  |
| MARBLE Music Tagging<br>(MTGTop50) | LLM-C $\uparrow$ | 0.00% | 0.10% | 0.00% | 0.00% | 1.40% | 0.20% | 0.00% | 0.60% | 0.10% |  |
| MARBLE Music Tagging<br>(MagnaTagATune) | LLM-C $\uparrow$ | 5.50% | 1.00% | 3.50% | 9.00% | 4.50% | 1.50% | 1.00% | 3.50% | 7.50% |  |
| MARBLE Genre Classification<br>(MTG-Genre) | LLM-C $\uparrow$ | 0.60% | 2.10% | 0.00% | 0.00% | 3.70% | 0.80% | 0.80% | 0.80% | 0.30% |  |
| MARBLE Genre Classification<br>(GTZAN) | LLM-C $\uparrow$ | 28.50% | 8.00% | 32.00% | 61.50% | 63.50% | 68.50% | 22.50% | 17.00% | 5.00% |  |
| Sound Effect Detection | LLM-C $\uparrow$ | 0.81% | 3.00% | 4.56% | 10.06% | 7.81% | 13.69% | 7.38% | 7.31% | 3.31% |  |
| Audio Duration Prediction | NAR $\downarrow$ | 33.00% | 37.00% | 10.50% | 55.00% | 0.00% | 0.00% | 0.50% | 46.50% | 14.00% |  |
| MSE $\downarrow$ | 13.3657 | 39.1111 | 31652.7877 | 2216.7111 | 28.985 | 62.515 | 3581.0352 | 1527.785 | 62.7674 |  |  |
| HEAR Music Speech Classification | LLM-C $\uparrow$ | 63.28% | 34.38% | 74.22% | 75.78% | 96.88% | 75.00% | 60.94% | 53.13% | 82.81% |  |
| Speech Detection<br>(LJSpeech) | LLM-C $\uparrow$ | 54.00% | 44.50% | 60.50% | 74.00% | 57.00% | 45.00% | 53.00% | 49.00% | 55.50% |  |
| Speech Detection<br>(LibriSpeech-TestOther) | LLM-C $\uparrow$ | 52.00% | 31.50% | 46.00% | 66.50% | 44.50% | 55.50% | 37.00% | 45.50% | 36.00% |  |
| Speech Detection<br>(LibriSpeech-TestClean) | LLM-C $\uparrow$ | 48.00% | 29.00% | 46.00% | 66.00% | 45.50% | 56.50% | 35.00% | 46.50% | 38.50% |  |
| Phonological Feature Classification<br>(VoxAngeles-ConsonantPlaceOfArticulation) | LLM-C $\uparrow$ | 25.77% | 1.37% | 1.88% | 1.37% | 1.54% | 3.24% | 1.88% | 1.54% | 0.51% |  |
| Phonological Feature Classification<br>(VoxAngeles-MannerOfArticulation) | LLM-C $\uparrow$ | 17.42% | 9.36% | 2.06% | 1.50% | 5.62% | 6.74% | 6.84% | 7.96% | 8.99% |  |
| Phonological Feature Classification<br>(VoxAngeles-Phone) | LLM-C $\uparrow$ | 5.73% | 0.09% | 1.76% | - | 3.60% | 3.97% | 0.00% | 0.00% | 0.09% |  |
| Phonological Feature Classification<br>(VoxAngeles-VowelFrontnes) | LLM-C $\uparrow$ | 50.71% | 17.52% | 41.14% | 38.09% | 48.47% | 57.23% | 36.46% | 42.97% | 10.39% |  |
| Phonological Feature Classification<br>(VoxAngeles-VowelHeight) | LLM-C $\uparrow$ | 24.44% | 17.11% | 31.77% | 35.23% | 29.53% | 36.86% | 37.88% | 39.10% | 24.44% |  |
| Phonological Feature Classification<br>(VoxAngeles-VowelRoundedness) | LLM-C $\uparrow$ | 43.38% | 21.18% | 27.29% | 61.30% | 46.44% | 69.65% | 38.90% | 21.79% | 18.94% |  |
| L2 English Prosodic<br>(speechocean762-Ranking) | LLM-C $\uparrow$ | 26.39% | 35.00% | 46.67% | 52.50% | 34.44% | 51.11% | 34.44% | 49.72% | 42.22% |  |
| NAR $\downarrow$ | 39.43% | 94.75% | 0.27% | 1.21% | 71.47% | 3.10% | 11.98% | 94.62% | 82.91% |  |  |
| PCC $\uparrow$ | 0.0435 | -0.1747 | 0.0427 | 0.0775 | 0.1446 | 0.0201 | 0.0461 | -0.1535 | 0.0973 |  |  |
| L2 English Fluency<br>(speechocean762-Ranking) | LLM-C $\uparrow$ | 31.39% | 21.94% | 50.00% | 50.00% | 40.28% | 50.56% | 27.78% | 49.44% | 45.00% |  |
| NAR $\downarrow$ | 14.67% | 88.16% | 0.00% | 0.67% | 8.75% | 0.81% | 41.86% | 63.53% | 55.72% |  |  |
| PCC $\uparrow$ | 0.0055 | 0.0332 | 0.0292 | 0.0183 | -0.0422 | -0.0858 | 0.0222 | 0.0505 | 0.0532 |  |  |
| Prosody Naturalness<br>(ProsAudit-Lexical) | LLM-C $\uparrow$ | 49.81% | 32.82% | 48.26% | 47.10% | 21.62% | 51.74% | 54.83% | 47.10% | 5.41% |  |
| Prosody Naturalness<br>(ProsAudit-Protosyntax) | LLM-C $\uparrow$ | 53.44% | 31.68% | 46.56% | 46.18% | 25.95% | 50.76% | 43.51% | 49.24% | 4.20% |  |
| SuperbSD | NAR $\downarrow$ | 93.00% | 99.50% | 100.00% | 100.00% | 99.50% | 98.00% | 97.50% | 100.00% | 60.50% |  |
| diarization error rate $\downarrow$ | 74.81% | 74.62% | - | - | 44.26% | 93.29% | 90.53% | - | 74.12% |  |  |
| SuperbSV | LLM-C $\uparrow$ | 53.00% | 40.00% | 65.50% | 93.50% | 47.50% | 55.50% | 66.50% | 49.00% | 33.50% |  |
| HEAR Speaker Count Identification | NAR $\downarrow$ | 30.40% | 42.10% | 35.90% | 42.50% | 3.50% | 0.40% | 22.00% | 23.20% | 26.20% |  |
| Acc $\downarrow$ | 17.53% | 16.93% | 13.26% | 17.22% | 10.26% | 13.76% | 5.51% | 13.80% | 17.48% |  |  |
| Speaker Counting | LLM-C $\uparrow$ | 18.50% | 6.00% | 14.50% | 13.50% | 17.50% | 26.50% | 14.00% | 4.00% | 12.00% |  |
| Multi Speaker Detection | LLM-C $\uparrow$ | 46.50% | 23.50% | 36.00% | 30.50% | 57.00% | 51.00% | 32.50% | 39.00% | 48.50% |  |
| Speaker Verification<br>(LibriSpeech-TestClean) | LLM-C $\uparrow$ | 48.00% | 37.50% | 45.00% | 54.50% | 43.50% | 52.00% | 19.00% | 24.00% | 13.00% |  |
| Speaker Verification<br>(VCTK) | LLM-C $\uparrow$ | 48.00% | 38.50% | 54.00% | 58.50% | 32.50% | 51.00% | 24.00% | 19.50% | 7.00% |  |

**Table 7: Task-level Evaluation Results: Part 3^†**
| Task | Metric | Whisper- | LTU-AS | SALMONN | SALMONN | Qwen- | Qwen2- | WavLLM | MU- | GAMA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA | 7B | 13B | Audio | Audio | LLaMA |  |  |  |  |  |
| SuperbST | sacreBLEU $\uparrow$ | 17.3720 | 0.1070 | 18.7884 | 16.7835 | 6.9846 | 23.3458 | 21.7423 | 0.0566 | 0.0219 |
| Third Tone Sandhi Recognition | NAR $\downarrow$ | 59.38% | 93.75% | 100.00% | 87.50% | 84.38% | 84.38% | 28.12% | 96.88% | 100.00% |
| IoU $\uparrow$ | 0.2179 | 0.5 | 0 | 0.75 | 0 | 0 | 0.1304 | 0 | 0 |  |
| Accent Classification | LLM-C $\uparrow$ | 17.50% | 5.50% | 3.00% | 4.50% | 26.50% | 14.00% | 7.00% | 27.00% | 4.00% |
| Stress Detection | LLM-C $\uparrow$ | 15.50% | 3.00% | 2.00% | 13.00% | 16.50% | 23.50% | 25.50% | 0.00% | 1.00% |
| Phoneme Segmentation Counting | NAR $\downarrow$ | 19.19% | 57.24% | 41.88% | 12.56% | 0.41% | 16.11% | 15.04% | 41.47% | 18.79% |
| Acc $\uparrow$ | 13.20% | 1.69% | 15.81% | 20.04% | 1.77% | 6.60% | 10.11% | 9.14% | 3.80% |  |
| Abs Diff $\downarrow$ | 41.236 | 3759.3878 | 5.4857 | 4.9331 | 3.1286 | 2.8706 | 45754.9952 | 5.4242 | 2.8694 |  |
| Phoneme Segmentation Counting | NAR $\downarrow$ | 16.29% | 48.64% | 39.09% | 11.62% | 5.99% | 1.73% | 0.50% | 2.67% | 72.22% |
| Acc $\uparrow$ | 13.56% | 11.77% | 11.84% | 8.25% | 5.42% | 13.46% | 9.55% | 0.47% | 2.29% |  |
| Abs Diff $\downarrow$ | 2.5378 | 1012.8103 | 6.5285 | 50.7724 | 7.1857 | 3.0622 | 7.9109 | 15.6777 | 5.0491 |  |
| SUPERB PR | Phoneme Error Rate $\downarrow$ | 100.12% | 102.75% | 25.36% | 24.60% | 100.58% | 100.96% | 99.99% | 110.27% | 100.37% |
| Spoof Detection<br>(ASVSpoof2015) | LLM-C $\uparrow$ | 41.00% | 19.00% | 55.00% | 14.00% | 13.00% | 23.00% | 19.50% | 69.50% | 0.50% |
| Spoof Detection<br>(ASVSpoof2017) | LLM-C $\uparrow$ | 49.00% | 4.50% | 63.00% | 38.00% | 27.00% | 32.50% | 21.00% | 64.00% | 0.00% |
| Sentence Grammar Acceptability | LLM-C $\uparrow$ | 52.78% | 6.75% | 27.38% | 42.06% | 28.97% | 49.40% | 49.80% | 50.79% | 3.57% |
| Nonce Word Detection | LLM-C $\uparrow$ | 48.43% | 30.20% | 48.43% | 48.43% | 21.08% | 50.14% | 48.15% | 35.04% | 19.37% |
| PoS Estimation<br>(LibriTTS_PoS) | POS $\downarrow$ | 2.5722 | 1.2792 | 2.9675 | 1.1990 | 3.0620 | 2.3091 | 3.1126 | 1.5327 | 1.9635 |
| PoS Estimation<br>(LibriTTS_PoS_with_transcription) | POS $\downarrow$ | 0.9081 | 1.3507 | 1.7583 | 1.0601 | 2.1580 | 1.7514 | 2.0463 | 1.2394 | 1.3911 |
| Code Switching Semantic Grammar Acceptability Comparison | LLM-C $\uparrow$ | 51.50% | 45.00% | 50.00% | 49.50% | 27.50% | 49.50% | 27.50% | 28.00% | 19.00% |
| MultiLingual Speech Recognition<br>(MLS-de) | WER $\downarrow$ | 73.80% | 132.35% | 34.46% | 25.21% | 79.60% | 24.96% | 49.56% | 99.37% | 105.81% |
| MultiLingual Speech Recognition<br>(MLS-en) | WER $\downarrow$ | 65.03% | 100.29% | 9.37% | 9.47% | 27.97% | 17.47% | 17.23% | 96.03% | 97.09% |
| MultiLingual Speech Recognition<br>(MLS-es) | WER $\downarrow$ | 72.49% | 123.26% | 23.45% | 16.10% | 58.82% | 18.19% | 44.90% | 103.19% | 101.75% |
| MultiLingual Speech Recognition<br>(MLS-fr) | WER $\downarrow$ | 71.09% | 133.86% | 26.52% | 21.27% | 46.27% | 19.01% | 42.35% | 102.25% | 102.01% |
| MultiLingual Speech Recognition<br>(MLS-it) | WER $\downarrow$ | 85.38% | 125.39% | 39.11% | 31.54% | 56.55% | 33.16% | 46.14% | 102.72% | 108.84% |
| MultiLingual Speech Recognition<br>(MLS-nl) | WER $\downarrow$ | 74.96% | 120.29% | 32.11% | 29.37% | 140.01% | 42.94% | 55.56% | 102.05% | 101.22% |
| MultiLingual Speech Recognition<br>(MLS-pl) | WER $\downarrow$ | 76.79% | 139.04% | 73.83% | 51.53% | 278.19% | 101.41% | 90.11% | 100.23% | 105.43% |
| MultiLingual Speech Recognition<br>(MLS-pt) | WER $\downarrow$ | 73.63% | 136.66% | 31.88% | 25.07% | 117.33% | 24.72% | 46.26% | 99.71% | 103.48% |
| Code Switching Speech Recognition | NAR $\downarrow$ | 68.20% | 12.20% | 5.40% | 88.80% | 1.60% | 0.00% | 30.40% | 69.40% | 14.60% |
| Acc $\uparrow$ | 28.30% | 50.57% | 58.14% | 14.29% | 44.31% | 60.80% | 10.92% | 13.73% | 9.84% |  |
| PTBR Speech Recognition | WER $\downarrow$ | 37.09% | 153.78% | 132.86% | 161.05% | 318.36% | 72.44% | 137.81% | 495.31% | 192.45% |
| AAVE Speech Recognition | WER $\downarrow$ | 21.73% | 97.34% | 23.99% | 31.56% | 96.81% | 38.55% | 34.91% | 136.50% | 102.92% |
| Code-switch Speech Recognition | MER $\downarrow$ | 424.58% | 215.51% | 293.50% | 185.27% | 165.54% | 116.88% | 172.03% | 130.18% | 193.47% |
| SUPERB ASR | WER $\downarrow$ | 33.96% | 96.28% | 15.11% | 2.79% | 69.35% | 36.70% | 6.87% | 103.67% | 116.66% |
| SUPERB OOD Asr Ar | WER $\downarrow$ | 51.04% | 245.85% | 216.51% | 178.02% | 289.25% | 178.21% | 149.15% | 225.28% | 504.53% |
| SUPERB OOD Asr Es | WER $\downarrow$ | 10.93% | 150.00% | 98.96% | 99.22% | 100.83% | 75.08% | 99.38% | 141.68% | 303.28% |
| SUPERB OOD Asr Zh | CER $\downarrow$ | 29.67% | 508.02% | 609.28% | 310.33% | 449.32% | 270.63% | 445.36% | 435.33% | 1097.41% |
| TargetSpeaker | WER $\downarrow$ | 143.02% | 133.86% | 273.90% | 187.15% | 207.01% | 132.13% | 266.26% | 108.32% | 140.37% |
| Reverberation Detection<br>(LJSpeech RirsNoises SmallRoom) | LLM-C $\uparrow$ | 45.00% | 4.50% | 9.00% | 19.50% | 24.00% | 48.00% | 42.50% | 14.00% | 6.50% |
| Reverberation Detection<br>(LJSpeech RirsNoises MediumRoom) | LLM-C $\uparrow$ | 40.00% | 4.50% | 14.50% | 25.00% | 31.00% | 48.00% | 41.00% | 10.50% | 5.00% |
| Reverberation Detection<br>(LJSpeech RirsNoises LargeRoom) | LLM-C $\uparrow$ | 44.50% | 11.00% | 19.00% | 34.00% | 28.00% | 48.00% | 42.50% | 17.50% | 6.00% |
| Reverberation Detection<br>(VCTK RirsNoises SmallRoom) | LLM-C $\uparrow$ | 47.00% | 18.50% | 8.00% | 13.00% | 17.50% | 46.00% | 36.00% | 10.50% | 11.00% |
| Reverberation Detection<br>(VCTK RirsNoises MediumRoom) | LLM-C $\uparrow$ | 46.50% | 22.50% | 8.50% | 19.50% | 20.00% | 46.00% | 38.50% | 14.00% | 6.50% |
| Reverberation Detection<br>(VCTK RirsNoises LargeRoom) | LLM-C $\uparrow$ | 43.00% | 24.50% | 18.50% | 25.00% | 18.00% | 46.00% | 37.00% | 14.50% | 7.50% |
| Noise SNR Level Prediction<br>(VCTK MUSAN Music) | LLM-C $\uparrow$ | 23.00% | 9.00% | 10.00% | 8.50% | 10.00% | 14.00% | 15.50% | 9.00% | 1.50% |
| Noise SNR Level Prediction<br>(VCTK MUSAN Speech) | LLM-C $\uparrow$ | 23.50% | 11.50% | 13.00% | 1.50% | 14.00% | 20.50% | 15.00% | 10.50% | 0.50% |
| Noise SNR Level Prediction<br>(VCTK MUSAN Gaussian) | LLM-C $\uparrow$ | 24.00% | 8.00% | 13.50% | 15.50% | 13.00% | 17.50% | 17.50% | 15.00% | 1.50% |
| Noise SNR Level Prediction<br>(VCTK MUSAN Noise) | LLM-C $\uparrow$ | 23.00% | 12.50% | 9.50% | 16.00% | 7.50% | 19.00% | 14.00% | 11.00% | 0.50% |
| Noise Detection<br>(LJSpeech MUSAN Music) | LLM-C $\uparrow$ | 52.00% | 8.50% | 52.00% | 47.00% | 50.50% | 47.00% | 44.50% | 33.00% | 6.00% |
| Noise Detection<br>(LJSpeech MUSAN Speech) | LLM-C $\uparrow$ | 48.00% | 13.50% | 53.00% | 36.00% | 50.50% | 47.00% | 45.00% | 50.50% | 51.00% |
| Noise Detection<br>(LJSpeech MUSAN Gaussian) | LLM-C $\uparrow$ | 45.50% | 18.50% | 50.00% | 47.50% | 42.00% | 49.00% | 39.00% | 41.00% | 21.50% |
| Noise Detection<br>(LJSpeech MUSAN Noise) | LLM-C $\uparrow$ | 47.00% | 13.50% | 50.50% | 49.00% | 50.50% | 49.50% | 44.50% | 48.00% | 9.00% |
| Noise Detection<br>(VCTK MUSAN Music) | LLM-C $\uparrow$ | 46.00% | 32.50% | 45.50% | 57.50% | 42.00% | 54.50% | 52.00% | 37.50% | 3.50% |
| Noise Detection<br>(VCTK MUSAN Speech) | LLM-C $\uparrow$ | 45.00% | 51.50% | 43.50% | 52.50% | 46.50% | 57.00% | 44.50% | 48.50% | 56.50% |
| Noise Detection<br>(VCTK MUSAN Gaussian) | LLM-C $\uparrow$ | 53.00% | 15.50% | 49.00% | 56.50% | 45.50% | 54.50% | 46.00% | 38.50% | 11.50% |
| Noise Detection<br>(VCTK MUSAN Noise) | LLM-C $\uparrow$ | 44.50% | 26.50% | 45.50% | 55.00% | 44.50% | 61.00% | 47.00% | 54.00% | 4.50% |
| HEAR Language Identification | LLM-C $\uparrow$ | 92.18% | 1.95% | 6.28% | 11.01% | 18.00% | 36.11% | 1.85% | 0.00% | 0.10% |
| Language Identification<br>(VoxForge) | LLM-C $\uparrow$ | 95.50% | 13.50% | 22.00% | 8.50% | 84.50% | 93.00% | 18.00% | 6.00% | 0.00% |

**Table 8: Task-level Evaluation Results: Part 4^†**
| Task | Metric | Whisper- | LTU-AS | SALMONN | SALMONN | Qwen- | Qwen2- | WavLLM | MU- | GAMA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA | 7B | 13B | Audio | Audio | LLaMA |  |  |  |  |  |
| Named Entity Localization<br>(SLUE-VoxPopuli) | NAR $\downarrow$ | 81.25% | 100.00% | 100.00% | 100.00% | 100.00% | 25.00% | 93.75% | 100.00% | 100.00% |
| F1 (SpeechBERT) $\uparrow$ | 1.4006 | - | - | - | - | 1.1063 | 0.8681 | - | - |  |
| Named Entity Recognition<br>SLUE-VoxPopuli) | NAR $\downarrow$ | 90.76% | 98.37% | 95.65% | 98.37% | 95.11% | 91.30% | 91.85% | 97.83% | 96.74% |
| IoU $\uparrow$ | 0.0000 | 0.3333 | 0.0000 | 0.0000 | 0.0000 | 0.0625 | 0.0222 | 0.0000 | 0.0000 |  |
| Sentiment Analysis<br>(SLUE-VoxCeleb) | LLM-C $\uparrow$ | 67.80% | 37.01% | 36.72% | 36.44% | 45.48% | 62.71% | 55.37% | 28.25% | 26.55% |
| Semantic Textual Similarity<br>(SpokenSTS) | LLM-C $\uparrow$ | 48.80% | 47.20% | 47.20% | 50.40% | 42.80% | 46.80% | 50.40% | 51.20% | 38.80% |
| Conversation Matching<br>(EnShortConversation) | LLM-C $\uparrow$ | 77.78% | 5.56% | 57.41% | 37.04% | 51.85% | 66.67% | 62.96% | 3.70% | 24.07% |
| Spoken Digit Arithmetic<br>(AudioMNIST) | LLM-C $\uparrow$ | 43.50% | 13.00% | 31.00% | 33.00% | 15.00% | 43.50% | 40.50% | 4.50% | 13.00% |
| Superb IC<br>(FluentSpeechCommands-Test) | LLM-C $\uparrow$ | 55.50% | 0.00% | 25.50% | 9.00% | 15.50% | 11.50% | 4.50% | 0.00% | 0.00% |
| Superb SF<br>(AudioSnips-Test) | NAR $\downarrow$ | 38.00% | 88.50% | 34.00% | 68.50% | 51.00% | 45.00% | 44.00% | 99.00% | 97.00% |
| slot type f1 $\uparrow$ | 0.8637 | 0.8864 | 0.9325 | 0.9396 | 0.8616 | 0.7704 | 0.9070 | 1.0000 | 0.8939 |  |
| slot value cer $\downarrow$ | 0.3178 | 0.6432 | 0.4059 | 0.3349 | 0.4417 | 0.3065 | 0.4752 | 1.5870 | 1.4250 |  |
| Dialogue Act Pairing<br>(DailyTalk) | LLM-C $\uparrow$ | 50.00% | 3.50% | 51.00% | 48.00% | 43.50% | 46.00% | 37.00% | 40.00% | 25.00% |
| Intent Classification<br>(FluentSpeechCommands-Action) | LLM-C $\uparrow$ | 71.00% | 43.00% | 75.50% | 54.50% | 62.50% | 76.00% | 66.50% | 6.50% | 2.50% |
| Intent Classification<br>(FluentSpeechCommands-Location) | LLM-C $\uparrow$ | 64.00% | 57.50% | 36.00% | 40.00% | 45.50% | 86.00% | 57.50% | 51.50% | 33.50% |
| Intent Classification<br>(FluentSpeechCommands-Object) | LLM-C $\uparrow$ | 63.00% | 17.00% | 73.00% | 57.50% | 61.00% | 69.50% | 64.50% | 12.00% | 4.00% |
| Sarcasm Detection<br>(Mustard) | LLM-C $\uparrow$ | 44.50% | 15.00% | 38.50% | 46.00% | 42.50% | 48.00% | 46.50% | 43.00% | 8.00% |
| Dialogue Act Classification<br>(DailyTalk) | LLM-C $\uparrow$ | 29.00% | 10.50% | 34.00% | 40.50% | 42.00% | 30.00% | 36.50% | 18.50% | 4.00% |
| Dialogue Act Classification<br>(SLUE-HVB) | LLM-C $\uparrow$ | 11.08% | 0.00% | 36.00% | 0.92% | 5.00% | 15.83% | 13.17% | 1.83% | 1.33% |
| Stuttering Detection | LLM-C $\uparrow$ | 49.10% | 51.00% | 50.50% | 50.30% | 50.50% | 52.40% | 55.00% | 49.60% | 47.40% |
| Voice Disorder Classification | LLM-C $\uparrow$ | 13.46% | 1.92% | 13.46% | 17.31% | 13.46% | 16.35% | 18.27% | 21.15% | 6.73% |
| Emoji Grounded Speech Emotion Recognition | LLM-C $\uparrow$ | 1.10% | 0.00% | 0.00% | 0.10% | 1.30% | 1.20% | 0.10% | 1.90% | 48.50% |
| Emotion Change Detection | LLM-C $\uparrow$ | 0.08% | 0.42% | 0.58% | 0.96% | 0.04% | 2.96% | 0.08% | 0.58% | 0.04% |
| Superb ER | LLM-C $\uparrow$ | 28.70% | 37.20% | 39.80% | 30.90% | 33.70% | 68.10% | 79.10% | 25.00% | 17.70% |
| HEAR Emotion Recognition | LLM-C $\uparrow$ | 6.70% | 9.30% | 18.80% | 18.10% | 62.50% | 61.10% | 25.60% | 12.40% | 13.20% |
| Emotion Recognition | LLM-C $\uparrow$ | 22.50% | 14.50% | 19.00% | 9.50% | 43.50% | 36.50% | 40.50% | 0.50% | 2.50% |
| Dialogue Emotion Classification | LLM-C $\uparrow$ | 33.50% | 16.50% | 27.00% | 11.50% | 33.00% | 54.00% | 18.50% | 19.00% | 33.00% |
| NBest Correction | LLM-C $\uparrow$ | 31.80% | 22.80% | 23.00% | 29.20% | 32.80% | 29.00% | 32.60% | 20.80% | 30.80% |
| Human Screaming Detection | LLM-C $\uparrow$ | 52.50% | 25.00% | 52.50% | 65.00% | 62.50% | 87.50% | 47.50% | 42.50% | 60.00% |
| Human Non Speech Sound Recognition | LLM-C $\uparrow$ | 2.86% | 5.00% | 44.29% | 30.71% | 27.14% | 23.57% | 14.29% | 17.14% | 32.86% |
| Covid19 Cough Audio Classification | LLM-C $\uparrow$ | 0.93% | 1.03% | 0.00% | 0.00% | 4.12% | 0.72% | 4.22% | 21.91% | 0.21% |
| Vocal Sound Recognition | LLM-C $\uparrow$ | 29.86% | 1.53% | 11.53% | 4.31% | 75.56% | 30.97% | 14.44% | 0.28% | 2.78% |
| Environmental Sound Classification | LLM-C $\uparrow$ | 17.50% | 11.00% | 8.50% | 3.00% | 82.00% | 29.00% | 16.50% | 0.00% | 2.00% |
| L2 English Accuracy | LLM-C $\uparrow$ | 24.72% | 33.89% | 50.00% | 49.72% | 34.44% | 50.00% | 41.67% | 48.06% | 45.56% |
| L2 English Accuracy | NAR $\downarrow$ | 15.44% | 95.03% | 0.40% | 0.27% | 37.72% | 0.94% | 9.53% | 90.47% | 50.07% |
| PCC $\uparrow$ | 0.0185 | -0.1830 | 0.0633 | 0.0438 | 0.0293 | -0.0159 | 0.0439 | 0.0727 | 0.0151 |  |
| Heteronym Differentiation | LLM-C $\uparrow$ | 55.00% | 26.00% | 37.00% | 45.00% | 52.00% | 44.00% | 51.00% | 30.00% | 20.00% |
| Multilingual Pronunciation Similarity | LLM-C $\uparrow$ | 38.40% | 13.50% | 47.20% | 48.50% | 23.70% | 44.70% | 48.00% | 25.10% | 40.60% |
| Deep Fake Voice Recognition | LLM-C $\uparrow$ | 27.25% | 7.75% | 50.75% | 40.00% | 48.75% | 50.50% | 43.50% | 45.00% | 19.75% |
| Fraud Robocall Recognition<br>(CallHome) | LLM-C $\uparrow$ | 73.33% | 13.33% | 0.00% | 100.00% | 100.00% | 100.00% | 90.00% | 26.67% | 100.00% |
| Fraud Robocall Recognition<br>(Promo) | LLM-C $\uparrow$ | 47.37% | 10.53% | 0.00% | 73.68% | 57.89% | 63.16% | 63.16% | 5.26% | 100.00% |
| Fraud Robocall Recognition<br>(Robocall) | LLM-C $\uparrow$ | 94.87% | 51.28% | 100.00% | 20.51% | 51.28% | 61.54% | 30.77% | 30.77% | 0.00% |
| Enhancement Detection | LLM-C $\uparrow$ | 50.50% | 30.50% | 30.00% | 46.50% | 55.00% | 56.50% | 30.00% | 50.00% | 21.00% |
| Speech Command Recognition | NAR $\downarrow$ | 4.80% | 76.40% | 3.07% | 0.40% | 0.67% | 0.00% | 0.67% | 1.60% | 76.40% |
| Acc $\uparrow$ | 88.80% | 77.40% | 96.70% | 74.43% | 96.24% | 77.07% | 93.42% | 9.89% | 8.47% |  |
| Superb KS | LLM-C $\uparrow$ | 36.50% | 1.00% | 30.50% | 2.00% | 60.50% | 47.00% | 43.00% | 4.00% | 2.00% |
| Superb QbE | LLM-C $\uparrow$ | 46.50% | 45.00% | 49.00% | 51.50% | 48.00% | 53.50% | 49.50% | 51.00% | 2.50% |
| HEAR Spoken Commands Classification | LLM-C $\uparrow$ | 50.50% | 19.00% | 50.50% | 32.00% | 77.50% | 67.00% | 72.50% | 5.00% | 3.00% |
| Spoken Term Detection | LLM-C $\uparrow$ | 83.50% | 33.50% | 57.00% | 53.50% | 74.50% | 79.50% | 51.00% | 25.00% | 25.00% |
| Spoken Term Detection<br>(LibriSpeech-TestOther) | LLM-C $\uparrow$ | 75.50% | 23.00% | 54.50% | 50.50% | 72.50% | 64.00% | 46.50% | 25.50% | 34.00% |
| Spoken Term Detection<br>(LibriSpeech-TestClean) | LLM-C $\uparrow$ | 76.50% | 28.00% | 60.00% | 54.50% | 77.00% | 61.50% | 51.50% | 24.00% | 37.50% |
| Multi Speaker Detection | LLM-C $\uparrow$ | 46.00% | 29.00% | 16.50% | 22.00% | 58.00% | 52.00% | 22.00% | 42.50% | 53.00% |
| Speech Text Matching | LLM-C $\uparrow$ | 83.50% | 44.50% | 57.00% | 60.00% | 67.00% | 90.00% | 52.00% | 37.50% | 41.00% |
| Speech Text Matching<br>(LibriSpeech-TestClean) | LLM-C $\uparrow$ | 86.50% | 43.50% | 57.00% | 59.50% | 64.00% | 92.00% | 52.00% | 44.50% | 42.00% |
| Speech Text Matching<br>(LibriSpeech-TestOther) | LLM-C $\uparrow$ | 80.50% | 42.50% | 56.50% | 60.00% | 66.00% | 92.50% | 53.00% | 47.00% | 40.00% |
| Speech Command Recognition | LLM-C $\uparrow$ | 36.50% | 11.50% | 36.00% | 22.00% | 73.50% | 77.50% | 77.00% | 1.00% | 1.50% |
| Superb OOD Asr Spon | WER $\downarrow$ | 0.6314 | 1.6941 | 2.1327 | 2.5971 | 1.3956 | 1.2727 | 0.7654 | 2.8084 | 5.9324 |
| Age Classification | LLM-C $\uparrow$ | 0.50% | 19.75% | 22.75% | 21.00% | 26.00% | 35.50% | 21.00% | 23.50% | 17.75% |
| Gender Recognition by Voice | LLM-C $\uparrow$ | 1.00% | 81.50% | 63.00% | 57.00% | 53.50% | 97.50% | 35.00% | 51.00% | 63.00% |

## Appendix D HEAR & MARBLE Evaluation Results

**Table 9: HEAR (audio) Results. The SpeechCommands-5hr task results are omitted as they duplicate the SUPERB Keyword Spotting (KS) task, presented in Table 2. LLM-C stands for LLM Classification, NAR stands for Not Applicable Rate, and TER stands for Token Error Rate.**
| Task | Metric | Whisper- | LTU-AS | SALMONN | SALMONN | Qwen- | Qwen2- | WavLLM | MU- | GAMA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA | 7B | 13B | Audio | Audio | LLaMA |  |  |  |  |  |
| Beehive States | LLM-C $\uparrow$ | 43.00% | 35.00% | 54.00% | 42.50% | 19.00% | 42.50% | 37.00% | 37.50% | 19.00% |
| CREMA-D | LLM-C $\uparrow$ | 6.70% | 9.30% | 18.80% | 18.10% | 62.50% | 61.10% | 25.60% | 12.40% | 13.20% |
| ESC-50 | LLM-C $\uparrow$ | 4.80% | 25.30% | 61.00% | 73.40% | 68.80% | 43.90% | 9.80% | 0.40% | 35.30% |
| VoxLingua107 Top10 | LLM-C $\uparrow$ | 92.18% | 1.95% | 6.28% | 11.01% | 18.00% | 36.11% | 1.85% | 0.00% | 0.10% |
| GTZAN Genre | LLM-C $\uparrow$ | 7.30% | 9.70% | 37.50% | 42.50% | 29.00% | 29.40% | 9.50% | 26.90% | 7.00% |
| GTZAN Music Speech | LLM-C $\uparrow$ | 63.28% | 34.38% | 74.22% | 75.78% | 96.88% | 75.00% | 60.94% | 53.13% | 82.81% |
| MAESTRO | NAR $\downarrow$ | 92.43% | 100.00% | 99.46% | 100.00% | 96.76% | 96.76% | 96.76% | 100.00% | 100.00% |
| TER $\downarrow$ | 1.0065 | - | 36.0000 | - | 8.3438 | 1.6875 | 6.2500 | - | - |  |
| Beijing Opera Percussion | LLM-C $\uparrow$ | 3.39% | 4.24% | 10.17% | 20.76% | 16.10% | 27.12% | 24.15% | 22.03% | 11.44% |
| Mridangam Stroke | LLM-C $\uparrow$ | 0.10% | 3.80% | 7.60% | 4.90% | 11.50% | 10.80% | 13.70% | 10.30% | 0.10% |
| Mridangam Tonic | LLM-C $\uparrow$ | 2.00% | 2.50% | 12.70% | 13.40% | 11.30% | 11.80% | 12.80% | 12.50% | 5.70% |
| DCASE2016 Task2 | LLM-C $\uparrow$ | 0.00% | 0.00% | 21.43% | 35.71% | 7.14% | 14.29% | 21.43% | 21.43% | 7.14% |
| LibriCount | NAR $\downarrow$ | 30.40% | 42.10% | 35.90% | 42.50% | 3.50% | 0.40% | 22.00% | 23.20% | 26.20% |
| Acc $\uparrow$ | 17.53% | 16.93% | 13.26% | 17.22% | 10.26% | 13.76% | 5.51% | 13.80% | 17.48% |  |
| Vocal Imitations | LLM-C $\uparrow$ | 15.17% | 5.50% | 1.17% | 1.83% | 18.67% | 11.17% | 17.50% | 3.83% | 5.83% |

**Table 10: MARBLE (music) Results. LLM-C stands for LLM Classification, NAR stands for Not Applicable Rate, and MSE stands for Mean Squared Error.**
| Task | Metric | Whisper- | LTU-AS | SALMONN | SALMONN | Qwen- | Qwen2- | WavLLM | MU- | GAMA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA | 7B | 13B | Audio | Audio | LLaMA |  |  |  |  |  |
| Giantsteps Key | LLM-C $\uparrow$ | 3.00% | 0.50% | 1.50% | 1.00% | 3.00% | 17.00% | 2.00% | 0.50% | 0.00% |
| MTG Top50 | LLM-C $\uparrow$ | 0.00% | 0.10% | 0.00% | 0.00% | 1.40% | 0.20% | 0.00% | 0.60% | 0.10% |
| MagnaTagATune | LLM-C $\uparrow$ | 5.50% | 1.00% | 3.50% | 9.00% | 4.50% | 1.50% | 1.00% | 3.50% | 7.50% |
| MTG-Genre | LLM-C $\uparrow$ | 0.60% | 2.10% | 0.00% | 0.00% | 3.70% | 0.80% | 0.80% | 0.80% | 0.30% |
| GTZAN | LLM-C $\uparrow$ | 28.50% | 8.00% | 32.00% | 61.50% | 63.50% | 68.50% | 22.50% | 17.00% | 5.00% |
| MTGMoodTheme | LLM-C $\uparrow$ | 0.10% | 1.50% | 0.00% | 0.00% | 1.30% | 0.50% | 0.10% | 0.00% | 0.10% |
| GTZAN-Rhythm | NAR $\downarrow$ | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00% |
| MSE $\uparrow$ | - | - | - | - | - | - | - | - | - |  |
| VocalSet | LLM-C $\uparrow$ | 8.00% | 1.00% | 11.00% | 6.50% | 6.50% | 15.50% | 8.00% | 8.50% | 2.50% |
| MTG Instrument | LLM-C $\uparrow$ | 1.00% | 1.00% | 0.90% | 0.00% | 9.70% | 3.30% | 1.10% | 1.00% | 0.60% |

## Appendix E LLM-based evaluation

Here, we list the prompt templates used in the evaluation. Figure 6 illustrates the prompt for classification task evaluation, while Figure 7 shows the prompt for regression task evaluation.
In addition to GPT-4o, we also employed open-source LLMs to ensure better reproducibility. Tables 11 and  12 present the performance evaluations of the proposed prompts on classification and regression tasks, respectively, comparing their correlation with a human-labeled evaluation set. We found that GPT-4o and LLaMA-3.1-70B-Instruct achieved very high accuracy in both classification and regression tasks. This demonstrates that our evaluation using GPT-4o is reliable and that researchers can leverage LLaMA-3.1-70B-Instruct to conduct fully reproducible evaluations.

Figure: Figure 6: The prompt used for post-processing the classification tasks inference outputs in the study.

**Table 11: Performance evaluation of the proposed post-processing prompt on classification tasks using a human-labeled evaluation set. Accuracy is determined by agreement between the model output and the corresponding human label.**
| Model Name | Accuracy | Correct Count |
| --- | --- | --- |
| gpt-4o | 98.67% | 148/150 |
| gpt-4o-mini | 98.00% | 147/150 |
| gpt-o1-preview | 98.67% | 148/150 |
| gpt-o1-mini | 99.33% | 149/150 |
| llama-3.1-70B-Instruct | 96.67% | 145/150 |
| llama-3.1-8B-Instruct | 92.67% | 139/150 |
| llama-3.2-3B-Instruct | 86.67 % | 130/150 |
| llama-3.2-1B-Instruct | 61.33% | 92/150 |

**Table 12: Performance evaluation of the proposed post-processing prompt on regression tasks using a human-labeled evaluation set. Accuracy is determined by exact matches between the post-processed model output and the corresponding human-generated post-processed output.**
| Model Name | Accuracy | Correct Count |
| --- | --- | --- |
| gpt-4o | 92.00% | 138/150 |
| gpt-4o-mini | 86.67% | 130/150 |
| gpt-o1-preview | 86.00% | 129/150 |
| gpt-o1-mini | 89.33% | 134/150 |
| llama-3.1-70B-Instruct | 90.00% | 135/150 |
| llama-3.1-8B-Instruct | 82.00% | 123/150 |
| llama-3.2-3B-Instruct | 61.33% | 92/150 |
| llama-3.2-1B-Instruct | 39.33% | 59/150 |

Figure: Figure 7: The prompt used for post-processing the regression tasks inference outputs in the study.

## Appendix F Call for tasks

Here we provide material from the call for tasks.
Figure 8 shows a task proposal that we ask each contributor to initiate.
Task proposers need to provide a high-level description of the task, explain its importance and challenges, and list the datasets they plan to use along with the dataset licenses.
Figure 9 presents the standard README format for each task.
In the README, task proposers include the task introduction, its challenges, the dataset, and the evaluation metric.
Importantly, we also ask them to provide a table of state-of-the-art performance on the task, even if not achieved by universal spoken language models.
These results can serve as a reference for researchers to better understand the task’s difficulty and how it has developed so far.
Then, Figure 10 shows the JSON format we require to record task information.
For each task, our evaluation pipeline uses this JSON file as input to automatically download data from Huggingface, formatting it for subsequent use.

Figure: Figure 8: An example of a task proposal from a task contributor.
Refer to caption: extracted/5986474/task_proposal.png

Figure: Figure 9: An example of the README file required in the task contribution.
Refer to caption: extracted/5986474/readme-example.png

Figure: Figure 10: An example of the JSON file required in the task contribution.
Refer to caption: extracted/5986474/json_example.png

## Appendix G Audio duration distribution

Figure 11 shows the distribution of all audio files in Dynamic-SUPERB Phase-2.
Only about 7% of the audio files have a duration longer than 30 seconds, which reflects an inherent limitation of models that incorporate Whisper in their framework.

Figure: Figure 11: Duration distribution of all audios in Dynamic-SUPERB Phase-2.
Refer to caption: extracted/5986474/duration_distribution.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to the weakness part.
soundness: 3
strengths: This work is comprehensive and well presented. It brings the largest benchmark by far for speech and audio evaluation. It also conducts detailed experiments to assess the performance of several popular audio LLMs on the proposed benchmark.
summary: This paper builds on the prior work, Dynamic-Superb, by expanding its scope as an audio LLM benchmark to include a broader range of tasks centered on audio and speech understanding. It includes diverse tasks across speech, music, and general audio comprehension. The authors also evaluate several well-known audio LLMs, including SALMONN and Qwen-Audio, on this benchmark, showing that none of the current open-source audio LLMs can yet achieve satisfactory performance across all tasks.
weaknesses: There're a few possible issues that may further improve this paper.
- What is the motivation of creating this new benchmark? How will it guide the research community in advancing audio LLM research? I think the authors could emphasize this point further.
- I understand that the number of tasks has significantly expanded in Phase 2 compared to Phase 1. However, what is the primary focus of these additional tasks? Are there specific challenges unresolved in Phase 1 that Phase 2 addresses?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: My questions are listed above.
soundness: 3
strengths: 1. Dynamic-SUPERB Phase-2 is the largest and most comprehensive benchmark for instruction-based universal speech models. It encompasses a wide range of tasks across speech, music, and audio, all paired with natural language instructions to evaluate models' cross-modal instruction-following abilities. This is a well-motivated and forward-looking approach that aligns with future trends in universal speech models, offering strong guidance for the field.
2. The paper is clearly written and easy to follow.
summary: This paper presents Dynamic-SUPERB Phase-2, a benchmark designed to evaluate the capabilities of instruction-based universal speech models. By expanding the task count of its predecessor to 180, it becomes the largest benchmark for speech, music, and audio, encompassing a diverse array of tasks contributed by the global research community. This work is poised to drive further advancements in developing universal spoken language models and their evaluation methodologies.
weaknesses: 1. The benchmark lacks tasks for audio, speech, or music generation.
2. A primary concern is that the evaluation metrics heavily rely on large language models (LLMs) as referees. The reliability of these metrics is highly dependent on the capabilities of the LLMs themselves, which affects the benchmark's robustness and comparability. This reliance may also limit the benchmark's ability to expand to more complex tasks.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: * The paper text reports that In Figure 3, Whisper-LLaMa performs best in speech recognition and understanding. However, when we look at Table 2, it is way behind SALMONN in phone recognition. Looking at ASR performances across tables, it does seem that SALMONN mostly falls behind on a few non-English transcriptions (SUPERB OOD Asr Zh). I wonder if there is a more nuanced story here than is told in S5.1.

* Table 7 has inconsistent WER reporting, ie "Children Song Transcript" has WER not scaled by 100% while it is scaled in other places.

* It would be nice to have the "N/A rate" reported for the models.
soundness: 3
strengths: * The paper tackles perhaps one of the most important problems in the audio-LLM space: the lack of audio-text benchmarks. In contrast to text-LLMs, where sets of hundreds of tasks are readily available, audio-LLM development is hindered by the lack of standard testing suites.
* I find the organization of the benchmarks to be thoroughly thought through.
* The paper runs a nice study on comparing off-the-shelf models, thus validating the proposed benchmarks.
* The paper validates referee model selection against human preferences.
summary: This paper introduces a "Dynamic-SUPERB Phase-2", a benchmark for evaluating of audio-in-capable LLMs. This benchmark contains 180 tasks that cover a a large variety of tasks, spanning from music and audio classification to speech recognition, nonce word detection, intent and age classification, etc. These tasks were collected collaboratively. The audio-LLM are assumed to take in text instruction, an audio snippet, and produce text output. The interaction with the model is performed solely by the instruction, no finetuning is assumed. Depending on the task, the output encodes either classification label, regression value, or a natural language string. The benchmark is split into (a) full version, (b) an easier to operate representative sub-sample of tasks.

Given the benchmark, the authors also run a study comparing several off-the-shelf audio-capable LLMs, such as AudioQWEN, SALMONN, WhisperLLAMA, etc. One particular issue encountered is that various audio-LLMs would produce outputs in different formats, so the authors leverage another textual LLM to "translate/interpret" their outputs to the desired format.
weaknesses: * Using GPT-4o as a referee, I believe, jeopardizes reproducibility of the reported results and usefulness of the paper. Imagine that in two years someone wants to compare their model to the benchmark numbers reported in this paper. Would GPT-4o still be around? Would this particular version of it be still accessible? This is even more puzzling, since, according to Appendix E, LLaMA-3.1-70B-Instruct is extremely close to GPT-4o here. It seems to me that using an openly accessible model here would be extremely beneficial.

* Regarding the model evaluation study: it is not clear how sensitive the tested models are wrt the prompts used. It would be very nice to have a small study on that.

* Whenever the evaluated model doesn't follow the instruction, its score is multiplicatively penalized by (1-N/A) or 1.0/(1-N/A) (depending on the metric). This is convenient as allows to have a single number but it totally hides how good models are at instruction-following.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: What is the justification of using Whisper + LLaMA as the baseline? There should be some explanation why this particular architecture has been chosen.

L.267: I think the phrase for distinguishing sources "speech is produced by humans" could be rephrased, as humans could also "produce" other non-speech sounds. Speech should rather be connected to the vocal chords.
L.286: The example of Speech Enhancement to mitigate does not become clear. Above it is stated that the models should produce text as output (which can maybe be interpreted as lists). Under Speech Enhancement for mitigating noise (unless you mean only e..g, SNR estimation) I would expect an audio output.
soundness: 3
strengths: The paper addresses an important point challenging the "universality" claim universal speech models. The authors claim to have gathered the largest benchmark by a large margin, improving on top of presumably their own work and provide a detailed analysis comparing several current speech models. The benchmark has the potential to have a large impact on the research community and it provides useful insights into the limitations of LLMs.
summary: The paper suggests a new (milestone of a) benchmark to test the audio analysis tasks of universal speech models. It presents an overview of 180 tasks, contributed by the research community and categorized into a a taxonomy, which all have in common that one or multiple audios, as well as a task prompt are presented to a to-be-evaluated model, which then provides a language response, aiming to solve the given task. The response is then evaluated by a post-processing pipeline, potentially including a "judge LLM". Besides providing the evaluation benchmark pipeline, the authors conducted an analysis of current universal (at times specialised) speech models. They conclude that no model excels at all tasks, but strengths and weaknesses can be observed across different models and tasks.
weaknesses: L.371: Maybe it is worth that the authors iterate why they didn't choose a more "direct" approach for the evaluation, in which the model is prompted to give a single word based on the fixed set of possible answers. In the case of emotion recognition, it could be "answer with a single word out of the choices 'happy, sad, neutral,...'", which would allow a more direct evaluation. Especially in the case of emotion recognition a non-pre-defined set of emotions will also make it difficult for the judge LLM (and humans for reference) to judge how accurate the result is. This might at least be considered as a second evaluation metric, this seems more or less as what is being done for regression, but why isn't this "extra" step directly asked from the model being evaluated? For sequences evaluation it is stated that the evaluation of unprocessed outputs "guarantees consistency and objectivity".

Section 5.1. It might be difficult to define for some tasks, but a comparison to chance level and/or state-of-the-art task-specific models and/or "perfect" scores would seem a very interesting insight. As is, it seems that the evaluation allows relative comparison between -quite elaborate- models, which allows to see which model is better amongst two, but doesn't really allow to see if the models are still doing reasonably well in these tasks and how much room for improvement there still is. For instance in the case of of spatial audio, none of the models are performing better than the baseline, but it not clear, if none of them or all of them are reasonably good at the task. This is somewhat done in section 5.2. but seems to be missing in 5.1.

Meta-review: Dynamic-SUPERB Phase-2 is the largest and most comprehensive benchmark for instruction-based universal speech models, covering a wide range of tasks across speech, music, and audio with natural language instructions.  It tackles the significant problem of the lack of audio-text benchmarks in the audio-LLM space, which hinders development compared to text-LLMs. The benchmark has the potential to significantly impact the research community by providing useful insights into the limitations of LLMs and challenging the "universality" claim of universal speech models.

Decision: Accept (Poster)