AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (2024)

Abstract: Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as automatic speech recognition, and lack an assessment of the openended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (Audio InstRuction Benchmark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIRBench encompasses two dimensions: foundation and chat benchmarks. The former consists of 19 tasks with approximately 19k singlechoice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended questionand-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research. * Equal contribution. † Intern at Alibaba. ‡ Corresponding to Zhou Zhao (zhaozhou@zju.edu.cn) and Chang Zhou (ericzhou.zc@alibaba-inc.com). Dataset and evaluation code are available at https://github.com/OFA-Sys/AIR-Bench.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1979–1998 August 11-16, 2024 ©2024 Association for Computational Linguistics
* Equal contribution. † Intern at Alibaba. ‡ Corresponding to Zhou Zhao (zhaozhou@zju.edu.cn)
and Chang Zhou (ericzhou.zc@alibaba-inc.com).
Dataset and evaluation code are available at https://github.com/OFA-Sys/AIR-Bench.

heading: 1 Introduction
text: Recent advancements in artificial general intelligence have been significantly driven by the emergence of large language models (LLMs) (Brown et al., 2020; OpenAI, 2022, 2023; Chowdhery et al., 2022; Anil et al., 2023; Touvron et al., 2023a,b; Bai et al., 2023a). These models exhibit remarkable abilities in retaining knowledge, engaging in intricate reasoning, and solving problems following human intents. Motivated by the striking progress in large language models (LLMs), the domain of large audio-language models (LALMs) has undergone a revolutionary transformation. To perceive and comprehend rich audio signals and further generate textual responses following human instructions, many works have been proposed, such as SALMONN (Tang et al., 2023a), BLSP (Wang et al., 2023a), Speech-LLaMA (Wu et al., 2023a), and Qwen-Audio (Chu et al., 2023), showcasing promising capabilities for audio-central dialogues.
However, previous LALMs (Tang et al., 2023a; Wang et al., 2023a; Wu et al., 2023a; Chu et al., 2023; Huang et al., 2023b; Shen et al., 2023; Gong et al., 2023; Wang et al., 2023b) have predominantly concentrated on evaluation in specific fundamental tasks. The absence of a standardized benchmark for assessing the generative instructionfollowing abilities of these models has resulted in a reliance on showcasing examples or releasing the chat models for public experimentation to demonstrate their conversational skills. This approach poses significant challenges for conducting fair and objective comparisons across different research endeavors. Moreover, it tends to obscure the models’ existing limitations, impeding the ability to monitor advancements within the domain of LALMs.
For evaluation in audio domains, the majority of research efforts have concentrated on the creation
1979
of benchmarks tailored to individual tasks such as LibriSpeech (Panayotov et al., 2015) and Common Voice benchmark (Ardila et al., 2019) for ASR. Beyond task-specific ones, benchmarks like SUPERB (Yang et al., 2021a) and HEAR (Turian et al., 2021) have been designed to test the versatility of self-supervised learning models in a wide variety of tasks. Regarding the assessment of LALMs’ ability to follow instructions, to the best of our knowledge, Dynamic-SUPERB (Huang et al., 2023a) is the only benchmark devoted to this aspect. Nevertheless, Dynamic-SUPERB only focuses on human speech processing and does not extend to the assessment of models’ capabilities in producing open-ended generations such as dialogues.
In this paper, we present AIR-Bench (Audio InstRuction Benchmark), a novel benchmark designed to evaluate the ability of LALMs to comprehend various audio signals and to interact following instructions. AIR-Bench is characterized by three primary features: 1) Comprehensive audio signals coverage. AIR-Bench offers comprehensive coverage of audio signals, including human speech, natural sounds, and music, ensuring a comprehensive evaluation of LALMs’ capabilities. 2) Hierarchical Benchmark Structure. The benchmark consists of foundation and chat benchmarks. The foundation benchmark comprises 19 distinct audio tasks with over 19,000 single-choice questions, with each question focusing only on a specific foundational ability. GPT-4 (OpenAI, 2023) extends the questions and candidate choices using dedicated designed prompts. The chat component consists of over 2,000 audio-prompted open-ended questions. To enhance the complexity of the audio and achieve a closer resemblance to the intricate audio encountered in real-life situations, we propose a novel audio mixing strategy that incorporates loudness control and temporal dislocation. Specifically, we adjust the loudness and introduce different temporal offsets during the mixing process of two audio clips. The resulting variations in relative loudness and temporal location are then recorded as additional meta-information, contributing to a more comprehensive textual representation of the audio. The quality of data is upheld through automated filtering by GPT-4, followed by manual verification. 3) Unified, objective, and reproducible evaluation framework. Models are required to generate hypothesis sequences directly across both benchmarks to align more accurately with practical scenarios. Then, we employ GPT-4 to generate
reference answers given meta-information through carefully constructed prompts. Given references and hypotheses, following Liu et al. (2023b); Bai et al. (2023b), we use GPT-4 (OpenAI, 2023) to judge whether the choice is correct for the foundation benchmark or score hypotheses for the chat benchmark. We further perform a second scoring by swapping their positions to eliminate the position bias. Based on comprehensive experiments on 9 LALMs, we observe that existing LALMs either have limited audio understanding or instructionfollowing capabilities, leaving significant room for improvement in this field.
Our contribution is summarized below:
• AIR-Bench is the first generative evaluation benchmark for large audio-language models, encompassing a wide array of audio such as speech, natural sounds, and music. AIRBench is a large and hierarchical benchmark, consisting of the foundation benchmark with 19 audio tasks and over 19k single-choice questions, alongside a chat benchmark with over 2k meticulously curated open-ended audio questions for comprehensive evaluation.
• We propose a novel audio mixing strategy with loudness control and temporal dislocation to enhance the complexity of the audio.
• A unified, objective, and reproducible evaluation framework has been developed to assess the quality of generative hypotheses.
• We conducted a thorough evaluation of 9 models for the purpose of benchmarking. The evaluation code, datasets, and an open leaderboard will be made publicly available soon.

heading: 2 Related Work
text: Benchmarks for Audio Processing. Previous studies have primarily focused on evaluating the specific fundamental capabilities of models. In the field of speech processing, automatic speech recognition is one of the most popular tasks, with representative benchmarks including Librispeech (Panayotov et al., 2015), Common Voice (Ardila et al., 2019), and FLEURS (Conneau et al., 2022). Additionally, there are various benchmarks available for different speech processing tasks such as speech-to-text translation (Wang et al., 2020a,b; Jia et al., 2022) and emotion recognition (Cao et al., 2014; Livingstone and Russo,
2018). In the field of sound processing, several benchmarks have emerged such as Clotho (Drossos et al., 2020) and Audiocaps (Kim et al., 2019a) for automatic audio captioning, and AVQA (Yang et al., 2022) for sound question answering. In the domain of music processing, numerous datasets are available, including MusicCaps (Agostinelli et al., 2023) for automatic music captioning, and MUSICAVQA (Li et al., 2022) for music question answering. Note that most existing question-answering benchmarks, such as Clotho-AQA, AVQA, and MUSIC-AVQA, have highly constrained answer formats for ease of close-ended evaluation or conversion into classification tasks, rather than supporting open-ended generation.
Besides the aforementioned datasets that focus on specific tasks, there are benchmarks like SUPERB (Yang et al., 2021b) and HEAR (Turian et al., 2022) for comprehensive evaluation of selfsupervised learning models. When it comes to assessing the ability of LALMs to follow instructions, Dynamic-SUPERB is the only benchmark dedicated to this aspect. However, Dynamic-SUPERB focuses on human speech processing and does not cover open-ended dialogue generation. In contrast, AIR-Bench is the first large-scale generative evaluation benchmark for large audio-language models, encompassing various audio types such as speech, natural sounds, and music.
Large Audio-Language Models following Human Instruction Recently, there has been significant interest in instruction-following end-toend audio-language models. Several models have emerged, each focusing on different audio domains. For instance, there are models specifically focusing on speech processing, such as SpeechGPT (Zhang et al., 2023), BLSP (Wang et al., 2023a), and LLaSM (Shu et al., 2023). Similarly, there are models tailored for sound processing, like LTU (Gong et al., 2023), and for music processing, such as LLark (Gardner et al., 2023). In contrast, SALMONN (Tang et al., 2023b) and Qwen-Audio (Chu et al., 2023) are trained using various audio types, showcasing strong universal audio understanding abilities. However, these models are evaluated on different fundamental tasks, making it difficult to conduct a fair comparison. Furthermore, these models rely on showcasing examples or public demos to demonstrate their conversational skills and do not perform rigorous experiments to evaluate their instruction-following
abilities. To address these issues, this paper introduces AIR-Bench, which proposes two benchmarks - the foundation benchmark and the chat benchmark, enabling a fair comparison of the models’ foundational abilities and their high-level instruction-following capabilities respectively.

heading: 3 AIR-Bench
text: There exist three unique characteristics that differentiate AIR-Bench from existing benchmarks for audio understanding: i) AIR-Bench is the first work to incorporate task evaluation from all types of audio in a hierarchical taxonomy; ii) AIR-Bench is the first generative evaluation benchmark that handles the free-form output of LALMs; iii) AIRBench adopts GPT-4-based automatic evaluation yielding trustworthy evaluation results with affordable cost. In Sec. 3.1, we present the hierarchical taxonomy of AIR-Bench and discuss the design philosophy behind it. In Sec. 3.2 and Sec. 3.3, we introduce how we collect the audio-central questionanswer pairs for foundation and chat tasks. In Sec. 3.4, we present the evaluation framework.

heading: 3.1 Overview
text: Chat interaction based on audio is a complex task that encompasses a variety of fundamental compe-
tencies. For instance, humans are able to respond to sound events due to their capacities for sound perception and common sense reasoning. Similarly, the ability to respond to others’ spoken words is predicated on foundational skills such as speech-totext recognition and emotion recognition. Based on the motivation, we propose the hierarchical benchmark AIR-Bench by dividing it into foundation and chat benchmarks. The fundamental one is designed to assess capabilities across individual subtasks, serving to diagnose weaknesses within the model, while the chat benchmark directly evaluates complicated audio-based open-ended questions. The data sample is denoted as (A,Q,R), where A denotes the audio, Q represents the query and R is the reference answer.
• Foundation benchmark: The purpose of the benchmark is to evaluate the individual capabilities of foundational tasks. To reduce the task difficulties and enable the evaluation of various models, we utilize the single-choice question-answering format. Specifically, the query Q is formed by concatenating a question q and candidate choices C, denoted as Q = (q, C). We curate a collection of 19 audio tasks that span multiple audio types, such as speech, music, and sound. These tasks include tasks like emotion recognition, acoustic scene classification, and music QA. 1
• Chat benchmark: The benchmark encompasses any form of question and answer pairs that could arise from audio signals, with the aim of reflecting the model’s ability to genuinely follow user instructions to perform perceiving, reasoning, and interacting within realworld applications. According to the type of audio, the benchmark is categorized into four dimensions: speech, sound, music, and mixed audio, where mixed audio refers to audio that is a mixture of multiple types of audio, such as human voice with background music.
The overview of AIR-Bench is shown in Fig. 1.

heading: 3.2 Foundation Benchmark
text: Data Source. We collected over 19k data samples for the foundation dimension, encompassing 19 different subtasks. The data source and statistics
1For transcription tasks such as ASR, we incorporate them into the chat benchmark since they are not suitable for the single-choice task format.
are provided in Table 1. To ensure a fair and comprehensive evaluation of each capability, we aimed for an even distribution of problems related to different abilities during the data collection process. All audio sources were obtained from the original dev or test subsets to prevent data leakage.
Single-choice Query and Reference. The query Q is formed by concatenating a question q and candidate choices C. For the question q, we mainly construct questions through GPT-4 (OpenAI, 2023), except for QA tasks since the datasets inherently contain questions and we can directly re-use them. Specifically, we design the prompt for the distinct task and provide three questions as demonstrations. Subsequently, GPT-4 generates additional diverse questions based on these inputs. The generated questions are manually reviewed, and 50 different questions are selected for each task. The variability in question format aims to evaluate the model’s ability to follow instructions rather than being overly reliant on specific templates. For each question, we further generate candidate choices C from different sources: 1) For tasks with choices in orig-
inal datasets like AVQA (Yang et al., 2022), we directly re-use it; 2) For classification tasks, we randomly select options from the predetermined set of categories to serve as candidate choices; 3) For other tasks, we prompt GPT-4 to generate candidate choices directly, consisting of one correct option and three incorrect options. We encourage these incorrect options to resemble the correct one, making the single-choice task more challenging. The reference answer is the golden correct choice. To avoid position bias, the candidate choices are randomly shuffled. We provide examples of each task in Table 5 of the Appendix.

heading: 3.3 Chat Benchmark
text: Data Source and Audio Mixing Strategy. As shown in Table 2, we have collected more than 2k data samples spanning various audio types including speech, sound, music, and mixed audio. The purpose of introducing mixed audio is to augment the complexity of the audio signals and make it closer to audio from real-world audio scenarios. To achieve this, we propose a novel mixing strategy involving loudness control and temporal dislocation, as illustrated in Fig. 2. Specifically, we can adjust the relative loudness and temporal relationship between two audio clips for mixing. Then, we can create a complex audio signal that combines their meta-information, such as speech transcription accompanied by a background music caption. Furthermore, the meta-information also includes labels indicating which audio clip is louder and
which is ahead in the temporal sequence.
Open-ended Query and Reference. To prompt GPT-4 to generate open-ended question-answer pairs for audio, we should interpret the rich information in each audio with texts. We collect all of meta-information such as gender, age, emotion, transcription, language for speech, caption for natural sound, and instrument, caption for music from the original dataset. Rather than relying on pretrained models to extract this meta-information for each audio clip, we adopt the ground truth metainformation to avoid potential errors.
After gathering meta-information about the audio, we manually construct prompts (see Appendix 5 for guiding GPT-4 in generating questionanswer pairs that specifically focus on different abilities). These prompts are carefully designed to ensure a comprehensive coverage of chat interactions, taking into consideration the diverse range of audio signals involved. We design the prompts to facilitate the generation of questions related to the perception and reasoning for different types of audio. For the natural sound, the prompts are further tailored to generate questions that involve determining appropriate responses to sound events within a specific scenario. For the music category, prompts are devised to elicit creative writing and story-generation questions based on music composition. To ensure the quality of the generated results, these prompts are designed in a manner that enables GPT-4 to automatically filter out responses that are not directly related to audio. Additionally, we manually reviewed all the question-answer pairs to ensure the quality of the questions and the reliability of the answers. The generated answers from GPT-4 are considered as references.

heading: 3.4 Evaluation Strategy
text: In this paper, we leverage a unified evaluation method, as shown in Fig. 3, by viewing both the single-choice question in the foundation benchmark, and the open-ended question in the chat benchmark, as the generation tasks for the purpose of better alignment with actual use case scenarios of LALMs. That is, given audio and questions, LALMs are required to directly generate the answers as hypotheses, rather than comparing the perplexity on the probability of different reference answers via teacher forcing. Automated and accurate evaluation of open-ended generation is a challenging problem. Traditional automatic metrics such
as WER, ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005) have shown a low correlation with human judgments (Liu et al., 2023a). Recently, LLM-based evaluation, such as GPT-4, shows better human preference alignment (Zheng et al., 2023; Liu et al., 2023a). In this work, we adopt referencebased GPT-4 evaluators to judge the generation quality of LALMs in the audio domain.
However, GPT-4 cannot be directly used as an evaluator since it cannot receive audio inputs. To address this limitation, we offer the GPT-4 model rich meta-information of audio to replace audio input. Subsequently, we present questions and employ GPT-4 to evaluate the hypotheses produced by LALMs. To ensure consistency and fairness for evaluation, each model’s answer is compared against the same reference answer for scoring. For the foundation benchmark, the reference answer is the golden choice, and we prompt the evaluator to determine whether the hypothesis is correct or not. For the chat benchmark, the reference answer is generated by GPT-4, and we prompt the evaluator to provide a score ranging from 1 to 10, based on the assessment of usefulness, relevance, accuracy, and comprehensiveness of the hypothesis. The prompts used in the evaluation process can be found in Appendix 5. Note that for the chat benchmark, the role of the reference is not to serve as the
ground truth answer, but rather as a reference for scoring by GPT-4, in order to stabilize its scoring. Additionally, to mitigate any potential position bias resulting from the order of hypothesis and reference, following Bai et al. (2023b), we perform a second scoring round by swapping their positions and then compute the average of the two scores. Unless otherwise specified, the GPT-4 evaluator is GPT-4 Turbo, the gpt-4-0125-preview version 2.

heading: 4 Experiments

heading: 4.1 Models
text: We evaluate the performance of various LALMs with instruction-following capabilities. These models are either open-sourced or accessible through public APIs, such as SpeechGPT (Zhang et al., 2023), BLSP (Wang et al., 2023a), SALMONN (Tang et al., 2023a), Qwen-AudioChat (Chu et al., 2023), and Qwen-Audio Turbo 3. Additionally, we consider large multi-modality models with audio understanding abilities like PandaGPT (Su et al., 2023), Macaw-LLM (Lyu et al., 2023), and NExT-GPT (Wu et al., 2023b). Besides, we also incorporate a sequential approach
2https://platform.openai.com/docs/models/gpt-4-and-gpt4-turbo
3https://help.aliyun.com/zh/dashscope/developerreference/qwen-audio-api
comprising Whisper-large-v2 (Radford et al., 2023) and GPT-4 Turbo (OpenAI, 2023) for tasks related to speech as a baseline. We evaluate the performance of all these models on both fundamental and chat benchmarks, utilizing their latest publicly available checkpoints. In cases of multiple checkpoints, we select the model with the largest parameter size. For all models, we directly follow their default decoding strategies for evaluation.

heading: 4.2 Main Results
text: The results of LALMs are presented in Table 3. The detailed results are shown in Table 6. For the foundation benchmark, we also conduct a comparison between the use of an exact matching strategy with our proposed GPT-4 alignment strategy. As an example, we try to match ‘B’, ‘B.’, ‘B)’, etc. with LALMs’ hypothesis for the exact matching. The results are shown in Table 4. We can find that BLSP and SALMONN have a high success rate in directly generating the choice, showcasing their strong ability to follow single-choice instruction.
However, we find that it is challenging to precisely extract the predicted choice from the hypotheses of other models due to significant variations in the output formats of different LALMs. However, with the assistance of GPT-4 as the evaluator, the success rate for all models can be improved to 100%.
According to Table 3, Qwen-Audio-Chat and Qwen-Audio Turbo demonstrate superior performance in the foundation benchmark, surpassing other models in the domains of speech, sound, and music. Second to the two models, PandaGPT and SALMONN also exhibit noteworthy performances. Regarding the chat benchmark, Qwen-Audio Turbo achieves the highest average score, followed by SALMONN and Qwen-Audio-Chat with scores of 6.11 and 6.08, respectively. Among these models, SALMONN outperforms others in terms of mixed audio understanding. Note that the speech dimension in the foundation benchmark includes tasks beyond speech transcriptions, such as speaker gender, age, and emotion prediction, while the speech in the chat benchmark primarily revolves around speech transcriptions. Thus, Whisper plus GPT-4 receives a relatively low score in the foundation benchmark but obtains the highest score in the chat benchmark.
Based on these results, we have several observations: 1) The existing LALMs either have limited audio understanding or instruction-following capabilities. For instance, Qwen-Audio Turbo achieves the highest average score in both foundation and chat benchmarks while the model displays a weak proficiency in following single-choice instructions such as often directly generating a full sentence semantically akin to one of the choices, and thus receives a low success rate for the exact matching; 2) As for chat abilities related only to speech tran-
scription, none of the models surpass the sequential baseline Whisper plus GPT-4.

heading: 4.3 Human Evaluation
text: To evaluate the consistency between the evaluations of GPT-4 and human judgments, we design experiments for both the foundation and chat benchmarks. For the foundation benchmark, we instruct the testers to determine which option aligns closest with the hypothesis. We then compare the option selected by human testers with the option chosen by GPT-4 to assess the extent of agreement. For this consistency analysis, we employed Qwen-AudioChat as a representative model and randomly selected 400 questions from the benchmark. These questions were then evaluated by three native English speakers. Additionally, we also compared the performance of GPT-4 with GPT-3.5 Turbo. As depicted in Figure 4 (a), GPT-4 Turbo, serving as the evaluator, exhibited a high level of consistency at 98.2% with human judgments. Comparatively, GPT-3.5 Turbo had a slightly lower consistency rate of 96.4%.
Regarding the chat benchmark, obtaining a numerical score on a scale of 1 to 10 directly from testers poses challenges. Therefore, we resort to a pairwise comparison of the models instead. Testers listen to audio and compare the performance of both models based on their usefulness, relevance, accuracy, and comprehensiveness to the given question, indicating their preference as either “A is better”, “B is better”, or “Both are equal”. Subsequently, we convert the GPT-4 scores into the same preference-based rating as the human testers for any two models. We then assess the consistency
between the two sets of results. For the chat benchmark, we conduct pairwise comparisons among Qwen-Audio-Chat, SALMONN, BLSP, and GPT4. We randomly select 200 questions and have them evaluated by three native English speakers. As depicted in Figure 4 (b), the pairwise preference consistency scored above 70%, demonstrating a high level of agreement.

heading: 4.4 Ablation Study of Positional Bias
text: In our evaluation framework, we adopt a strategy of scoring twice by interchanging the positions of the hypothesis and reference and calculating the average of the two scores. This approach helps mitigate the bias that may arise from the positional placement. The outcomes of these two evaluations are presented in Figure 4 (c). We observe that the GPT4 evaluator exhibits a clear bias in scoring when the hypothesis is placed before the reference. This highlights the importance of conducting a second scoring to account for addressing this bias.

heading: 5 Conclusion
text: In this paper, we present AIR-Bench, the first generative evaluation benchmark designed specifically for audio-language models. AIR-Bench comprises 19 audio tasks with over 19k single-choice questions in the foundation benchmark, as well as over 2k open-ended audio questions in the chat benchmark. Notably, the benchmark covers diverse audio types such as speech, natural sounds, and music. We also propose a novel audio mixing strategy to simulate audio from real-world scenarios more accurately. A standardized, objective, and reproducible evaluation framework is employed to au-
tomatically assess the quality of hypotheses generated by LALMs. We conduct a thorough evaluation of 9 prominent open-source LALMs. Additionally, we plan to launch and maintain a leaderboard that will serve as a platform for the community to access and compare model performance consistently over time.

heading: 6 Limitations
text: The objective of AIR-Bench is to develop a largescale, extensive and generative evaluation framework that encompasses a wide range of audio domains and tasks. However, AIR-Bench currently has several limitations. Firstly, it does not incorporate tasks involving multiple audio comparisons, such as assessing music coherence, for both the foundation and chat benchmark. Besides, AIRBench does not encompass the evaluation of multiturn dialogues that may involve multiple audio inputs. For evaluation, AIR-Bench relies on a powerful and robust evaluator such as GPT-4. However, the availability and accessibility of the GPT-4 API are external factors beyond our control. In the event that GPT-4 transitions to a closed-source model or implements higher pricing standards in the future, alternative evaluators will need to be explored and considered.

heading: 7 Ethical Considerations
text: The AIR-Bench initiative uses publicly available datasets to create a collection of relevant questionand-answer data. It then uses automated methods to evaluate this data, which is a more efficient alternative to manually evaluating it. However, there are challenges with this automated evaluation approach, including the potential for data misuse and the introduction of biases. To prevent data misuse, we follow the licenses and usage guidelines associated with the original open-source materials when generating related data. It’s important to point out that the automated evaluation could be biased. These biases may come from the datasets themselves or the scoring algorithms used, causing differences between automated evaluation results and human judgment. Therefore, the outcomes obtained from automated evaluations should be viewed with caution and used as a general benchmark, rather than a definitive measure.

heading: A Detailed Results of Foundation Benchmark
text: In Table 6, we delineate the performance assessment for each model across the various tasks on the foundation benchmark. With the exception of Speaker Gender Recognition and Synthesized Voice Detection, which are binary-choice tasks, all other tasks necessitate a selection from four options. As such, a random selection in the Speaker Gender Recognition and Synthesized Voice Detection datasets would theoretically achieve an accuracy of 50%, while the expected accuracy for random choices across the remaining datasets stands at 25%. Consequently, any performance metrics that approximate these random baselines are indicative of an absence of discernible proficiency in the respective tasks.

heading: B GPT Prompts for the Chat benchmark
text: In Figure 5, we display the carefully crafted prompts that we have developed on our chat benchmark. The figure is divided into two sections, the upper section contains prompts designed specifically for generating question-answer pairs related to reasoning, while the lower section features prompts aimed at assessing the chat performance scores of the models.
When generating questions and reference answers, we guide the process by specifying the type of questions to be elicited, allowing GPT-4 to automatically exclude data that is less amenable to question formulation. For the evaluation of the chat performance scores, we instruct GPT-4 to take a multifaceted approach, scoring both the reference answers and the model responses. This ensures that the reference answers consistently serve as a standard for comparison.

heading: C Prompts Engineering for GPT Scoring
text: In this section, we partially demonstrate the process of adjusting the prompt aimed at assessing the chat performance scores of the models.
• If we streamline our prompt by removing the descriptions pertaining to helpfulness, relevance, accuracy, and comprehensiveness, specifically by omitting "Please rate the helpfulness, relevance, accuracy, and comprehensiveness of their responses." and "In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.", we found that across multiple tests, many responses that were originally scored a perfect 10 were downgraded to a 9 or 8, while the unequivocally incorrect responses saw their scores rise from an initial 1 to a 2 or 3. This suggests that including these ’superfluous’ descriptions aids the model in assigning more precise scores during the evaluation process and helps to avoid ’normalization’ of scores.
• If we change the positioning, such as moving the entire [Detailed Audio Description] section behind the [Question] and [Answer], or swapping the positions of [Question] and [Answer]; these alterations impact the scoring, turning originally correct evaluations incorrect. Absolutely correct answers were inexplicably awarded scores as low as 5, whereas absolutely incorrect responses occasionally received scores around 5 as well. Therefore, our conclusion is that the prompt exhibits a strong sensitivity to the permutation of positions. Minor punctuation or grammatical errors do not affect the scoring.

heading: D Examples of the Foundation Benchmark
text: In Table 5, we present data examples for each task within the foundation benchmark.

heading: E Examples of LALMs’ responses
text: In Figure 6, we illustrate a representative response from various models on the foundation benchmark. The upper portion of the figure displays the question along with the metadata for the corresponding audio. This metadata is not provided as input to the models under evaluation, the models only have access to the audio and the question posed. The lower two columns of the figure document the responses from the 9 models being tested. Similarly,
an example of responses from various models on the chat benchmark can be seen in Figure 7.

heading: F Details in Human Evaluation
text: We conducted a pairwise crowd worker evaluation to assess the alignment between the judgments derived from GPT-4 and those of human evaluators for both the foundation and chat benchmarks. Each pair of evaluations was scrutinized by three native English-speaking judges. During the evaluation process, we required that the entire test be conducted in a quiet environment, with human evaluators wearing headphones to listen to the audio and to isolate noise. After obtaining the test results, we conducted sample feedback; if we identified any instances of erroneous annotations, we would report back to the outsourcing platform for them to carry out a re-evaluation.
• For the foundation benchmark, we randomly selected 400 questions from the pool of model responses. These were accompanied by both GPT-3.5 and GPT-4 alignment results. Evaluators were instructed to ascertain whether the responses provided by GPT-3.5 Turbo and GPT-4 Turbo was accurate. The screenshots of instructions for the foundation benchmark is shown in Figure 8.
• For the chat benchmark, we randomly chose 200 dialogues from the responses generated by Qwen-Audio-Chat, SALMONN, BLSP, and GPT-4, respectively. Evaluators were tasked with determining which model exhibited superior or equivalent performance. The screenshots of instructions for the chat benchmark is shown in Figure 9.
• For the chat benchmark, we further analyzed correlation with human judgment based on task and audio type. After conducting a statistical analysis of the randomly selected QA pairs, we found that Speech accounts for 42%, Sound for 22%, Music for 16%, and Mixed Audio for 20%. To further confirm the association between human judgment and audio type, we categorized the results from Figure 4(b) by audio type. As shown in Table 7, the statistical results presented in the table indicate that QAs involving Music and Mixed Audio categories tend to have slightly higher alignment most of the time, whereas QAs involving Sound and Speech categories tend to have slightly
lower alignment most of the time. We speculate that the reasons for the discrepancies might be: there are many situational questions in the Sound category QAs (such as ’What would you do if you heard this sound’), and many reasoning questions in the Speech category QAs. These more complex questions pose relatively greater challenges for GPT’s evaluation.

heading: Future Work
text: In the event that GPT-4 transitions to a closed-source model or implements higher pricing standards in the future, alternative evaluators will need to be explored and considered