AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large
  Language Models
Authors: name: Kai Li, name: Can Shen, name: Yile Liu, name: Jirui Han, name: Kelong Zheng, name: Xuechao Zou, name: Zhe Wang, name: Xingjian Du, name: Shun Zhang, name: Hanjun Luo, name: Yingbin Jin, name: Xinxin Xing, name: Ziyang Ma, name: Yue Liu, name: Xiaojun Jia, name: Yifan Zhang, name: Junfeng Fang, name: Kun Wang, name: Yibo Yan, name: Haoyang Li, name: Yiming Li, name: Xiaobin Zhuang, name: Yang Liu, name: Haibo Hu, name: Zhuo Chen, name: Zhizheng Wu, name: Xiaolin Hu, name: Eng-Siong Chng, name: XiaoFeng Wang, name: Wenyuan Xu, name: Wei Dong, name: Xinfeng Li

The rapid advancement and expanding applications of Audio Large Language
Models (ALLMs) demand a rigorous understanding of their trustworthiness.
However, systematic research on evaluating these models, particularly
concerning risks unique to the audio modality, remains largely unexplored.
Existing evaluation frameworks primarily focus on the text modality or address
only a restricted set of safety dimensions, failing to adequately account for
the unique characteristics and application scenarios inherent to the audio
modality. We introduce AudioTrust-the first multifaceted trustworthiness
evaluation framework and benchmark specifically designed for ALLMs. AudioTrust
facilitates assessments across six key dimensions: fairness, hallucination,
safety, privacy, robustness, and authentication. To comprehensively evaluate
these dimensions, AudioTrust is structured around 18 distinct experimental
setups. Its core is a meticulously constructed dataset of over 4,420 audio/text
samples, drawn from real-world scenarios (e.g., daily conversations, emergency
calls, voice assistant interactions), specifically designed to probe the
multifaceted trustworthiness of ALLMs. For assessment, the benchmark carefully
designs 9 audio-specific evaluation metrics, and we employ a large-scale
automated pipeline for objective and scalable scoring of model outputs.
Experimental results reveal the trustworthiness boundaries and limitations of
current state-of-the-art open-source and closed-source ALLMs when confronted
with various high-risk audio scenarios, offering valuable insights for the
secure and trustworthy deployment of future audio models. Our platform and
benchmark are available at https://github.com/JusperLee/AudioTrust.