How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Authors: name: Ke-Han Lu, name: Szu-Wei Fu, name: Chao-Han Huck Yang, name: Zhehuai Chen, name: Sung-Feng Huang, name: Chih-Kai Yang, name: Yi-Cheng Lin, name: Chi-Yuan Hsiao, name: Wenze Ren, name: En-Pei Hu, name: Yu-Han Huang, name: An-Yu Cheng, name: Cheng-Han Chiang, name: Yu Tsao, name: Yu-Chiang Frank Wang, name: Hung-yi Lee
Organization: National Taiwan University

Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.