Title: FastALM: Hierarchical Frame Q-Former for Effective Audio Modality Adaptation
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Audio Language Model (ALM), Q-Former, Multimodal

Abstract: Recent advances in large language models (LLMs) have demonstrated human-expert-level capabilities, driving significant interest in their potential for achieving artificial general intelligence (AGI). In particular, there is growing momentum in adapting LLMs to various modalities—including vision, video, and audio—through the development of multimodal LLMs (MLLMs). However, existing audio-language models (ALMs) research has largely overlooked cost-effective adaptation strategies for leveraging LLMs in the speech domain. In this paper, we propose FastALM, a lightweight yet efficient ALM designed for effective understanding and reasoning over long-form audio. To address the challenge of aligning high-frame audio features with LLM, we introduce the hierarchical frame querying transformer (HFQ-Former), which compresses frame-level audio features while capturing both local and global context. Furthermore, we present a novel three-stage training strategy that enhances generalization across a wide range of audio-related tasks. Experimental results demonstrate that FastALM achieves competitive or superior performance compared to state-of-the-art models, while operating with significantly fewer FLOPs and parameters and representing audio with only 1.67 tokens per second. The source code and model checkpoints are available at https://anonymous.4open.science/r/FastALM-1D6B.