AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Authors: name: Xilin Jiang; name: Sukru Samet Dindar; name: Vishal Choudhari; name: Stephan Bickel; name: Ashesh Mehta; name: Guy M McKhann; name: Adeen Flinker; name: Daniel Friedman; name: Nima Mesgarani
Published: 2025-02-24 03:06:45

Abstract: Auditory foundation models, including auditory large language models (LLMs),
process all sound inputs equally, independent of listener perception. However,
human auditory perception is inherently selective: listeners focus on specific
speakers while ignoring others in complex auditory scenes. Existing models do
not incorporate this selectivity, limiting their ability to generate
perception-aligned responses. To address this, we introduce Intention-Informed
Auditory Scene Understanding (II-ASU) and present Auditory Attention-Driven LLM
(AAD-LLM), a prototype system that integrates brain signals to infer listener
attention. AAD-LLM extends an auditory LLM by incorporating intracranial
electroencephalography (iEEG) recordings to decode which speaker a listener is
attending to and refine responses accordingly. The model first predicts the
attended speaker from neural activity, then conditions response generation on
this inferred attentional state. We evaluate AAD-LLM on speaker description,
speech transcription and extraction, and question answering in multitalker
scenarios, with both objective and subjective ratings showing improved
alignment with listener intention. By taking a first step toward
intention-aware auditory AI, this work explores a new paradigm where listener
perception informs machine listening, paving the way for future
listener-centered auditory systems. Demo and code available:
https://aad-llm.github.io.