Title: AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound
Venue: NeurIPS 2025
Accepted: yes
Keywords: audio-language model

audio dataset

audio reasoning

semantics of sound
Abstract: Audio-language models have shown promising results in various sound understanding tasks, yet they remain limited in their ability to reason over the fine-grained semantics of sound. In this paper, we present AudSemThinker, a model whose reasoning is structured around a framework of auditory semantics inspired by human cognition. To support this, we introduce AudSem, a novel dataset specifically curated for semantic descriptor reasoning in audio-language models. AudSem addresses the persistent challenge of data contamination in zero-shot evaluations by providing a carefully filtered collection of audio samples paired with captions generated through a robust multi-stage pipeline. Our experiments demonstrate that AudSemThinker outperforms state-of-the-art models across multiple training settings, highlighting its strength in semantic audio reasoning. Both AudSemThinker and the AudSem dataset are released publicly.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AudSemThinker, a new audio-language model designed for structured reasoning over fine-grained auditory semantics. To support this, the authors introduce AudSem, a dataset curated for audio semantic reasoning tasks. The dataset is built using a multi-stage pipeline that filters caption-audio pairs to minimize data contamination, aiming to preserve zero-shot integrity. The proposed model shows improved performance across several tasks and is especially strong in music-related scenarios. Ablation studies demonstrate the utility of semantic decomposition and control over reasoning length (“thinking budget”).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Timely problem focus: The paper addresses a meaningful challenge in audio-language modeling: moving beyond perception into structured semantic reasoning.\n\n- Dataset construction: The paper attempts to mitigate data contamination, a known issue in zero-shot evaluation, which is commendable.\n\n- Model design and ablation: The use of semantic descriptor decomposition and thinking budget analysis provides some insight into model behavior and optimization.\n\n- Performance improvement: The proposed method outperforms selected baselines under the presented settings.\n\nWeaknesses\n- Incomplete dataset documentation:\n\n- - Section 3 only includes 3.1, making the subsection label unnecessary.\n\n- - There is no manual validation or dataset example shown, leaving the actual quality of the generated dataset unclear. It's unclear whether the questions require reasoning, perceptual matching, or are language-biased. Are audio inputs essential or optional?\n\n- Omission of key related work: The Omni-style models like Qwen-Omni and Qwen2.5-Omni are not cited or compared, even though they currently surpass older models like Audio-CoT in reasoning capabilities.\n\n- Outdated baseline: Gemini 1.5 Pro, used as a baseline, was deprecated well before NeurIPS submission deadlines. Newer versions like Gemini 2.0 Pro have been shown to outperform many models, including the proposed one.\n The claim of \"state-of-the-art\" is misleading without up-to-date comparisons.\n\nBenchmark scope is limited:\n- The proposed model is not tested on more challenging reasoning benchmarks, such as MMAR (multi-step audio reasoning) or the OmniBench reasoning questions with image captions.\nWithout these evaluations on more difficult reasoning tasks, it's hard to gauge true reasoning capabilities under generative or compositional settings.\n\nQUESTIONS:\n- Please clarify the role of audio in the tasks. Are there examples where the model must truly listen to the audio to answer the question, or can it succeed via language priors alone? Including such examples would help assess audio dependency and reasoning depth.\n\n- Show dataset examples and quality checks. Manual inspection results (e.g., sample correctness, audio-caption match) would greatly increase confidence in the dataset's usefulness.\n\n- Update baselines. Why is Gemini 2.0 Pro omitted? It is publicly available and more relevant than Gemini 1.5 Pro. Comparing with stronger baselines like Qwen2.5-Omni is essential if you are claiming SOTA.\n\n- Include harder reasoning benchmarks. Can you evaluate your model on MMAR or OmniBench? These tasks involve multistep reasoning and would more rigorously validate your model’s claimed strengths.\n\n- Revise related work. Include recent, competitive Omni-style models and discuss how your approach differs or complements them in terms of reasoning ability.\n\nLIMITATIONS:\nNo detailed discussion on limitations. Besides, the proposed dataset should include a descrioption on copyright and ethic issue on the YouTube audio.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-18"}

comments: SUMMARY:
The paper introduces AudSemThinker, an audio language model that performs structured reasoning over semantic audio descriptors inspired by human cognition. It also introduces a novel dataset called AUDSEM curated from YouTube closed captions to reduce overlapping with existing audio datasets which are mainly sourced from AudioSet and Freesound. The proposed model is trained on AudSem on both SFT and GRPO training paradigms. AudSemThinker acheives strong results on MMAU and AudioBench datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
1. A novel dataset AudSem, creation of this dataset uses a multi-modal approach which makes the dataset more reliable.
2. The approach to semantic reasoning based architecture is novel.
3. Evaluates on most of the state-of-the art audio language models.

Weaknesses:
1. Same LLM used for caption generation and validation. In my view, this is not a good practice to validate captions, LLM will mark it's own generation as correct most of the time.

QUESTIONS:
1. Could you elaborate more on how captions of larger audios are generated?
2. I am curious how is the performance on music dataset so good as the distribution of music in the dataset characteristics chart shows the dataset contains more of sound events. I would also be interested to know more about the dataset distribution for sound events. Did you filter out the events to balance the ratio?
3. As the dataset is created from the closed youtube captions. Are you going to open-source it?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AUDSEMTHINKER, an audio-language model that enhances audio understanding and reasoning through structured semantic reasoning inspired by human cognitive perception of sounds.\n The authors also introduce AUDSEM, a new large-scale dataset curated from YouTube closed captions, designed to be more diverse and to minimize overlap with existing datasets. \nThe AUDSEMTHINKER model is trained using both supervised fine-tuning and reinforcement learning, taking advantage of semantic descriptors (such as “who, what, how, and where/when”). \nExperiments demonstrate that their approach achieves good performance on multiple audio-language benchmarks, showing improved interpretability and reasoning over the semantics of sound.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper introduces a new audio-language model, based on structured semantic reasoning inspired by human auditory cognition (“who/what/how/when/where”). It also presents a new, diverse dataset (AUDSEM) created via a rigorous multi-modal, multi-stage pipeline, with careful filtering to reduce data overlap and leakage. In my opinion, the data is of great value. \n2. The work is well-documented, highly reproducible, and open-source commitments are made. \n\n**Weaknesses**\n\n1. The paper does not report results on the MMAR[1] benchmark, which limits the completeness of its evaluation on the core idea (thinking process and audio reasoning). \n2. There is no comparison with the recent Omni-R1[2] model, which achieves strong performance on MMAU (>70) using only answer reward with GRPO. This makes it unclear how the proposed approach stands relative to the state-of-the-art models.\n3. There appears to be an inconsistency between Table 3 (SFT + AUDSEMTHINKER-QA (ours)) and Table 4 (SFT + Full + semantic + LoRA) results for MMAU Test-Mini; the numbers do not match up as expected and should be clarified. \n\n[1] MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix\n[2] Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?\n\nQUESTIONS:\nHow did the authors extract the audio segments from YouTube videos? Since video data is typically sparsely sampled, how did the pipeline ensure that the extracted audio clips are aligned with the most relevant or “key” audio events described in the captions? Could the authors clarify the process by which they locate and select these important audio segments?\n\nLIMITATIONS:\nSee above. In my opinion, the evaluation (on MMAR) and performance improvement (compared with Omni-R1) are limited.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of improving the reasoning capability of audio language models on understanding semantics of sounds focusing mostly on acoustic and musical sounds (i.e, focus is not speech recognition). They propose a new dataset called AudSem and use this to further train the Qwen2.5-Omni 7B model using Parameter Efiicient Fine tuning (PEFT) with Low Rank Adapters (LoRA). Two training paradigms are explored - Supervised Fine-tuning (SFT) and Reinforcement Learning with Group Relative Policy Optimization (GRPO) to see which paradigm can extract the most out of their dataset. Performance of these models is assessed on publicly available benchmarks - MMAU and AudioBench.\n\nOverall the most important contribution from this work is the AudSem Dataset and the necessary evaluation to show its utility. This dataset is mined from Youtube. this dataset is then filtered for sound and music descriptors with square brackets in closed captions and corresponding video and audio snippets are extracted. These extracted snippets are then labelled for sound semantics using expert audio and video classification models. A large Multimodal LLM (Qwen2.5 72B Instruct) is used to generate a caption by consuming the semantic information from both video and audio. Finally this dataset is further filtered using CLAP model to ensure relevance of generated semantic descriptions with the original labels.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. A sound pipeline to process and construct the dataset with minimal manual intervention using both video and audio modalities. \n2. Paper is well written, easy to understand and widely accessible. \n3. The curated dataset has minimal overlap with existing popular audio language datasets. \n4. Interesting to see the results of SFT vs GRPO and to know that there is no clear winner. \n\nWeaknesses:\n1. The dataset has mostly been labelled using expert models - these expert models such as BEATs, LP-MusicCaps are far from being perfect. It is also not clear which specific version of BEATs the authors are using and it is possible that this model is overfitted on to AudioSet. I am also guessing that the authors used this model to generate the dataset characteristics shown in Figure 3. It is not clear what further analysis or cleaning the authors did to verify the quality of the labels from BEATs. A similar limitation holds for CLAP models and using this to filter the dataset is also problematic as there will be domain mismatch in the language of closed captions and that from Qwen2.5 72B model. It is not clear if CLAP's text encoder is capable of such generalization. \n2. It is not clear if AudSem is a high quality dataset or simply a large enough dataset which has sufficient quality to help existing models do better on MMAU and AudioBench datasets. It this is the case, the authors would need to show on a few other existing models ( at least one more other than Qwen2.5 models) that similar gains can be observed on these benchmarks.\n\nQUESTIONS:\nRaised in the Strengths and Weakness section.\n\nLIMITATIONS:\nSocietal Impact has not been discussed. At a cursory level the limitations of all underlying expert models used in data curation and training would hold as limitations of this model. Further it would be good to bring out known limitations of the dataset and discuss whether quality or quantity is helping in improving the reasoning capability of the trained models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}