Title: Mellow: a small audio language model for reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: audio-language

multimodal learning

audio reasoning

audio processing
Abstract: Multimodal Audio-Language Models (ALMs) can understand and reason over both audio and text. Typically, reasoning performance correlates with model size, with the best results achieved by models exceeding 8 billion parameters. However, no prior work has explored enabling small audio-language models to perform reasoning tasks, despite the potential applications for edge devices. To address this gap, we introduce Mellow, a small Audio-Language Model specifically designed for reasoning. Mellow achieves state-of-the-art performance among existing small audio-language models and surpasses several larger models in reasoning capabilities. For instance, Mellow scores 52.11 on MMAU, comparable to SoTA Qwen2 Audio (which scores 52.5) while using 50 times fewer parameters and being trained on 60 times less data (audio hrs). To train Mellow, we introduce ReasonAQA, a dataset designed to enhance audio-grounded reasoning in models. It consists of a mixture of existing datasets (30\% of the data) and synthetically generated data (70\%). The synthetic dataset is derived from audio captioning datasets, where Large Language Models (LLMs) generate detailed and multiple-choice questions focusing on audio events, objects, acoustic scenes, signal properties, semantics, and listener emotions. To evaluate Mellow’s reasoning ability, we benchmark it on a diverse set of tasks, assessing on both in-distribution and out-of-distribution data, including audio understanding, deductive reasoning, and comparative reasoning. Finally, we conduct extensive ablation studies to explore the impact of projection layer choices, synthetic data generation methods, and language model pretraining on reasoning performance. Our training dataset, findings, and baseline pave the way for developing small ALMs capable of reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Mellow, a small audio-language model (ALM) designed for reasoning tasks. Mellow achieves competitive or superior performance compared to many larger models, including some with over 50× the parameters and training data. The authors also release ReasonAQA, a new dataset combining synthetic and existing data to support audio-grounded reasoning. Extensive evaluations are conducted across multiple tasks—deductive reasoning, comparative reasoning, audio captioning, and multimodal understanding—along with thorough ablation studies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Timely and Relevant Contribution: Addresses an underexplored but important niche: reasoning with small ALMs, which is valuable for edge devices.\n\n- Strong Empirical Results: Mellow outperforms all other small ALMs and even some large ALMs (e.g., Qwen-Audio) on reasoning tasks like MMAU and Audio Entailment.\n\n- Novel Dataset (ReasonAQA): The construction of ReasonAQA demonstrates thoughtful design and use of LLMs for QA generation grounded in audio.\n\n- Comprehensive Evaluation: Covers a wide spectrum of reasoning capabilities (deductive, comparative, binary QA, captioning), including OOD evaluation.\n\n- Thorough Ablation Studies: Provides valuable insights on design choices (projection layers, LM architecture, audio encoder) and training strategies (fine-tuning vs. LoRA, etc.).\n\n- Resource-Efficient: Only 167M parameters and trained on ~152 hours of audio, making it suitable for practical deployment.\n\nWeakness:\n\n- Speech Reasoning Gap: Mellow performs poorly on speech-based tasks due to lack of speech data in training. This limits generalizability.\n\n- Synthetic Data Dominance: 70% of ReasonAQA is synthetic; concerns may arise around the diversity and generalizability of such data.\n\n- Comparison with SOTA Models: While results are impressive, the evaluation would benefit from comparison with recent speech-capable models on more speech-focused tasks.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Mellow, a small (167M parameter) Audio-Language Model (ALM) designed specifically for reasoning tasks. Unlike prior work that relies on scaling model and data size, Mellow focuses on improving reasoning abilities through architectural and training innovations. It is trained on a new dataset, ReasonAQA, which blends real and synthetically generated audio-grounded question-answer pairs. Despite its small size and limited data (~152 hours of audio), Mellow achieves state-of-the-art performance among small ALMs and outperforms several much larger models on multiple benchmarks, including multimodal reasoning (MMAU), deductive reasoning (Audio Entailment), comparative reasoning (Audio Difference), and audio captioning. The paper also conducts extensive ablation studies, revealing the importance of language model pretraining, fine-tuning strategies, audio encoder quality, and targeted synthetic data in boosting reasoning performance without relying on large-scale data.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths:\n    * The structure of this paper is fairly clear, and the experiments are thorough. Despite the relatively small model size, it still achieves good performance.\n    * The paper conducts ablation studies from multiple perspectives, providing valuable insights for researchers who are new to the field.\n\n* Weaknesses:\n    * My main concern is that the paper lacks sufficient novelty. The methods used have already become relatively common practices in the field. Furthermore, the ablation studies do not introduce particularly novel analytical perspectives, and many of the conclusions drawn are largely aligned with what is already widely recognized by researchers. This paper feels more like a detailed experimental report rather than a research paper.\n    * The paper attempts to justify the development of SALM from the perspective of applications for edge devices. However, it lacks analysis specific to resource-constrained scenarios. As a result, it fails to demonstrate the advantages of the proposed method in such contexts—for example, through comparisons with alternative solutions such as distilling large models into smaller ones or directly quantizing large models.\n    * The fairness of the experimental comparisons in the paper is questionable. Although Mellow shows strong performance on the benchmarks tested, it seems more like an indication that Mellow may be overfitted to these specific tasks rather than a demonstration of general effectiveness. Perhaps showcasing the model’s performance on entirely unseen tasks would better demonstrate the generalization ability of the proposed approach.\n\nQUESTIONS:\n* The comparison in Table 5 may not be entirely fair. If I understand correctly, the linear probe is clearly not a strong baseline, and the modeling capacity of a language model is naturally expected to perform better.\n* The paper emphasizes the high quality of ReasonAQA and compares it with OpenAQA by analyzing its shortcomings. However, it does not clearly explain how the data generation process for ReasonAQA ensures such high quality. Is controlling the prompts alone truly sufficient to produce significantly better data?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Mellow, a small audio language model for audio reasoning. The paper also introduces ReasonAQA dataset for training Mellow. ReasonAQA is built based on AudioCaps and Clotho data. Experiments show that training on ReasonAQA enables Mellow to achieve relatively good performance on the MMAU benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The Mellow model with only 167M parameters can achieve promising performance on the MMAU benchmark.\n\nWeaknesses:\n- About the novelty. The model structure seems to merely replace a large language model with a small one, which is somehow trivial, and the model's improvement seems to stem from the ReasonAQA data. It is not surprising to obtain better results through training with higher-quality data. Therefore, I believe the novelty of this paper is limited.\n- Does Mellow understand speech? I believe it cannot understand speech content at all, as the annotations in AudioCaps and Clotho don't include any conversation-related information. If that's the case, the lack of speech understanding ability in Mellow also limits its novelty.\n- Regarding the rationality of the results. I am concerned that many baseline models are underestimated. Most models in Table 4 have scores below 60, but Mellow exceeds 91, which is rather questionable. Could you provide the results of some state-of-the-art (SOTA) models for reference, such as Qwen-Omni or Gemini? In addition, as for the caption performance of Table 4.4, the caption results of some models are lower than what I expected. For example, the SPICE score of SALMONN-13B is less than 10, which is weird.  When testing audio LLMs, did the authors adjust different prompts?  Some audio LLMs are quite sensitive to prompts, and inappropriate prompts could lead to lower results. The authors should adjust the test prompts to avoid underestimating the performance of the tested models.\n\nQUESTIONS:\n- Did the authors try to use ReasonAQA data for fine-tuning audio LLMs such as SALMONN or QwenAudio? Will this approach yield more remarkable results?\n- How does Mellow perform on other audio benchmarks besides MMAU?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper advances the understanding of building efficient ALMs with regard to trade-offs between model size, data constraints, and performance. While authors do not employ novel, groundbreaking techniques, I think this represents solid technical work with a good and important direction for the LLM-reasoning community. Although given the speech performance and limited audio knowledge, the claims regarding immediate applicability require further consideration.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nAmidst the trend towards scaling LLMs to billions of parameters, I think this paper is well-motivated towards closing the gap by showing that their 135M model is able to achieve competitive reasoning over its bulkier counterparts.\n\nOverall the paper is well-written and clear to understand. The evaluation methods are well-explained and cover multiple reasoning types {deductive, comparative, ood}\n\nImportant ablations are provided across tuning methods (lora, finetuning, prefix), data scaling effects with wav2caps, and prompting styles.\n\nWeakenss:\nWhile the authors have intentionally constrained to using only Audiocaps+clotho to isolate reasoning improvements from those of scaling parameters, I think this severely limits the model's knowledge of audio concepts and of course generalization. \nWould a model with double Mellow's parameters but trained on the same constrained data still underperform due to insufficient audio concept coverage?\nWould the simple projection and HTSAT encoder scale effectively to larger parameter counts? Is this reasoning ability transferable or clotho-specific pattern matching? \nI think demonstrating reasoning improvements in isolation, may not provide sufficient evidence for the broad claim that small models can maintain competitive performance in practical applications where both larger models AND larger datasets are available\n\nCould you please provide Fig. 3 to compute the drop against only music and sound subsets of MMAU? I think this should help validate “(1) Speech-based task constitute nearly 30% of MMAU … limited speech training ..” In my opinion, this is not just a limitation but a fundamental weakness for an audio understanding model.\n\nWhile 135M parameters are indeed smaller than cloud-scale models, it still requires significant resource (>500 mb;  authors, please confirm this.). Most successful applications for on-device or even always-on are in 1-5M param range - considered truly on-device AI which occasionally trigger server-side computation for main LLMs. Given their param count, currently both Mellow and Qwen2 Audio sit in the same bucket - i.e both are for server side deployment.\nIf a device triggers a late-stage process, it already transmits the audio to the server. Given that the server hypothetically has no compute constraints, why would one opt for Mellow there instead of passing the audio to bulkier QewnAudio for potentially better results?\nI think some clearer articulation of the intended deployment scenarios would immensely help this paper.\n\nQUESTIONS:\nPlease see the weakness section above. All questions are embedded with the weakness I found.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}