Mellow: a small audio language model for reasoning
Authors: Soham Deshmukh
Satvik Dixit
Rita Singh
Bhiksha Raj
Venue: NeurIPS 2025 poster ()
Keywords: audio-language
multimodal learning
audio reasoning
audio processing
Primary area: applications
TL;DR: a small audio-language model for audio reasoning that achieves SoTA performance with 50 times fewer parameters and 60 times fewer audio hours.

Abstract: Multimodal Audio-Language Models (ALMs) can understand and reason over both audio and text. Typically, reasoning performance correlates with model size, with the best results achieved by models exceeding 8 billion parameters. However, no prior work has explored enabling small audio-language models to perform reasoning tasks, despite the potential applications for edge devices. To address this gap, we introduce Mellow, a small Audio-Language Model specifically designed for reasoning. Mellow achieves state-of-the-art performance among existing small audio-language models and surpasses several larger models in reasoning capabilities. For instance, Mellow scores 52.11 on MMAU, comparable to SoTA Qwen2 Audio (which scores 52.5) while using 50 times fewer parameters and being trained on 60 times less data (audio hrs). To train Mellow, we introduce ReasonAQA, a dataset designed to enhance audio-grounded reasoning in models. It consists of a mixture of existing datasets (30\% of the data) and synthetically generated data (70\%). The synthetic dataset is derived from audio captioning datasets, where Large Language Models (LLMs) generate detailed and multiple-choice questions focusing on audio events, objects, acoustic scenes, signal properties, semantics, and listener emotions. To evaluate Mellow’s reasoning ability, we benchmark it on a diverse set of tasks, assessing on both in-distribution and out-of-distribution data, including audio understanding, deductive reasoning, and comparative reasoning. Finally, we conduct extensive ablation studies to explore the impact of projection layer choices, synthetic data generation methods, and language model pretraining on reasoning performance. Our training dataset, findings, and baseline pave the way for developing small ALMs capable of reasoning.

Mellow: a small audio language model for reasoning
Soham Deshmukh
Satvik Dixit Carnegie Mellon University
Rita Singh
Bhiksha Raj
{sdeshmuk, satvikd, rsingh, bhikshar)Sandrer. cmu cmu.edu edu
Abstract
Multimodal Audio-Language Models (ALMs) can understand and reason over both audio and text. Typically, reasoning performance correlates with model size, with the best results achieved by models exceeding billion parameters. However, prior work has explored enabling small audio-language models to perform reasoning tasks, despite the potential applications for edge devices. To address this gap, introduce Mellow, small Audio-Languael Model specifically designed for reasoning. Mellow achieves state-of-the-art performance among existing small audio-language models and surpasses several larger models in reasoning capabili- ties. For instance, Mellow scores 52. on MMAU, comparable to SoTA Qwen2 Audio (which scores 52.5) while using 50 times fewer parameters and being trained on 60 times less data (audio hrs). To train Mellow, we introduce ReasonAQA, a dataset designed to enhance audio-grounded reasoning in models. It consists of a mixture of existing datasets (30% of the data) and synthetically generated data (70%). The synthetic dataset is derived from audio captioning datasets, where Large Language Models (LLMs) generate detailed and multiplc-choice questions semantics, focusing on audio events, objects, acoustic scenes, signal properties, and listener emotions. To evaluate Mellow's ' reasoning ability, we benchmark it on a diverse set of tasks, assessing on both in-distribution and out-of-distribution data, including audio understanding, deductive reasoning, and comparative reasoning. Finally, we conduct extensive ablation studies to explore the impact projection layer choices, synthetic data generation methods, and language model pretraining on reasoning performance. Our training dataset, findings, and baseline pave the way for developing small ALMs capable of reasoning.
Introduction
In recent years, the field of computational audition 7 and machine listening has transitioned from task- specific approaches [13] unified models [26 capable of performing multiple tasks. These unified systems, known as Audio Foundation Models 69 have demonstrated impressive performance across a variety of tasks, often surpassing task-specific and dataset-specific models. Most Audio Foundation Models leverage language mode of supervision during pretraining, utilizing relatively
simple objectives such as contrastive learning 255 27 73 or next-token prediction
14 [6 [3 The
simplicity of these objectives makes them highly scalable, allowing for the scaling of both data and model parameters with increased computational resources.
Over the years, advancements in architectural design, along with the scaling of both data and model parameters, have sigmificantly enhanced performance on existing benchmarks [36] 66 33 I0)
Early contrastive audio-language models, such as CLAP [25], were trained on relatively small datasets, such as 128k audio-text pairs. contrast, recent next-token prediction models leverage datasets containing millions of pairs, incorporating multiple encoders and projection layers, with some models exceeding 10 million pairs 34). Beyond benchmark improvements, these larger models have begun to exhibit emergent abilities [16] skills not explicitly trained for and not easily inferred from their individual components. Notable examples include the ability of Audio- Aunio-Language Models (ALMs) to serve ceneral-purposs. audio quality estimators []6,, enhanced reasoning over
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ALM performance MMAU
Mellow capabilities and examples
50-
Mellow
Whtt citeroe natures
Pooi seourd peear coseente hasural coming
moctect
20
thoae wff
GARA LiiaMs
ranegann noar paavet coltent
U. sow de sira to ather tos
30-
MMAU avg.
20-
Audio Flgringg (C)
Mustling RzUSen
U. ASITU
tey te sourd menes nemne with
de inde mnyine Te bovigg corrgroee
murdd with cortegiii
BLD "unling
10-
Peng]
Owen2
conggemill calenos trons hes repuess the
pellinn mnse the codi oigee dudie
oi ylu of
liee of
10
12
Parameter count (billions)
Figure 1: The left plot shows the performance of different ALMs on MMAU their parameter size. We plot models whose parameter counts are known, (l) indicates Instruction-tuned, (C) indicates Chat. The right figure shows Mellow's different capabilities and examples, the full examples are available in Figure
audio-text pairs (4],, and logical reasoning, particularly deductive reasoning [8] Among these capabilities, the ability to perceive audio, reason over audio and text, and generate informed responses using world knowledge is particularly promising, with broad practical applications. This reasoning ability not only allows ALMs tackle novel tasks but also provides a unified means to evaluate core skill-one whose improvement leads to performance gains across multiple downstream applications.
Understanding and reasoning are related but distinct processes. Literature 42 defines understanding as grasping meaning and coherence, while reasoning involves drawing inferences. Similarly, [28] highlights that reasoning relies on distinct processing modes, reinforcing its distinction from general understanding. In audio models, reasoning can improve through two approaches. First, expanding data (audio) coverage (e.g., knowledge of audio objects and scenes). which provides broader foundation upon which the model can apply its existing reasoning capabilities. Second, enhancing the model's 's inherent reasoning ability through architectural, learning or post-training techniques.
18 has primarily followed the first approach, where increasing audio coverage indirectly enhances both understanding and reasoning performance. While effective, this approach does not improve the inherent reasoning ability constrained by architectural, learning, post-training techniques, or reasoning chains found in language not audio data. By fixing training audio, one can conduct controlled evaluations of ALMs, which can later scale via data expansion. We explore the second approach, aiming to enhance reasoning independently of data expansion.
Prior work [1]] 36
33
66
In this paper, we introduce Mellow, a small Audio Language Model (ALM) that can reason over text and audio. The paper's 's main contributions are:
Mellow: small Audio-Language Model for reasoning. We introduce Mellow, a small- scale ALM optimized for reasoning. Mellow surpasses all models within its parameter category and, in certain benchmarks, outperforms models with 50 times more parameters, while trained on only 155 hrs of audio. We provide model checkpoints, training data, and evaluation code to facilitate further research.
ReasonAQA: a training dataset for audio models. We create ReasonAQA, a dataset designed to enhance the logical reasoning capabilities of ALMs. The dataset comprises mixture of generated data (70%) ) and existing deductive and comparative reasoning datasets (30%) from the literature. To generate synthetic data, leverage audio descriptions from AudioCaps and Clotho, using L.Ms to create QA pairs focused on audio events, objects, acoustic scenes, acoustic properties, signal characteristics, semantics, and listener emotions. In total, ReasonAQA uses 56k (~1552 hrs) audio files and consists of 1M AQA instances, with predefined train, validation, and test splits.
Ablations and findings. We start with a minimal audio-language model from the literature 14,, which has previously been shown to perform at random on audio reasoning tasks Then we conduct extensive ablation studies to analyze the impact of architectural choices, synthetic data generation, and training strategies on reasoning performance. We highlight the key findings in Section 5
18 63
2 ReasonAQA
We construct the ReasonAQA dataset to train Audio- Audio-Language Language Models (ALMs) for reasoning over audio and text. This dataset derived from a fixed set of audio files, allowing the study of improvements driven by architectural choices, synthetic data generation, and learning methodologies,
2

---

rather than those resulting from scale-related factors such as larger datasets, broader domains, or an increased number of learned concepts. The dataset construction and composition are detailed in Section 2. and Section 2.2 respectively.
2.1 Data construction
Audio sources. We use two audio datasets with human-labeled descriptions-Audiice.. 143 and Clotho [23]. These datasets are sourced from AudioSet and Freesound, respectively, and cover wide range of audio events, acoustic scenes, and audio concepts. By restricting our audio sources for training and evaluation, we ensure consistency in audio concepts and isolate performance improvements resulting from model design and learning methods, rather than those driven by data scaling. AudioCaps annotations are generated using Amazon Mechanical Turk (MTurk), where annotators have access both visual and audio content. This multimodal access allows for more detailed descriptions compared to those generated from audio-only annotations. In contrast, Clotho restricts annotators to audio-only listening, without access to visual textual cues. Unlike AudioCaps, where audio clips are limited to 10 seconds, Clotho contains recordings ranging from 15 to 30 seconds, with each audio file accompanied by five descriptions, each containing
20 words. These datasets have been widely used for building and benchmarking audio captioning models [53] 54| [5 56 17
and text-to-audio retrieval models 45 Additionally, they have been extended construct datasets for audio question-answering [48], deductive reasoning [18). and comparative reasoning [19] Given their diversity and extensive utilization across multiple tasks, we select these two datasets as the foundation for generating data in ReasonAQA.
Audio reasoning tasks. In the literature, audio reasoning tasks utilizing AudioCaps and Clotho primarily include audio entailment, a classification task, and audio difference explanation, a de- scription task, in addition to the traditional audio captioning task. To incorporate these tasks into ReasonAQA, we convert both audio captioning and audio entailment into guestion-answering (QA) formats. Specifically, audio captioning is transformed into descriptive question-answeriig task, while audio entailment is converted into multiple-choice question (MCQ) format. The audio difference explanation task is already QA-based, so we integrate it directly into ReasonAQA.
ReasonAQA datasel (1.24M)
Type
Audio Source
Train
Val
Test
Dataset
Binary MCQ Desc. AudioCaps Clotho # Audio # QA Pairs Per % # Audio # QA Pairs Per % Audio QA Pairs Per %
Clotho
23
3839 48660
19195 48660
2.0% 5.0%
1045 494
5225 494
4.7% 0.4%
1045
5225 4785
3.2%
AudioCaps
13
957
3.0%
Sum
67855
7.0%
5719
5.1%
10010
6.2%
ClothoAQA
48
14088
1.5%
4128
3.7%
5676
3.5%
CLE M
11511
1.2%
3132
2.8%
3135
1.9%
ACE
14355
8.9%
CLD
19
57585
5.9 %
15675
14.0%
15675
9.7%
145980
7368
6.6%
14040
8.7%
Sum
229166
23.7%
30303
27.0%
52881
32,7%
Clotho-MCQ
/
95551
9.9%
26054
23.2%
26031
16.
Clotho-Detail
94838
9.8%
25785
23.0%
25851
AudioCaps-MCC
4 /
241531
24.9%
12236
10.9%
23672
14.6%
AudioCape- Detail
239132
24. 7%
12110
10.8%
23355
14.4%
Sum
671052
69.3%
76185
37.9%
98909
61.1%
Total
52499
968071
100%
112207
100%
161800
100%
Table 1: The composition of the ReasonAQA dataset. The training set is restricted to AudioCaps and Clotho audio files and the testing is performed on 6 tasks Audio Entailment, Audio Difference, ClothoAQA, Clotho MCQ. Clotho Detail, AudioCaps MCQ and AudioCaps Detail.
Synthetic data. We use audio descriptions from AudioCaps and Clotho to prompt an LLM for generating question-anse answer (QA) pairs. This method of LLM prompting has previously been used to create various datasets and benchmarks [37, 36 synthetic data with two key distinctions. First, we produce a mix of detailed and multiple-choice (MCQ) questions, unlike OpenAQA, which contains only descriptive questions. Second, the questions are explicitly grounded in audio, setting them apart from existing datasets (6), which shown in the failure modes of existing datasets (Table 12
I8 19. Following this approach, we generate
We refer the reader to Appendix C for details on
3

---

the data generation methodology, prompting setup, data analysis, and comparative evaluation with existing datasets, including OpenAQA.
2.2 Data composition
Using AudioCaps and Clotho audio files, along with existing audio tasks and synthetic data, we construct the train, validation, and test subsets of ReasonAQA. The dataset composition for each split is shown in Table [ audio1, audio2, question, answer, where audio, is empty when the task does not require two audio inputs.
Each example in the dataset is structured as a quadruple:
3 Mellow
In this section, we introduce Mellow, a small Audio Language Model that processes two audios inputs and text prompts to generate text-based outputs. The architecture of Mellow is illustrated in Fig.
Prefix
+ Audio Encoder
Text
Mapper
Projection
Map
output
Audio
Linear
Projection
+ Audio Encoder
Small
c 1,
KeeL
Map
Language
Downsampple
Audio 2
Model (SLM)
Linear
Linear
Layer
#1.6
x(1 d]
Text prompt
x Text Embedder
Imput
Output (3)
Figure 2: Mellow takes two audio recordings and a text prompt as input and generates a text output. The two audio inputs encoded by an audio encoder and projected into the language model space using mapper (map). Simultaneously, the text prompt is embedded by the text embedder. The audio projection 1, audio projection text embedding are concatenated to form the prefix. During concatenation, a separator token (s) inserted between the three components (as shown Fig. 2}. The prefix is then used to prompt the small Language Model, which generates natural language response.
Audio Encoder. The raw audio with its high sampling rate is unsuitable for the Language Model to operate on. Therefore, we encode the raw audio using an audio encoder, which produces a fixed- length latent representation. We use HTSAT [9. a Hierarchical Token Semantic Transformer, as the default audio encoder. HTSAT, based on the Swin Transformer, has demonstrated state-of-the-art performance on AudioSet 31 The audio encoder output consists of latent prediction, which is then expanded by token-semantic CNN to generate an event presence map with temporal information. The audio encoder outputs passed to the mapper are: (1) the latent prediction (1 x l), which acts CLS token, and (2) the event presence map (time frames x d), which retains temporal information across the audio. The encoder processes 10-second audio clips sampled 32 kHz.
Mapper. The audio encoder output projected into the language model space using a mapper. The mapper consists of linear layer, projection layer, and downsampler. The linear layer projects the latent prediction (1 I), which acts as CLS token into the same space as the event presence map (time frames d), which retains temporal information across the audio. After the linear layer projection, the two outputs are concatenated and passed to projection layer. Existing literature explores various projection mechanisms (26] 27 14 33), ranging from simple linear transformations to full monnformmr-basee mappers. We use a non-lineat projection consisting of two linear layers whose outputs are combined before passing through a layer normalization. Finally, the output of the projection layer is downsampled before latent prompting of Language Model. The details of the mapper and projection layers, along with ablation studies, provided in Appendix E and Section 5
Language Model. We specifically choose small Language Models (SLMs) over large or traditional I Language Models, aiming to develop minimal yet effective recipe for small Audio-Lang Languag Models that can reason. For this purpose, we select smolL.2 (2), one of the hest-performing SLMS
on various benchmarks [12 38, 62 74 65 58). We conduct ablation studies to evaluate the

---

impact of language model pretraining and architecture on audio reasoning performance. These results are presented in Section and Table
Training. Mellow is trained on the next-token prediction task, where the next-token is predicted based on past-tokens and the two input audios (A1, A2). For each token, cross-entropy (P) is computed between predicted probability and ground-truth, which is then summed over tokens. The objective is:
P(x: A1,
(I)
using cross-entropy for all 1 < it is conditioned the text sequence #1. along with the input audios A1 and Ag.
Inference. The literature uses different inference sampling techniques ranging from greedy decoding, to top-k and top-p [36 34 [9]. beam search [14), etc. Each technique offers advantages such task-specific benefits, increased diversity, and improved performance. For all evaluations in this work, we employ top-p sampling where is set to 0.8 and a temperature of
4 Results
In this section, we evaluate Mellow across various audio reasoning tasks. Specifically, we compare Mellow against other Audio-Language Models on the following benchmarks: MMAÜ (Section 4.1 Audio Entailment (Section 4.2), Audio Difference (Section 4.3), Audio Captioning and Binary AQA (Section 4.4).
The multiple tasks benchmark different types of reasoning ability of Mellow.
4.1 Understanding and reasoning
In this section, we evaluate Mellow on its multimodal audio understanding and reasoning capabilities. We use the MMAU benchmark [63]. which consists of 10k annotated Audio Question-Answerty (AQA) pairs spanning the speech, sound, and music domains. This benchmark assesses model's ability across 27 distinct skills covering various reasoning tasks. Notably, Mellow is trained exclu- sively on AudioCaps and Clotho audio files, whereas MMAU sources its audio from 10 different datasets, with no overlap with Mellow's training data. This makes MMAU an ideal benchmark for evaluating Mellow's 's reasoning performance in an Out-of- -iit---Distribttion (OOD) setting. Additionally, it allows to assess how effectively our training data (ReasonAQA) enhances reasoning ability in ALMs while minimizing improvements driven by the scaling of audio concepts.
Sound
Music
Speech
Avg
Models
Size
(So, Mu, Sp)
Test-mini
Test-mini
Test-mini
Test
Test-mini
Random Guess
26.72
25.7
24.55
26.53
26.72
25.50
26.00
25 92
l'requent Choice
20.35
29.12
30.33
Human (test- mini)
86.31
78.22
82.17
82.23
Large Audio Language Models (LALMs)
LTU
7B
22.52
25.86
09.69
12.83
17.71
16.37
16.89
18.51
LTU
7B
/
V
23.35
24.96
9.10
10.45
20.60
21.30
17.68
18.90
MusiLingo
7B
23.12
27.76
03.96
06.00
05.8%
06.42
10:98
13.39
MuLLaMa
7B
X
40.84
44.80
32.63
30.63
22.22
16.56
31,90
30.66
M2UGen
7B
03.60
03.69
32.93
30.40
06.36
04.53
14.28
12.87
GAMA
7B
Y
V
41.44
45.40
32.33
30.83
18.91
19.21
30.90
31.81
GAMA-IT
7B
V
43.24
43.23
28.44
28.00
18.91
15.84
30.20
29.02
Qwen-Andio-CN
X
44.00
27.95
Qwen2 Audio
8.4B
V
07
08.20
05.14
06.16
03.10
04.24
05.24
06.20
Qwen2-Audio- Instructt
45.90
SALAMONN
138
41.00
40.30
34,80
33.76
25.50
24.24
33.70
32.77
Gemini Pro vss
56.75
54.46
49.40
48.56
58.55
55,90
54,90
52.97
Large Langnage Models (LLMss
GPTHo + weak cap.
39.33
35.30
39.52
41.9
$8.25
68.27
45.70
4% 48.65
GPT4o + strong cap.
49.70
Llama-3-Ins. + weak cap.
8B
34.23
33.73
38.02
42,36
54.05
61.54
42.10
45.87
Llama-3-Ios, strong cap.
50.75
49.10
50.29
55.25
62.70
$2.10
53.57
Small Language
(SALMs)
Pengi
323M
06.10
08 08:00
02.90
03.05
01.20
01.50
03.40
04.18
Audio Flamingo Chat
2.28
23.42
28.26
15.26
18.20
11.41
10.16
16.69
18.87
Mellow
167M
61.26
64.90
$4.19
52.67
29.73
38.77
48.40
52.11
Table Results on MMAU benchmark. The third column indicates the training data used train these models containing sound, music, and speech. The best berfoperfo models in cach category are highlighted bold, and the second- -bsst scores are underlined,
5

---

Results. The results are presented in Table We compare Mellow against three types of models: Large Audio Language Models (LALMs), Large Language Models (LLM) and Small Audio Language Models (SALMs). For LALMS and SALMs, the models take audio and text questions as input and produce text responses as output. In contrast, L.MMs do not support audio input. To accommodate this, audio captions using a LALM [ are fed to the L.M. The strong and weak captions refer to different captioning models and prompting setups; for details, we refer readers to the MMAU paper [63]. Analyzing the results, we see the following trends. First, Mellow achieves state-of-the-art (SoTA) performance in reasoning over sound and music, outperforming all existing SALMs, LALMs, and LLMs. Second, Mellow is trained on only 51k unique audio files and has 167M parameters. In contrast, LALMs utilize 100x more audio data and 100x more parameters to achieve similar reasoning capabilities. This makes Mellow a highly competitive option for on-device audio understanding and reasoning. Third, Mellow performs poorly on speech-based tasks since it is not trained on any speech content (e.g., ASR-based datasets). However, it still learns attributes such as gender and music emotion, achieving a score of 29, which is the best among SALMs and above random performance.
LM reliance. Historically, Audio Question-Answering (AQA) models have exhibited a heavy reliance on the Language Model (LM), often to the extent of overlooking audio input while still achieving strong performance [48] 63). To investigate this issue, conduct an experiment similar to those in ClothoAQA [48] and MMAU [63], where replace the audio input with Gaussian noise and evaluate the model's performance. The results of this experiment are presented in Fig. 3 We observe that Mellow's performance drops by approximately 13% when Gaussian noise used instead of actual audio. Compared to Mellow, Gemini 1.5 Pro and Qwen2 Instruct exhibit stronger grounding in audio, which can be attributed to two key factors: (1) Speech-based tasks constitute nearly 30% of MMAU, and Mellow produces near-random responses on these tasks due to limited speech training, rarely conditioning its outputs on the audio input. (2) Models like Gemini Pro are natively multimodal, meaning they share tokenizers for both audio and text, whereas Mellow an LM conditioned on audio rather than a fully integrated multimodal model. Overall, there is a need to reduce the perception gap in Mellow [88 32), as improving its ability to incorporate audio cues effectively can significantly enhance its audio reasoning capabilities.
20-
15-
5
Mullems SALNOWN 6MVA
Hellw Gamin Pr Owar2
Figure 3: Absolute difference in accuracy between performance with real audio input and performance with Gaussian noise.
Performance distribution. We conduct an error analy- sis of Mellow on MMAU, specifically examining its per- formance across question difficulty levels and task cate- gories. MMAU categorizes question difficulty into easy, medium, and hard, while tasks are divided into 27 distinct skills. Table 3 presents Mellow's performance across dif- ferent difficulty levels. Notably, Mellow performs better on
Model
Easy
Medium
Hard
SALMONN
20.31
39.33
30.63
GAMA
31.36
35.70
22.85
Qwen2
50.59
55.63
46.99
Gemini Pro v.5
57.04
51.49
52.07
Mellow
50.68
60.10
32.89
questions compared to easy and hard ones, suggesting that it has stronger reasoning capabilities but weaker general understanding. Across different task categories, Mellow performs best on: acoustic source inference, ambient sound interpretation, and music texture interpretation. However, it struggles with phoneme stress pattern analysis, emotion flip detection, and lyrical reasoning. These results indicate that Mellow performs poorly on speech-related tasks, while its performance on sound and music tasks is relatively stronger.
Table 3: Model performance across difficulty levels on MMAU
4.2 Deductive reasoning
In this section, we evaluate Mellow's deductive reasoning ability across both audio and text using the Audio Entailment benchmark [18]. In this benchmark, each example consists of audio (AA) and text hypothesis (?(). The task is to determine whether the hypothesis is true given the audio input. The answer falls into one of three categories: Entailment (7L is true given A), Neutral (H is plausible but not necessarily true given A), or Contradiction (H is false given A). Beyond assessing deductive reasoning, this task also helps identify audio hallucinations, which typically appear in two forms. The first is inferred cues, where the model introduces audio concepts that are not present the input. The
second is contextual assumptions, where the model interprets sounds based on text likelihood rather

---

than actual audio evidence. This evaluation provides insight into Mellow's ability to reason logically over multimodal inputs while also highlighting potential failure modes related audio hallucination.
ACE
LLM
Models
(param)
ACC
P
R
F1
EACC
NACC
CACC
ACC
R
F1
EACC
NACC
CACC
CLAP 22
110M
45.90
54.99
45.90
46.56
60.00)
40.29
37.42
44.34
44.35
43.34
43.32
43.32
56.41
45.08
LCLAP
51.13
51.13
51.61
50.14
56.93
28.67
CLAP 23
124M
51.64
51.55
51.63
51.59
41.53
40.38
73.01
48.60
46,78
48.60
46.56
48.80
20.02
76.99
Pengl-nocne
124M
27.81
18.43
27.81
22.16
49.67
0.00
33.78
26.29
16.99
26.29
20.45
53.12
0.00
25.75
Pengivenc
124M
37.26
24.65
37.26
28.88
75.41
0,00
36.36
38.67
25.58
38.67
30.39
73.35
0,00
42.65
LTU-AS
7B
36.81
37.37
36.81
34,20
62.78
31.87
15.79
36.33
37.72
36.33
33.34
67.02
24.35
17.62
Qwen-A
7B
36.20
40.12
36.20
31.17
76.75
13.88
17.99
35.63
35.62
35.63
32.19
66.69
13.23
26.96
Awen-AC
7B
54.42
56.04
54.42
49.75
90.24
15.69
57.32
52.16
56.69
52.16
49.18
93.00
28.21
35.28
GAMA
48.26
61.51
48.26
45.34
81,44
41.24
22.11
52.48
65.31
52.48
49.13
78.27
58.85
20.31
GAMA-IT
7B
39.74
56.04
39.74
34.35
79.23
29.47
10.53
41.67
56,72
41.67
38.28
78.52
26.96
19.54
SALMONN
13B
52.22
50.54
52.22
45.15
67.75
7.08
81.82
56.22
55.51
56.22
48.26
71.14
6.98
90.55
Mellow
135M
91.16
91.35
91.16
91.10
90.53
85.26
97.70
89.66
90.71
89.66
89.34
95.82
73.48
99.67
Table 4: Deductive reasoning ability of different ALMs. The evaluation is performed on Audio Entailment task
Results. The results on the CLE and ACE datasets for audio entailment are presented in Table The first three entries correspond to contrastive Audio-Language Models (ALMs), which are naturally suited for classification tasks, while the remaining entries represent next-token prediction ALMs. Mellow outperforms all existing ALMs, regardless of whether they small or large language models and whether they are contrastive or next-token prediction models. Mellow excels at identifying hypotheses that definitely true or definitely false given the audio but struggles with detecting plausible scenarios. For instance, on the ACE dataset, Mellow correctly classifies plausible (NACC) hypotheses only 75% of the time, whereas its accuracy for entailment and contradiction cases is in the high 90s. Despite this limitation, Mellow still outperforms the similarly sized next-token prediction ALM, Pengi 14) which fails detect plausible scenarios effectively.
4
Linear-probe. Zero-shot evaluation is not a fair compar- ison, as Mellow has been explicitly trained for deductive reasoning. To provide a more balanced assessment, we compare Mellow against linear-probs (supervised) per- formance. In this setup, the audio encoder extracts the audio embeddings, while the text encoder extracts the text embeddings from the hypothesis. These embeddings are then fed into linear classifier, which is trained to predict one of three entailment categories: entailment, neutral, or contradiction. The classifier is trained on the training split and evaluated on the test split of the CLE dataset. In Table [5] the syymbo indicates the "caption- rerorerreassning" method. which incorporates an additional audio captioning step before reasoning. This approach has been shown to improve performance [18, (3). Overall, even in the supervised setup, Mellow outperforms all other ALMs, demonstrating its strong deductive reasoning capabilities.
Models
ACC
P
R
F!
CLAP
71.10
71.30
71.10
71.18
LCLAP
74.35
74.70
74.35
74.45
CLAP 23
83.29
83.61
83.29
83.36
Pengi-ene
76.27
76.74
76.27
76.42
CLAP 23
86.40
86.71
86.40
86.47
Mellow
91.16
91.35
91.16
91.10
Table 5: Supervised performance on CLE dataset of Audio Entailment task
4.3 Comparative reasoning
In this section, we evaluate Mellow's 's comparative reasoning ability (reasoning by analogy) over two audio inputs and text. We use the Audio Difference benchmark [19], where each example consists of two audio recordings and text question, with the goal of identifying the differences between the two audios. The text question specifies the level of detail required in the answer and is categorized into three tiers: Tier (concise), Tier 2 (bricf), and Tier (detailed). Beyond assessing comparative reasoning, this task requires the model to integrate audio information with world knowledge to effectively distinguish between the two audios. It demands an understanding of signal propertics (c.g.. frequency, amplitude) and contextual cues (e.g.. genre, environment) to identify both differences and similarities, making it a strong benchmark for evaluating a model's ability to combine audio perception with world knowledge.
Results. Table presents the results on the Audio Difference (ACD and CLD) datasets. We compare Mellow with various ALMs on this benchmark. All models are trained on the ACD and CLD training sets and evaluated on their respective test sets. Among large Audio Language Models, we have QwenAC, where the labels (L) and (F) for QwenAC indicate LoRA and full finetuning the training
7

---

CLD-I
CLD-2
CLD-3
ACD-1
ACD-2
ACD-3
Models
LLM
(param)
BLEUU,
SPICE
BLEUU,
SPICE
BLEU,,
SPICE
BLEU,,
SPICE
BLEU,,
SPICE
BLEU,
SPICE
Baseline
125M
8.8
6.4
26.5
26.4
13.7
17.2
13.1
10.1
21.7
21.5
15.1
13.8
QwenAC (L)
7B
9.3
8.2
26.4
21.1
13.5
14.5
14.5
9.5
22.0
24.0
16.1
16.0
QwenAC (F)
7B
7.6
9.5
28.5
27.3
12.2
14.9
12.3
9.4
21.7
21.9
14.5
14.3
ADIFF
125M
15.3
11.9
24.5
23.2
17.1
16.7
14.8
12.7
23.4
22.2
16.9
17.1
Vieloow
135M
17.3
13.9
26.1
25.0
17.9
17.3
15.6
13.9
24.3
23.7
17.9
18.7
Table 6: Comparative reasoning ability of different ALMs. The evaluation is performed on the Audio Difference Explanation task on datasets CLD and ACD. The sampling setup is kept constant across models (Appendix [E.).
data, respectively. Among smaller ALMs, we have naive baseline built on a Pengi-like architecture
[1] and ADIFF [19], which improves upon Pengi and achieves state-of ootie--ffth-aann the-art results on this task. From these results, we see that Mellow consistently outperforms existing ALMs on Tier-I and Tier-3. On Tier-2, however, QwenAC outperforms Mellow in terms of BLEU, on both CLD and ACD. Linguistically, Tier-1 is the hardest to learn, followed by Tier-3, with Tier-2 being the easiest. In particular, Tier-2 contains roughly 15% of words related to linguistics and stop words rather than specific audio-contrasting information, whereas Tier-1 contains fewer words but primarily focuses on audio details. Consequently, Tier-2's higher scores, even subsequent experiments, can be attributed to its linguistic simplicity. As result, Mellow's 's smaller LM falls short of the larger QwenAC LM in producing more coherent language structures on Tier-2.
4.4 Audio captioning and binary AQA
In this section, evaluate Mellow on audio captioning and Audio Qhestion-Ansseriigg (AQA) tasks. Higher performance audio captioning indicates a smaller perception gap and better grounding Traditionally, ALMs have also been benchmarked on binary question- answering, which involves yes/no questions covering count-based and wridence-based reasoning. Therefore, assess Mellow's performance both Audio Captioning and the Clo
in audio for ALMs [8 32).
Results. The results are presented in Ta- ble 7 Mellow outperforms the existing small ALM, Pengi, across all three evaluation metrics. Specifically, Mellow achieves a SPICE score of 17.8 for audio captioning, significantly higher than Pengi's 12.7, and an accuracy of 71.4% on ClothoAQA, surpassing Pengi's 63.6%. These results suggest that Mellow is particularly ef- fective at generating semantically meaningful captions for audio events and performing bi- nary pnestion-answering tasks. Compared to larger models like Qwen-Audio and GAMA, Mellow achieves competitive performance on binary AQA while using just 2.4% of their parameter size. However, captioning, we Mellow falling short of LALMs, where large audio concept knowledge is necessary to produce better captions.
Audio Caption
AQA
LLM (param)
Models
AC (SPICE)
CL (SPICE)
CL
(ACC)
Large Audio Language Models (LALMs)
LTU
7B
11.7
SALMONN
13B
8.3
7.6
23.1
AudioGPT
GAMA
7B
18.5
13.5
71.6
QwenAC
TB
9.8
32.3
Small Audio Language Models (SALMs)
Pengi
124M
12.7
7.0
63.6
Mellow
9.4
Table 7: Captioning and AQA performance of ALMs.
5 Ablation findings
We perform ablation studies to identify which components most effectively enhance reasoning in audio-language models, with a particular focus on Small Audio-Language Models. In particular, we examine the effects of audio encoders (Section F.2). language model choice (Section [.3), projection layers (Section F.T), prefix-tuning versus fine-tuning (Section F.4). LoRA adaptation (Section F..)). synthetic data generation (Section F.6). and scaling data (Section F.7 The results are shown in Table
gland the detailed experimental setups and analyses are available in Appendix F The key observations from these ablation studies are:
Fine-tuning outperforms prefix-tuning. Prefix-tuning and LoRa adaptation underperform relative when larger ransformer-basee mappers used. In contrast, fine-tuning the
to full fine-tuning, language model with a small linear or two-layer non- non-linear mapper offers a better balance between computational cost and performance. For LoRA ablation and settings, please refer Appendix F.5)
Better LM pretraining improves ALM reasoning on open-ended tasks. Replacing GPT-2 with SmolLM2, while keeping the data and architecture unchanged, results in higher performance

---

Andio Caption
B-AQA
Audio Entailment Audio Difference
MMAU (lest-mini)
Models
Size
AC
CL
ClotboQA
CLE (ACC)
ACE
CLD:3
ACD-3
Sound Music Speech Avg.
(SPICE) (SPICE)
(ACC)
(ACC)
(SPICT) (SPICE) (ACC) (ACC) (ACC) (ACC)
The SLM is GPT2 and frozen and the projection loyer changed
Lincar
156M
5.98
53.23
30.79
30.15
8.39
16.80
31.25
30.10
26.05
Non- con-linear] 2
157M
4.79
6.10
55.18
32.12
29:56
5.56
8.62
17.42
33.23
27.03
25.89
Transformer
14
195M
10.26
7.89
62.35
43.89
42.10
11.88
12.50
28.65
35.23
26.68
The SLM is GPT2 and finermeed and the projection layer is changed
Lincar
156M
9.87
7.10
70.90
93.45
92.90
13.65
14.21
47.64
45.39
27.00
40.01
Non- Non-linear
2
157M
10.51
7.15
71.25
93.40
93.27
13.77
14.01
48.05
48.50
27.33
41.29
Transformer
14
195M
10.77
7.23
70.89
92.32
93.65
13.45
15.21
47.89
49.10
27.10
The SLM is frozm, projection layer is non linear and the STM is changed
GPT2 frozen
157M
4.79
6.10
55.18
32.12
29.56
5.56
8.62
17.42
33.23
27.03
25.89
SmolLM2 frozen
167M
5.26
8.58
45.70
33.40
31.66
11.43
13:62
35.14
30.54
20.12
28.60
The SLM finetuned. projection layer non lineur and the SLM is chunged
GPT2 finetune
157M
10.51
7.15
71.25
93.40
93.27
13.77
14.01
48.05
10.50
27.33
41.29
SmolLM2 finetune
167M
18:60
9.83
71.65
92.00
90.85
17:33
18.68
59.46
50.60 30.60
28 82
46.29
Diferent finetining methods.
The SLM Sm2lLM2, projection layer non-linear
Prefix- -taing
167M
5.26
8.58
45.70
33.40
31.66
11,43
13.62
35.14
30.54
20.12
28.60
LoRA (8, 16)
167M
18.53
9.25
64.64
79.33
84.15
14.23
14.98
45.95
42.51
28.83
39.10
LoRA (256. 512)
18IM
19:01
10.59
65.54
86.60 66
89.87
15.36
16.51
50.75
49.10
33.33
44.40
Finctuning
167M
18.60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28.82
46.29
The SLM SmolLM2 and finetuned. projection layer is now-linear and the andio encoder is changed
CNN14
219M
15.97
7.91
65.82
91.06
92.39
16.27
16.95
54.05
47.60
28.23
43.30
HTSAT
167M
18:60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28 28.82
46.29
The SLM SmolLM2 and finetuned, projection layer is non-linear, and training data
changed
Type 1
167M
18.60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28.
46.29
Type 2
167M
16.47
8.23
71.05
92.50
93.20
16.98
18.09
59.45
42.81
37.84
46.70
Type
167M
17.43
9.88
66.83
91.87
94.25
17.37
18.67
61 56
45.21
32.43
46.40
Type
167M
17.79
9.38
71.39
91.16
89.66
17.21
18:54
61.26
54.19
29.73
48.40
The SLM SmolLM2 and finetuned, projection laver is non-lineor, and Way added training
Type 1
167M
18.60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28.82
46.29
WavCCaps
167M
14.83
9.66
71.32
92.47
92.69
17.92
19.13
59.16
60.48
23.72
47.80
Mellow
167M 17.79
71.39
91.16
89.66
17.21
18.54
61.26 54.00
29.73
Table 8: Ablation study results. The experimental setup for each experiment described in Appendix F
reasoning tasks. The benefits of improved pretraining are most apparent on open-ended reasoning, where the model must generate long, coherent responses grounded in audio. For deductive reasoning, performance is comparable and only shows improvement when larger LM 1>BB params) is used.
Better unimodal audio representations improve LM reasoning via coverage. Pretraining audio models on large audio corpora- through supervised or SSL approaches- is essential for learning fundamental audio concepts that an I.LM can reason over. Enhanced audio representations, as shown by linear-prohe evaluations or downstream task performance, directly lead to better reasoning by improving coverage. For example, HTSAT achieves higher mAP on AudioSet than CNN14, resulting in improved performance on MMAU and captioning. This improvement primarily stems from increased coverage and leveraging existing reasoning patterns rather than learning new ones.
Reasoning- focuseed synthetic data addition enhances performance. Our ReasonAQA experiments show that massoning-focusee training improves performance on reasoning tasks. When generating QA pairs for ReasonAQA, we rely on a generalized prompt for LLMs. However, one can also incorporate expert-written questions the prompt elicit more specific audio- and signal-centric reasoning queries. We observe that expert-prompe prompt-generater questions (Type-2 and Type-3) are valuable, and combining them with ReasonAQA (Type-1) leads to further performance gains (Type-4). While additional QA pairs generated via expert-question prompts can continue to boost performance.
6 Conclusion
We introduce Mellow, a small audio-language model for reasoning, demonstrating that sub-billion parameter models can achieve ffte-oo--he--a performance. Mellow is evaluated on diverse tasks, including multimodal audio understanding and reasoning, deductive reasoning, comparative reasoning. and audio captioning, and beats several larger models on the tasks. Through ablation studies, we identify key factors that improve reasoning in small models, including language model pretraining, projection layers, sassonnn--occuse synthetic data, and more. We hope this work inspires further research on improving reasoning in small audio audio-language models independent of data (audio) scaling.

---

References
[1] M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[2] L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, A. Marafioti, H. Kydlíček, A. P. Lajarín, V. Srivastav, et al. Smollm2: When smol goes big-data-centric training of small language model. arXiv preprini arXiv:2502.02737, 2025.
[3] Anderson, B. Fernando, M. Johnson, and S. Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 382-398. Springer, 2016.
[4] S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In J. Goldstein, A. Lavie, C.-Y. Lin, and C. Voss, editors, Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Sunmarization, pages 65 Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL
wrg/ 20zg/W05-0/ 1 0909
[5] L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdul- mohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint mTXiv:2407.07726. 2024.
[6] Y. Bisk, R. Zellers, Gao, Y. Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[7] G. J. Brown and M. Cooke. Computational auditory scene analysis. Computer Speech & Language, 8(4):297-336, 1994.
[8] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman. Vggsound: A large-scale audio-visun dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEFE, 2020.
[9] K. Chen, X. Du, B. Zhu, Z. Ma, T. Bong-Kirkpatrick, and S. Dubnov. HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[10] Y. Chu, J. Xu, X. Zhou, Q Yang. S. Zhang, Z. Yan, C. Zhou, and J. Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint 10X.v::311..79111 2023.
[11] Y. Chu, J. Xu. Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou. Qwen2-audio technical report, 2024.
[12] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun L. Kaiser, M. Plappert, J. Tworek,
J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.2111.1444 14.1668, 2021.
[13] S. Deshmukh, Raj. and R. Singh. Improving Weakly Supervised Sound Event Detection with Self Superviuee Auxiliary Tasks. In Interspeech 2021, pages 596 990. ISCA, 2021. doi: 2221437//iersspecch2221 2021
[14] S. Deshmukh, B. Elizalde, R. Singh, and H. Wang. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36: 18000-188888. 2023.
[15] S. Deshmukh, B. Elizalde, and H. Wang. Audio Retrieval with WavText5K and CLAP Training. In Proc. INTERSPEECH 2023, pages 2948-2952 2023. doi: 00224477/nerrpeechh223-3 136.
[16] S. Deshmukh, D. Alharthi, B. Elizalde, H. Gamper, M. Al Ismail, R. Singh, B. Raj, and H. Wang. Pam: Prompting audio-Language models for audio quality assessment. In Interspeech 2024, pages 3320-3324, 2024. doi: 10.214377/ntersseecch.e 202
[17] Deshmukh, B. Elizalde, D. Emmanouilidou, B. Raj, R. Singh, and H. Wang. Training audio captioning models without audio. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 371-375, 2024, doi: 10. 109/ ICASSP48485.200 2024. 202.445442244114441155 10448115.
10

---

[18] S. Deshmukh, S. Han, H. Bukhari, B. Elizalde, H. Gamper, R. Singh, and B. Raj. Audio entailment: Assessing deductive reasoning for audio understanding, 2024. URL https:// arxiv. arxiv.org// www.ooggaaaa.o Bwrg/abs/200 abaaaabs///0 18062
[19] S. Deshmukh, S. Han, R. Singh, and B. Raj. ADIFF: Explaining audio difference using natural language. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openzevviexx wwt.:/wwwwwweeeeeeeeeeeeeeeeeeee fore://orrm?m 1dd4
[20] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages
Transformers for Language Understanding.
Minneapolis, Minnesota, jun 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/119-444 URL
Barg/NN9-1422
[21] S. Dixit, S. Deshmukh, and B. Raj. Mace: Leveraging audio for evaluating audio captioning systems. arXiv preprini arXiv:2411.0032) 2024.
[22] S. Dixit, L. M. Heller, and C. Donahue. Vision language models are few-shot audio spectrogram classifiers. arXiv preprint arXiv:2411.1206 2024.
[23] K. Drossos, Lipping, and T. Virtanen. Clotho: an Audio Captioning Dataset. In IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. doi:
10.1 N0.100//CAASSP4777..220000 202..99077..220099999999
[24] A. Dubey, A. Jauhri, A, Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
[25] B. Elizalde, S. Deshmukh, M. Al Ismail, and Wang. Clap learning audio concepts from natural language supervision. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[26] B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang. CLAP: Learning Audio Concepts from Natural Language Supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023. doi: (0.1109/CASSP4993377
[27] B. Elizalde, S. Deshmukh, and H. Wang. Natural language supervision for general-purye -urross audio representations. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 336-340. IEEE, 2024.
[28] S. Evans. In two minds: dual process accounts of reasoning. Trends in Cognitive Sciences, 7 (10):454 459, 2003. ISSN 3333666613 doi: https://doi.orr///// 10. 1...11066..iiiio wtw..../...ii.........o. www................ 012. URL https://www. www.ssciiieiietttt... httpe//iiirtcttccccoeeeeeeeeeeee www.o///cciineeeariiiiiioooooeee http://eeeeeet.tt.ee///////ee//e
[29]
E. Fonseca, J. Pons Puig, X. Favory, F. Font Corbera, D. Bogdanov, A. Ferraro, S. Oramas, A. Porter, and X. Serra. Freesound datasets: a platform for the creation of open audio datasets. In Hu X, Cunningham SJ, Turnbull D, Duan Z, editors. Proceedings of the 18th ISMIR Conference; 2017 oct 23-27; Suzhou, Chinaa/Canadaj, International Society Music Information Retrieval. International Society for Music Information Retrieval (ISMIR), 2017.
[30] F. Font, G. Roma, and X. Serra. Freesound technical demo. In Proceedings of the 21st ACM international conference on Multimedia, pages 41 1-2 2013.
[31] Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780, 2017, doi: MLL110//CASSPP2217.799222
[32] S. Ghosh, C. K. R. Evuru, S. Kumar, U. Tyagi, O. Nieto, Z. Jin, and D. Manocha. Visual description grounding reduces hallucinations and boosts reasoning in Ivlms. arXiv preprint M.Ki:22405.1688. 2024.
[33] S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406. /768, 2024.
[34] Y. Gong, A. Liu, H. Luo, L. Karlinsky, and J. Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023.
11

---

[35] Y. Gong, H. Luo. A. H. Liu, L. Karlinsky, and J. Glass. Listen, Think, and Understand. arXiv preprint arXiv:2305.10799, 2023.
[36] Y. Gong, H. Luo, A. H. Liu, Karlinsky, and R. Glass. Listen, think, and understand.
In The Twelfth International Conference on Learning Representations, 2024. URL https //openreviev. net/ (tm.//oorrrmmm1mm222XX11111
[37] S. Gunasekar, Y. Zhang. Aneja, C. C. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffimann, G. de Rosa, O. Saarikivi, et al. Texthooks are all you need. arXiv preprint
arXiv:2306. 16644, 2023.
[38] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding. In International Conference on Learning Representations, 2021. http:::t///ppeereeiiiee net. wwwneee/fooreee forttm//mm j
[39] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. /ttpptttps//ooerreezee net/forum? id nZeVKeeFY19,
[40] M. Javaheripi, Bubeck, M. Abdin, J. Aneja, S. Bubeck, C. C. T. Mendes, W. Chen, A. Del Giorno, R. Eldan, S. Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):, 2023.
[41] M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqu: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 1601-1611, 2017.
[42] K. Khalifa. Understanding, Explanation, and Scientific Knowledge. Cambridge University Press, 2017.
[43] C. D. Kim, B. Kim, H. Lee, and G. Kim. AudioCaps: Generating Captions for Audios in The Wild. In NAACI-HLT, 2019.
[44] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In ICLR (Poster), 2015.
URL httt:://ttatiii/ org/ 0tbs//412.65.
[45] A. S. Koepke, A.-M. Onceseu, J. F. Henriques, Z. Akata, and S. Albanie. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multimedia, 25: 2675-2685, 2022.
[46] Q. Kong, Y. Cao, T. lqbal, Y. Wang, et al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. /EEE/ACM Trans. Audio, Speech and Lang, Proc., 2020. ISSN 2329-9290. doi: 10.1 A.1009TTASLP22000000000099 URL http:/ddioooogg//////// 1109/TASLP
2020. 3030497
[47] Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05466 054663,,0..,,.... 2023.
[48] Lipping. P. Sudarsanam, Drossos, and T. Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140-1144, 2022. doi: 10.2391 //EESSSCOSS00.....O.. 20w.:055553..022.9990 29099..... 10.2391/EUSSPCOSSSS
[49] H. Liu, Z. Chen, Y. Yuan, Mei. X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 2023.
[50] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy. Improved image captioning via policy gradient optimization of spider. In Proceedings of the IEEE international conference on computer vision, pages 873-881, 2017
[51] Z. Liu, C. Zhao, F. landola, C. Lai, Y. Tian, I. Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krish- namoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, 2024.
[52] S. Mehta, M. H. Sekhavat, Q. Cao, M. Horton, Y. Jin, C. Sun, I. Mirzadeh, M. Najibi, D. Belenko, Zatloukal, et al. Openelm: An efficient language model family with open-source training and inference framework. arXiv e-prints, pages ar aivi--2404 2024.
[53] X. Mei, X. Liu, M. D. Plumbley, and W. Wang. Automated audio captioning: An overview of recent progress and new challenges. EURASIP journal on audio, speech, and music processing, 2022( I) 1):2,, 2022.
12

---

[54] X. Mei, X. Liu, J. Sun. M. D. Plumbley, and W. Wang. Diverse audio captioning via adversarial training, In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 0882-8886. IEEE, 2022.
[55] X. Mei, X. Liu, J. Sun, M. D. Plumbley, and W. Wang. On Metric Learning for Ludio-Text Cross-Modal Retrieval. arXiv preprini arXiv:2203. / 2022.
[56] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang. Wav- Caps: A ChaiGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research. arXiv preprini arXiv:2303. 17395, 2023.
[57] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang. Wavcaps: A chatgpt-assised weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22:3339-3354, 2024. doi: 10.1 10.109/TASLLP:224444199999
[58] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
[59] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[60] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, and Others. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[61] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the Limits of Transfer Learning with a Unified Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URLhttp: /ttp::///mm...or/ppppss////a
20-074.htm1
[62] K. Sakaguchi, R. L. Bras, C. Bhagavatulla and Y. Choi. Winogrande: an adversarial winograd schema challenge scale. Commun. ACM, 64(9):99-106, Aug. 2021. ISSN 0001-0782. doi: 10. 10.1145/3474381. Jttpps/////dd org/ 10. 1 1145/3474381
[63] S. Sakshi, U. Tyagi. S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410. /9168, 2024.
[64] A. Steiner, A.S. S. Pinto, M. Tschannen, D. Keysers, X. Wang, Y. Bitton, A. Gritsenko, M. Min- derer, A. Sherbondy, S. Long, et al. Paligemma 2: A family of versatile vims for transfer. arXiv preprint arXiv:2412.033:
2024.
[65] A. Talmor, J. Herzig, N. Lourie, and J. Berant. CommonsenseeA: A question answering challenge targeting commonsense knowledge. In J. Burstein, C. Doran, and T. Solorio, edi- tors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4149-4155 Minneapolis, Minnesota, June 2019. Association for Computational URL /https::t////t /ccl anthology. org. NN9 1421/
Linguistics. doi: 20.18653v1/N19911
[66] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfih International Conference on Learning Representations, 2024. URL httpss:ts::///tt /ttttps://openrvveeee wttprrvvvviieeeefrouuuuutu net/forum? id=14rn7HpKVi
[67] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, I.. Sifre, M. Rivière, M. S. Kale, Love, et al. Gemma: Open models based on gemini research and technology.
arXiv preprint arXiv:2403.088 08090,00895.288 2024.
[68] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju. L. Hussenot, T. Mesnard,
B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at a practical size.
arXiv preprint arXiv:2408.0011 00 2024.
[69] A. Triantafyllopoolos,, I. Tsangko, A. Gebhard, A. Mesaros, T. Virtanen, and B. Schuller. Computer audition: From task-specific machine learning to foundation models, 2024. URL https://arxiv org/ abs/ 15672
[70] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, I. Beyer, Y. Xia, B. Mustafa, et al. Siglip 2: Multilingual lasion-language en- coders with improved semantic understanding, localization, and dense features. arXiv preprint 20w.22222.14786.. 2025.
13

---

[71] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4566 4576, 2015.
[72] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models zero-shot learners. In International Conference Learning Representations, 2021.
[73] Y. Wu, K. Chen, T. Zhang, Y. Hui, T.
and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-ter swwordto-ccaption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)) pages 1-5. IEEE, 2023.
[74] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? Proceedings of the 57th Annual Meeting of the Association for Compatational Linguistics, pages 4791-4800, 2019.
14

---

Appendix
Table of Contents
A Responsible A1
16
Ethics statement
16
A.2 Reproducibility statement
16
B Related work
ReasonAQA
17
C.1 Generating synthetic data
17
Analysis of synthetic data
19
Comparing with OpenAQA
21
D.1 Limitations OpenAQA
21
D.2
Qualitative comparison
22
E
Experimental setup
23
E.1
Architecture
23
E.2
Training
23
E,3
Inference
23
Evaluation
23
F
Ablation studies
24
F.1
Projection layer
25
F2
Choice of audio encoder
................................
25
F3
Choice of SLM
26
F4
Freezing vs finetuning the LM
26
F3
LoRA adaptation vs linetuning the LM
26
F6
Synthetic data for training
27
Scaling audio data
28
G Metric SPICE
29
H
Instruction following
29
Hallucination
31
Limitations
32
15

---

A Responsible AI
A.1 Ethics statement
The development and evaluation of Mellow prioritize fairness, transparency, and societal benefit. We acknowledge that models, especially those trained on multimodal data, can inherit biases present in their training datasets. To mitigate potential biases, we carefully curated the ReasonAQA dataset and removed any harmful generations from the data generation process. Mellow's primary applications are designed to enhance accessibilityy, improve audio understanding in real-world scenarios, and contribute to research in efficient, on-device AI models. However, we recognize that models of this nature could be misused for tasks such as unauthorized surveillance or generating mislcading audio-text outputs. To address this, we release Mellow with explicit usage guidelines and encourage responsible Al practices. To support transparency, we openly share our dataset and benchmarks for external audits. Additionally, Mellow's 's lightweight design and efficient training methods help reduce the environmental impact of Al research.
A.2 Reproducibility statement
We have implemented and thoroughly documented multiple measures to support the reproducibility of our work across the main paper, appendix, and supplementary materials. To this end, we make the following artifacts publicly available. (1) Model Checkpoints: We release the trained weights of Mellow to enable further research, fine-tuning, and benchmarking (2) Training Data: We provide the ReasonAQA dataset, including the synthetic question question-answer pairs derived from AudioCaps and Clotho, along with details on how the dataset was generated using large language models. (3) Experimental Details: We document all key hyperparaaeetess model architectures, and training procedures, allowing researchers to replicate our training pipeline. (4) Ablation Studies: We present thorough ablation studies on projection layers, language model pretraining, and synthetic data generation to provide insights into the model's design choices (5) All experiments were conducted on vell-documented benchmarks, ensuring comparability with existing methods. Furthermore, to promote transparency, we include error analyses highlighting areas where Mellow underperforms, such as speech-related reasoning tasks, and provide insights into potential improvements. By sharing these resources, we aim to enable the broader research community to build upon our results, findings, and further develop efficient audio- aadio-language reasoning models.
B Related work
Audio-text Learning. Recently, text has been increasingly used as a supervisory signal for learn- ing audio representations. This has led to the development of two primary learning approaches: contrastive audio-text learning (25) and audio-conditioned next-token prediction [14] Contrastive methods produce ladio-lngguage models capable of zero-shot classification and retrieval at test time, while audio-conditioned next-token prediction enables models to perform open-ended tasks such as audio question answering. In parallel, task-specific models that leverage language are also being developed, including models designed for audio captioning 23 53 audio-text retrieval [5]. and
text-to-audio generation 449.
Audio-Language Models.
With these two pretraining methods, the field is moving toward general- purpose Audio-Language Models. These models are pretrained on millions of audio-text pairs and can be prompted at test time to perform multiple tasks. Contrastive Audio-Language Models [25] 27
73
achieve -f-te-of-the-art (SoTA) performance on closed-ended audio tasks such as classification and retrieval, surpassing task-specific models. Similarly, generative Audio-Language Models [14] 36 [3]
10 (66] achieve SoTA performance on open-ended tasks such as audio captioning and audio question answering. Over the years, research has focused improving audio encoders [66] 33], language models (27], and pretraining and post-training strategies [0, I 19. A consistent trend across these improvements has been increasing both the training data and language model parameters, enabling ALMs to acquire novel capabilities previously unseen in smaller-scale models.
Audio reasoning, Scaling Audio-Language Models in terms of both data and compute has led to the emergence of novel abilities [16] [8] that were not explicitly trained for. In real-world scenarios, these models must process diverse types of queries, requiring them to listen (perceive) to the audio, understand the user's question, integrate world knowledge, and reason over both audio-text information and external knowledge to formulate responses. Enhancing the reasoning capabilities of
16

---

Audio- Audio-Language Models can significantly improve performance across multiple tasks. Consequently, recent studies [8 19 have focused on benchmarking reasoning ability. These benchmarks such as
evaluate various reasoning skills 633 and different types of logical reasoning [I 19]
deductive reasoning, inductive reasoning, and comparative reasoning.
Small Language Models. In recent years, there has been a growing emphasis on developing small language models that maintain strong performance while significantly reducing computational overhead. This research direction has given rise to a range of models, including the Phi series I. the smolLM series 22. OpenELM [52], MobileLLM [5T]. and others. These models employ techniques such as knowledge distillation, embedding and block wise weight sharing, training on curated synthetic and textbook data, deep yet narrow architectures, grouped-query attention mechanisms, and quantization. These approaches enable small models to achieve performance comparable to larger models while maintaining lower memory requirements and reduced energy consumption, making real-time, on-device inference feasible.
37
Small Audio-Language Models. The prevailing trend in audio-language modeling has been to scale up training data and computational resources (i.e., language model parameters), leading to unified models (66 TO [ capable of understanding and reasoning across diverse audio domains, including music and speech. However, it is essential to explore methods for enhancing comprehension and reasoning while operating under limited data and computational constraints. Models with fewer than 1 billion language model parameters, referred to as Small Audio-Language Models, offer unique advantages in terms of memory and energy efficiency, making on-device inference possible. Currently, the only Small Audio-Langun Language Model in the literature, Pengi [14), exhibits known limitations in performing open-ended tasks such as audio question answering (36] and reasoning- based tasks (63 [8 [9]. In this paper we explore and push the ceiling of Small Audio-Language
Model performance.
C ReasonAQA
The construction of ReasonAQA is described in Section 2 The data construction process consists of three main components. First, we select audio sources to build ReasonAQA around. This includes audio files from AudioCaps and Clotho, ensuring the dataset covers a wide range of audio events, acoustic scenes, and audio concepts. Additionally, by restricting the dataset to these two audio sources, maintain consistency in audio concepts and isolate performance improvements from data scaling. Second, we incorporate existing audio reasoning datasets from the literature, such as audio entailment and audio difference explanation, and convert them into the AQA format for training. Third, we generate a synthetic dataset for the selected audio sources in ReasonAQA. This synthetic data constitutes 70% of the total training dataset for ReasonAQA.
C.1 Generating synthetic data
In this section, we describe the third part of our data construction process: generating a synthetic training dataset for ReasonAQA. The process is illustrated in Figure Our goal with synthetic data generation is to create questions that are grounded in audio, elicit reasoning, and include a variety of question types, resulting in both detailed and multiple-choice (MCQ) answers. We use Llama 3 8B [24], an open- model with static weights, which enables the ReasonAQA pipeline to be reproduced on consumer- grade GPUs. The resulting synthetic dataset constitutes approximately 70% of the total OpenAQA training data, with 35% allocated to detailed QA (333k instances) and 35% to MCQ QA (337k instances).
Method. We use audio descriptions from AudioCaps and Clotho to prompt an LLM to generate
question-answer (QA) pairs. The process is illustrated in Figure 4 This approach of LLM-based
prompting has been widely used to create various datasets and benchmarks [57 36 18 19].
Detail AQA. The detailed AQA subset consists of question-answer pairs where the questions require detailed responses that incorporate audio events, acoustic scenes, signal characteristics. and their compositional and temporal relationships. To achieve this, we prompt the LLM with example questions such as: "what is the sound event present in the clip?", "what should I do when hear this sound?", "what is this sound most similar to?". This encourages the L.M to generate diverse question-answer pairs covering audio events, acoustic properties, psychological impact, and signal characteristics. The prompt used for generating this data is shown in Figure 5
17

---

Detall Prompt:
istering reading audio caplions. You just heard: (caption]g] Generate question-answer pairs about what you
return them JSON format. Some example questions look lkke what is the sound event present in the clip?. what should do when hear this similar Only return the JSON...
present the cip?
A: The sound event present talkinng nearay, accompanted by the sound of water pouring. The woman's voice the primary source...
the is
sound?,
Caption:
A woman talks nearby as
water
MCQ Prompt:
lstaning audios instead reading audio captions. just heard: [caplion(]]: Generale Mulliple Choices question-anewer pairs sbout what you heard return
What is the sound event singing B) Water pouring C) woman talking D) A car driving B) Water pouring
person
AudioCaps Clatho
example questions look like what the sound event present in the clip?, what should hear this sound?, most similar to?. Only return the JSON...
this sound
Figure 4: The data generation pipeline for creating the training data of RcasonAQA consists of three main steps. First, audio captions are sampled from the AudioCaps (43) and Clotho [3] datasets. Next, these audio captions inserted into detailed and multiple-choice (MCQ) templates construct text prompts. Finally, these text prompts are used to query a large language model (LLM),, which generates detailed and MCQ-based audio question-ane answer pairs.
MCQ AQA. The multiple-choice AQA subset consists of question- pairs where each question includes multiple-choice options, and the answer is a single selected option. MCQ-based questions are easier to evaluate due their classification nature, avoiding the challenges associated with evaluating open-ended responses. Furthermore, by explicitly providing answer options within the question, we mitigate issues related to ambiguous correct-yet-ireeevaan answers. The MCQ questions cover a wide range of audio concepts, including audio events, acoustic scenes, signal characteristics, and their compositional and temporal relationships. The prompt structure is similar that of the detailed QA generation process, with example questions such as: "What sound event is present in the clip?", "What should do when hear this sound?", "What is this sound most similar to?". The exact prompt used for MCQ generation is shown in Figure
{
helpful assistant expert knoxledge audio, acoustica, and 4 psychoacoustice. You study sudio, which is the study of sound and its propertiee. You study wares. range study Peychology shich posits that sound is complex stimulus that enconpasees of acoustic properties involving aspects of cogniticn, psychoscoustics, and peychonechanics, Your task is generation question-ansves paira based the captions of audio content using natural languago. To anaver the questions, you utilize words related to their acoustic relations, their spectro-tenporal characterietice, frequency, loudness, duration, materials, interactions, and sound sources",
syyaten_pronpt's
acoustice, shich revolve around the generation, propagation,
reception
propertiee,
"user_ proupt_detaill "You listening to the audios instead reading audio captions. You just ~ heard: [capticn[s]), Generate question-ansee ansver pairs about shat you just heard and return in detail. Sona example questions to inspiration + from vhat is the sound event present in the clip?, what should I do when I bear this sound?, dictionary containe
then in J80N fornat. Provide
to?. Only question key and an ansxer key',
sound
"user_prouph p.ppupp_tcq': "You listening the audios instead of reading audio captions. You just + heard: (capticn[s]}, Generate Maltiple Choices question-anssen pairs about what you just detail. Sone example questions like vhat is the sound event present in the clip?, what should I do when I bear this sound?, to?. Only
JSDN format. Frovide what sound nost question key, choices kay with alphabets and an ansver key with alphabets."
return
where dictionary containe
Figure 5: I.I. system prompt used to generate MCQ and descriptive questions for ReasonAQA, The "user prompt detail" and "user prompt mcq" shows the prompt used to generate the detailed and MCQ audio question- answer pairs respectively. In Ablation Table [4] this referred to as Type data generation.
18

---

C.2 Analysis of synthetic data
In this section, we analyze the data generated by the process described in C.1 which constitutes the synthetic portion of ReasonAQA. Specifically, this process uses an LLM to create multiple-choice (MCQ) and detailed question-answert pairs based on audio captions sourced from the AudioCaps and Clotho datasets.
Data distribution. We first examine the data distribution for the MCQ and detailed synthetic data, with Table reporting the word length and vocabulary size for both questions and answers. The ReasonAQA dataset comprises 209,412 multiple-choice (MCQ) and 222,499 detailed question- answer pairs, sourced from AudioCaps and Clotho. MCQs have an average question length of 11.39 words with a vocabulary size of 10,035, while their answers are shorter, averaging 7,356 words with a limited vocabulary of 2.31 words. contrast, detailed questions are significantly shorter (3.01 words on average) but exhibit much greater vocabulary diversity (25.65 words), with their answers being substantially longer, averaging 16,161 words. In all, the MCQs prioritize concise, factual understanding, whereas detailed QA pairs emphasize linguistic richness and deeper reasoning. making ReasonAQA well-suited for training models on both structured knowledge retrieval and complex audio reasoning.
Question
Answer
Category Dataset # of QA pairs (# of words) (# of words) (# of words) (# of words)
Length
Vocab,
Length
Vocab.
AC 113861
11.33
7697
2.28
5411
MCQ
Clotho 95551
11.46
8430
2.35
6009
Overall
209412
11.39
10035
2.31
7356
AC 127661
3.06
3034
25.90
13181
Detail
Clotho
94838
2.95
3443
25.32
13445
Overall 222499
3.01
4295
25.65
16161
Table 9: Data statistics of synthetically generated data for ReasonAQA
Data vocabulary. Since the data is generated by an LLM,, essential to analyze the characteristics of the generated QA pairs. By tokenizing the text and identifying audio-related words, we determine the most frequently occurring terms in both MCQ and detailed question-answe answer pairs. The results of this analysis are presented in Fig. 6 highlighting three primary areas of focus:
Acoustic properties. Both questions and answers frequently include terms related to fundamental acoustic characteristics. Words such frequency, range, loud, loudness, pitch, Hz, and descriptive attributes like low, mid, high, gentle, soft emphasize the analysis of sound's ss physical properties. Questions explicitly reference these aspects using terms like acoustic, psychoacoustic, frequency loudness, pitched, while answers reflect similar vocabulary.
Sound events and composition. Another major focus of the dataset is the identification and description of sound events within the audio. Questions often include terms like event, present, primary, background, sounds, noise, prompting the model to describe the compo- sition of the soundscape. Similarly, answers feature words such as primary, background, present, noise, highlighting how different elements within the audio contribute to the overall scene.
Inference and perceptual understanding. The dataset also contains questions and answers that require higher-level reasoning beyond direct acoustic analysis. For example, answers contain Words like likely, sense, similar, may, which suggests responses involve making inferences or drawing conclusions based on the audio. Likewise, questions containing terms such as emotional, similar, likely, sense are geared towards inferring meaning, context, and subjective perception from audio.
randomly sampled detailed and MCQ example from ReasonAQA is shown in Table 10aand Table respectively.
11
19

---

43000
1000
May
0000
Figure 6: Comparison of most frequent words in detail questions (top left), detail answers (top right), MCQ
questions (bottom left) and MCQ answers (bottom right)
Caption: The wind is blowing and rustling occurs
Question: What the sound event present in the clip?
Answer: The sound event present in the clip
rustling of wind through leaves or other materials, likely creating a gentle,
soothing background noise.
Question: What is the primary churacteristic of this sound?
Answer: The primary characteristic of this sound is gentle, soft quality, with a subtle variation in intensity and pitch due to
the changing direction and speed of the wind.
Question: What should do when I hear this sound?
Answer: When you hear this sound, you might want tn take moment to relax and calm down, as the gentle rustling of the wind cun have a soothing effect on the mind and body.
Question: What this sound most similar to?
Answer: This sound is most similar to the gentle rustling of leaves grass in breezy day, or the soft whispering of gentle streum or river.
Question: What acoustic properties are present this sound?
Answer: This sound exhibits properties of gentle amplitude modulation, with subtle changes in frequency and amplitude due to the varying wind speed and direction, creating sense of movement and texture.
Table 10: ReasonAQA detail type examples for sampled audio events.
Caption: The wind is blowing and rustling occurs
Question: What is the primary sound event present the clip? A) Bird chirping B) Wind hlowing C) Rain falling D) Ocean waves crashing
Answer: B) Wind blowing
characteristic of the sound the clip? A) Sharp and piercing B) Soft and gentle C) Rustling and whooshing D) Crushing and booming
Question: What
Answer: C) Rustlig and whooshing
Question: What is the sound most similar to? A) A fan blowing B) A leaf blower C) The wind blowing through trees D) A
vacuum cleaner
Answer: C) The wind blowing through trees
Question: What is the sound indicating? A) A storm is approaching B) The weather is calm C) A strong gust of wind is
coming D) A tornao forming
Answer: B) The weather calm
Question: What should you do when you hear this sound? A) Take cover and seek shelter B) Go outside and enjoy the weather
C) Check the weather forccast D) Close all windows and doors
Answer: B) Go outside and enjoy the weather
Table 11: ReasonAQA MCQ type examples for sampled audio events.
20

---

D Comparing with OpenAQA
The most widely used audio question-answering dataset in the literature is OpenAQA 35 OpenAQA
was created using GPT-3.5 Turbo to generate audio-based question-answer pairs from audio metadata. It comprises 845K unique audio clips sourced from datasets such as AudioSet [31], AudioSet-SI.
VGGSound [8]. FreeSound [30 29]. AudioCaps [43], Clotho [23], and SoundBible [[5]. In total,
OpenAQA contains 5.6 million QA pairs, including 3.7 million open-ended questions and 1.9 million
close-ended questions. is widely used for training audio-language models such as LTU [35]
LTU-AS 34, GAMA 33
and many more.
Answerable by World Knowledge
Sample: The sizzling of food while dish is clanking
Question: What kind of sound is sizzling?
Answer: Sizzling is the high-pitched sound that results from food being cooked on hot surface.
Question: What is the significance of the sizeling sound in cooking?
Answer: The sizzling sound in cooking indicates that food is being cooked properly and that the temperature of the cooking surface sufficiently high.
Refused Answers
Sample: Person whistling
Question: What the duration of the whistling?
Answer: The duration of the whistling is unknown.
Question: What is the pitch of the whistle?
Answer: It unclear what the pitch of the whistle is.
Repetitive Questions
Sample: The sizzling of food while dish clanking
Question: Does the audio clip convey sense of urgency emergency?
Answer: No, the audio clip does not convey any sense of urgency or emergency, as it is simply the sounds of food being
cooked.
Question: What time of day could this audio clip be recorded?
Answer: The audio clip could be recorded during any time day or night, cooking common activity that occur
at any time.
Bad Generations
Sumple: Someone has hiccup while lyping
Question: How might the sound events in the audio clip affect someone who hears it?
Answer: It's difficult to determine how someone else might react to this audio clip, as it depends on the individual's ' perception interpretation of the sounds.
Question: What might the perception of the audio clip chunge if other sounds added it?
Answer: The addition of other sounds would likely affect how the audio clip is perceived and interpreted, but it's difficult
say what those sounds might be without more information.
Table 12: Categories of audio question generation issues in OpenAQA
D.1 Limitations of OpenAQA
OpenAQA the largest open-source audio audio literature to train audio-language models. However, closer examination reveals several issues with the generated question-answer pairs, as illustrated in Table 12 These issues can be categorized into four types:
dataset and is widely used in the
Answerable by World Knowledge. A significant portion of the generated questions do not require any audio-specific reasoning and be answered solely using world knowledge. For instance, the question "What kind of sound is sizzling?" results in an answer that merely defines the term "sizzling" rather than analyzing the actual content of the audio. Such data does not ellict audio-grounded reasoning in the model.
Refused Answers. The generated questions frequently receive refusal as the answer, even when the necessary information is present in the audio. For example, in response questions about the sound of a person whistling, the reference answers include "The duration of the whistling is unknown." and "It is unclear what the pitch of the whistle is." Despite these aspects being extractable from the audio, the reference answer is a refusal. Such responses prevent the model from learning any acoustic features of the audio. While refusals appropriate when a question cannot be answered based on the audio alone, OpenAQA frequently refuses to answer even when a valid response is possible. As a result, large portions of the dataset contain refusals rather than meaningful acoustic reasoning.
Repetitive Questions. OpenAQA contains numerous redundant questions across different audio clips. Questions such as "Does the audio clip convey sense of urgency or emer-
21

---

gency?" and "What time of day could this audio clip be recorded?" appear frequently, even when they are not meaningful or contextually relevant to the given audio.
Bad Generations. OpenAQA also suffers from poorly formulated irrelevant questions. For example, a question like "What might the perception of the audio clip change if other sounds were added to it?" is vague and does not meaningfully engage with the specifics of the provided audio. This issue likely arises because the LLM is required to generate ten questions per caption, leading to an increase in illogical or poorly structured questions.
Caption: Someone fires gun with others talking in the background
OpenAQA examples
Question: What is the acoustic feature that allows you to differentiate between the gunshot and gunfire?
Answer: Gunshots and gunfire differ in their duration and intensity. Gunshots have a shorter duration and are louder compared
gunfire. Therefore, the acoustic feature that differentiates the two their duration and intensity.
Question: What is the potential scenario that this audio clip represents, given the comhination of sound events?
Answer: The socnario could be shooting. where somcone fires a gun and others in the background react by talking.
Question: What is the temporal relationship between the gunshot/gunfire and the speech in the audio clip?
Answer: The gunshot/gunfire happens before the speech in the audio clip. This is because the person fires the gun first, and then others in the background react by talking.
Question: What be inferred about the mood atmosphere that this audio clip conveys?
Answer: The mood atmosphere could be tense and chaotic since there background.
gunshot/gunfite and people are talking in the
Question: What can you conclude about the proximity of the penson firing the gun based on the audio clip?
Answer: It difficult to determine the proximity of the person firing the gun based on the audio clip alone since there no
information regarding microphone placement and other environmental factors.
Question: How can you tell whether the gun was fired indoors or outdoors based on the audio clip?
Answer: It is difficult to tell whether the gun was fired indoors or outdoors based solely on the audio clip since there is no
information about ambient noise or echocs.
Question: Who you think fired the gun based on the audio clip alone?
Answer: It impossible to determine who fired the gun based soiely the audio clip since there is no information regarding the person, their location, or the circumstances leading up to the gunshot.
Question: What is the difference between gunfire and single gunshot based on their acoustic features?
Answer: Gunfire refers multiple gunshots. Gunfire tends to have a longer duration and a lower intensity compared to a single gunshot, which is louder and has a shorter duration.
Question: What can you infer about the location of the other people in the audio clip who are talking in the buckground?
A Answer: It is impossible to determine the location of the other people in the audio clip based solely on the audio clip since there is no information regarding their location or the environment.
Question: What could be the reason for firing a gun the scenario represented by the audio clip? Answer: It impossible to determine the reason for firing gun in the scenario represented by the audio clip based solely on the audio clip since there is no context or additional information provided.
ReasonAQA exomples
Question: What the sound event present in the clip?
Answer: The sound event present in the clip gunshot, accompanied by background conversations and ambient noise. The gunshot is a sudden, sharp, and loud sound. followed by decaying reverberution, indicating a sudden releuse of energy. The background conversations and ambient noise provide sense of context, suggesting chaotic or noisy environment.
Question: What is the dominunt acoustic property of the gunshot?
Answer: The dominant acoustic property of the gunshot is its siggurryyy content, particularly in the range of kHz 20 kHz. This is because the sudden releuse of energy from the gunpowder creates a high-pressure wave that propagates through the air. producing sharp, crisp sound with geguurrryeyccy
Question: How does the gunshot sound different from the background conversations?
Answer: The gunshot sounds different from the background conversations in terms of its spectral characteristies, duration, and intensity. The gunshot has a much higher intensity, with a peak amplitude that is significantly higher than the background conversations. The gunshot also has much shorter duration, lasting around 0.1 0.1- seconds, whereas the bac background conversations are more sustained and have a longer duration.
Question: What is the psychological impact of hearing this sound?
Answer: Hearing this sound can have a significant psychological impact, particularly due to its association with violence. danger, sudden, intense events. The sudden and intense nature of the gunshot can trigger fight-or-fligh response, releasing stress hormones such as adrenaline and cortisol. The background conversations may also contribute to a sense of chaos or confusion, making it difficult to process the information and respond appropriatcly. Question: What is this sound most similar to?
A Answer: This sound is most similar to other sudden, intense sounds, such as firecrackers, car crashes, or explosions. The sharp. high- high-frequene frequency content and sudden release of energy are common characteristies of these types of sounds, making them easily recognizable and evncative of strong emotions and reactions.
Table 13: Comparing OpenAQA examples with ReasonAQA examples. From both dataset, sample detail questions per audio event.
D.2 Qualitative comparison
Both OpenAQA and ReasonAQA contain open-ended QA data, making OpenAQA somewhat compa- rable to the detailed question-answer pairs in ReasonAQA. Table 13 presents a direct comparison
22

---

of the generated QA pairs for the same audio file in both datasets. One key distinction is that ReasonAQA provides longer, more detailed answers for open-ended questions, allowing the model to generate richer and more informative responses. Additionally, ReasonAQA reduces the number of generated questions per iteration from ten to five, prioritizing quality over quantity and mitigating the redundancy and poorly formulated questions observed in OpenAQA. Unlike OpenAQA, ReasonAQA also includes multiple-choice (MCQ) question-answer pairs. By presenting multiple answer choices, the model learns to focus selecting the correct option, thereby improving its reasoning ability. Furthermore, ReasonAQA offers more precise descriptions acoustic properties such as frequency ranges, loudness, and duration, making it more structured and informative dataset for training audio-language models.
E Experimental setup
E.1
Architecture
the audio encoder. HTSAT truncates the audio to 10 seconds and produces three outputs: framewise, clipwise, and latent. The framewise output c) are the time-presence probabilities for AudioSet classes and the clipwise output (1 x <) are per-class probabilities averaged across time, The latent output (1 x () is the hidden state output before expanding to framewise probabilities using -eken-semantic CNN.
Audio encoder. The audio sampling rate is 32 kHz and we use HTSAT (
Mapper.
The output of the audio encoder is projected into the language model space using a mapper use the framewise output (t c) to retain temporal
(middle subplot in Figure 2). For Mellow, information and the latent output (1 x 7) as the audio summary and equivalent of CLS token [20]. The framewise output is projected using linear layer and concatenated with latent output, leading to the audio embedding output (1 + x I). This audio embedding output is passed to the projection layer. The output of the projection layer is then 8x downsampled using 2D average pooling. While downsampling, we retain the latent (CLS) output.
Projection. The projection layer consists of two linear layers, whose outputs merged followed by LayerNorm. This is shown on the right side of Figure
E.2 Training
Mellow is trained using next-token prediction where the model predicts the next-text token conditioned on audio and text input. The input to the model is audio I (zi), audio 2 (x²), text prompt (t') and the output is text (c"). The audios are encoded by an audio encoder (aş) and the text prompt is embedded
by text embedder (gr). The audio embeddings are projected using mapping network (m²), and then concatenated with the input text embedding. During concatenation, we add a separator token (s) between audio 1, audio 2, and text embedding.
p' - ..., pk concat(mc((c.((t)) 8, mc((a(()))) s,90 (f')}
The total prefix {p)) - is then used to prompt the language model (Jo) to produce text output. The model learns to predict next-token o¹ based on the prefix p². The loss is summation of cross-entropy over tokens:
ΣΣ ly po (o/lp) ... .ich pho of....,off of)1)
where y are Mellow's trainable parameters and consists of Ç. 0. We use Adam Optimiser 44] with cosine learning rate schedule with a maximum learning rate of le-3. We train Mellow and all the ablation models for 30 epochs.
E.3 Inference
We use top-p sampling where p is set to 0.8 and a temperature 1.
E.4
Evaluation
We use different evaluation metrics depending on the task, categorizing them into close-ended and open-ended tasks.
23

---

Close-ended tasks. For close-ended tasks such as classification and multiple-choice questions (MCQ). we use standard evaluation metries, including Accuracy, Precision, Recall, and Fc-score. These metrics are applied in tasks like Audio Entailment [18]. ClothoAQA 48), and MMAU 63
Open-ended tasks. For descriptive tasks, such as audio difference explanation 19 and audio captioning 43 23 we use text-based evaluation metrics, including BLEU (59), METEOR 44) SPICE CIDEr [7]], and SPIDEr [50]. to compare generated responses against ground-truth descriptions. BLEU (Bilingual Evaluation Understudy) assesses the precision of n-grams between generated and reference text, providing a simple and efficient approach, though it does not account for recall, word order, or deeper semantic meaning. SPICE (Semantic Propositional Image Caption Evaluation) improves upon BLEU by analyzing the semantic content of text using scene graphs, capturing deeper meaning rather than relying on surface-level n- n-gram matches, though at higher computational cost. SPIDEr, a combination of SPICE and CIDEr, integrates both semantic and consensus- based evaluation, balancing precision with deeper content understanding. Since the existing Audio-Language Model (ALM) literature predominantly uses SPICE, we adopt the same metric to ensure fair comparison with prior work.
F Ablation studies
We perform ablation studies to identify which components most effectively enhance reasoning in
audio-language models, with particular focus on Small ALMs. In particular, we examine the effects of audio encoders (Section F.2). language model choice (Section F33, projection layers (Section F.I) prefix-tuning versus fine-tuning (Section F.4 LoRA adaptation (Section [.5). synthetic data generation (Section [66), and scaling audio data (Section F.7
The results are shown in Table g
Audio Caption
B-AQA
Audio Entailment
Audio Difference
MMAU (lext-mini)
Models
Size
AC
CL
ClothoAQA
CLE
ACE (ACC)
CLD-3
ACD-3
Sound Music Speech Avg.
(SPICE) (SPICE)
(ACC)
(ACC)
(SPICE) (SPICE) (ACC) (ACC) (ACC) (ACC)
The SLM is GPT2 and frozen and the projection layer changed
Linear
156M
4.53
5.98
53,23
30.79
30.15
5.21
8.39
16.80
31.25
30.10
26.05
Non-lincar
2
157M
4.79
6.10
55.18
32.12
29:56
5,56
8.62
17:42
33.23
27.03
25.89
Transformer
14
195M
7,89
62.35
43.89
42.10
12.50
28.65
35.23
26.68
The SLM is GPT2 and finehmeed and the projection layer changed
Linear
156M
9.87
7.10
70.90
93.45
92.90
13.65
14.21
47.6 64
45.
27.00
40.01
Non-lincar
2
157M
10:51
7.15
71.25
93:40
93:27
13.77
14.01
48.05
48.50
27.33
41.29
Transformee
14
195M
7.23
70.89
92.32
93.65
13:45
15.21
47.89
49.10
27.10
The SLM is frozen, projection layer is non linear and the SLM is changed
GPT2 frozen
157M
4.79
6.10
55.18
32.12
29.56
5.56
8.62
17.42
33,23
27.03
25.89
SmolLM2 frozen
167M
5.26
8.58
45,70
33.40
31.66
11.43
13:62
35.14
30.54
20.12
28.60
The SLM finetuned, projection layer is non-linear and the SLM is chonged
GPT2 linetune
157M
10.51
7.15
71.25
93:40
93.27
13.77
14.01
48.05
48.50
27.33
41.29
SmolLM2 finetune
167M
18.60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28.82
46.29
Diferrnt finenaning methodds
The STM Smoll projection layer non--beea
Prefix-moinn
167M
5.26
8.58
45,70
33.40
31 66
11.45
13.62
35.14
30.54
20.12
28.60
LoRA (8. 16)
167M
18.53
9.25
64.64
79.33
84.15
14.23
14.98
45.95
42.51
28.83
39.10
LoRA (256, 512)
181M
19.01
10.59
65.54
86.66
89.87
15.36
16.51
50.75
49.10
33.33
44.40
Finetuning
167M
18.60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28.82
46.29
The SLM Mm and finetuned, projection laver is non-lineor and the audlo encoder is changed
CNN14
219M
15.97
7.91
65.82
91.06
92.39
16,27
16.95
$4.05
47.60
28
43.30
HISAT
167M
18.60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28.82
46.29
SLM SmolLM2 and finetuned, projection layer is non linear, trnining
changed
Type I
167M
18.60
9.83
71.65
92.00
90.85
17.33
18.68
59.46
50.60
28 82
46.29
Type 2
167M
16.47
8.23
71.05
92.50
93.20
16.98
18.09
59.45
42.81
37.5
46.70
Type
167M
17.43
9.88
66.83
91.87
94.25
17.37
18.67
61.56
45.21
32.43
46.40
Type
167M
17.79
9.38
71.39
91.16
89.66
17.21
18.54
61.26
54.19
29.73
48.40
The SLM is MmolI.2 and finetuned, projection layer is non- -innornnor, and WauCaps is addedd to mainnng
Type 1
167M
18.60
9.83
71.65
92.00
90.85
17.35
18.68
59.46
50.60
28.82
46.29
+ WavCaps
167M
14.83
9.66
71.32
92.47
92.69
17.92
19.13
59.16
60.48
23.72
47.80
Mellow
167M 17.79
9.38
71.39
91.16
89.66
17.21
18.54
61.26 $4.19 29.73
Table 14: Ablation study results, For the reader's convenience, we reproduce here the same table presented
Table 8
24

---

Mapper
Linear
Non-linear
Transformer
Cutput (i)
Linear
Linear
Projection
p| 1,
Input (+)
Output (8)
GeLU
Drop
Downsample
Transformer
Linear
Linear
=[1.4]
u|
Layer Norm
Concal
constant
Imput (+)
Output
Input (#9
Figure 7: The common type projection layers used literature. The Mapper used in Mellow shown on the left side. For the projection layer in the mapper. we ablate with different popular projection layer structures simple linear, non-linear, and transformer with learnable constant.
E. Projection layer
We experiment with different projection layers to assess their impact on model performance. In previous studies involving frozen language models, the literature [14] 17 has primarily used stack of transformer layers with learnable constants. More recently, Q-Former [33 has been intro- duced, which increases the number of learnable constants while also making them query-dependent. In our experiments, we explore the use of linear, non-linear, and transformer-basse projections in the mapper. For consistency across projection ablations, the linear layer and downsampling in the mapper remain fixed and are designed as a function of the chosen audio encoder.
The ablation results, presented in Table 14] indicate that transformer- baasooeder-b-eedddd projection consistently outperforms both linear and non-linear projections when the language model is kept frozen. Since the projection layer is primarily responsible for steering the frozen language model toward generating the desired output, higher parameter count in the projection layer becomes essential for effectively guiding Audio-Language Models (ALMs). However, when the language model is trained instead of frozen, the advantage of a nigh-parameter projection layer diminishes, and simpler linear or non-linear projection layers perform comparably.
F.2 Choice of audio encoder
Audio encoders are trained on large-scale audio datasets using either self-supervised learning or cross-entropy loss over labeled data. In the audio literature, pretraining audio encoders on AudioSet in a supervised setup has been common practice [14 25
33 36). These pretrained audio encoders are trained to predict 527 audio events, and their latent representations can be effectively applied various downstream tasks across sound and music using a linear-probe setup, where linear layer is trained to classify downstream task labels. For Audio-Language Models (ALMs), we can assume that higher performance on AudioSet and improved linear-probe performance on downstream tasks indicate better separability in embeddings, which, in turn, enhances reasoning capabilities by covering a broader range of audio concepts.
27
To evaluate this, we conduct an ablation study using CNN14 46 and HTSAT 9 as the chosen audio encoders. CNN14 is M-arammee CNN-based model released in 2019, achieving 43.4 mAP on AudioSet and serving as one of the seminal models in audio literature. In contrast, HTSA is M-parmeeter transformer-basee and previously used as an audio encoder in ALM literature [25] 14). Both models exhibit strong linear-probe performance, with HTSAT consistently outperforming CNN14. For the ablation study, we independently evaluate these audio encoders, followed by a mapper network and a non-linear projection, as illustrated in Figure 7 The linear layer dimensions in the mapper are set according to the latent dimensions of CNN14 and HTSAT. The ablation results, presented in Table [4] show that HTSAT outperforms CNN14 on MMAU, audio captioning, and binary AQA, while both models achieve comparable performance on audio difference and audio entailment tasks. These findings suggest that improving audio encoders enhances overall performance by increasing the coverage of audio concepts, thereby improving reasoning over broader range of audio events. However, better audio encoders do not inherently enhance reasoning ability itself, as indicated by the similar performance of CNN14 and HTSAT on the audio entailment and audio difference tasks.
model, released in 2022, achieving 47.1 mAP AudioSet
25

---

F33 Choice of SLM
In recent years, several Small Language Models (SLMs) have been developed 60 2 37
47
68
While the definition of what constitutes a "small" model has evolved over time initially referring to models around IB parameters and now extending up to 8B parameters the performance of SLMs on reasoning tasks has shown continuous improvement. Our objective is to analyze whether the latest advancements in SLM pretraining and architectural modifications lead to improved reasoning performance for
Models (ALMs).
In the audio-language modeling literature, the smallest existing Audio-Language Model is Pengi 14. which utilizes GPT-2, ~ 125M parameter language model released in 2019. As state-of-ther tfrth--rrt (SoTA) SLM, we select SmolLM2 (2. which has a comparable parameter count (~135M). To isolate the effect of the language model, we keep the audio encoder, mapping network, and projection layer unchanged while replacing the language model. We evaluate both settings: keeping the language model frozen and fine-tuning it. The results, presented in Table [4] indicate that training with SmolLM2 outperforms GPT-2 in both scenarios, whether the language model is frozen or fine-tuned. Notably, in the audio difference explanation task-which requires the model to employ comparative reasoning to describe detailed (~155 words) differences between two audio samples-we observe a significant performance improvement when using a more advanced SLM.
E4 Freezing vs finetuning the LM
In this ablation study, we compare prefix tuning with fine-tuning the language model. Our results show that fine-tuning the language model consistently improves performance compared to keeping it frozen. While this outcome is expected, using Small Language Model (SLM) makes full fine-tuning feasible, unlike Large Language Models (LLMs) with 8B+ parameters, where full fine-tuning is often impractical. Moreover, the performance improvement is substantial, prefix-tuned models perform nearly at random on reasoning tasks.
If the language model must remain frozen, the mapping network needs to be proportionally increased in size relative to the language model to compensate for the lack fine-tuning. Prior research [19] has demonstrated that three-stage training micces--unmmdia pretraining, multimodal grounding, and fine-tuning-yicn -velds the best downstream performance. Previous studies relied on larger transformer- based projection layers, making the multimodal grounding stage essential. In contrast, we reduce the projection size by employing a simpler two-layer linear network and instead prioritize fine-tuning the model. This leads to a two-stage training process consisting of unimodal pretraining and fine-tuning. Our results indicate that using a non-linear mapper, we achieve performance comparable to- to-though slightly lower than the ttree-stage training approach.
F5 LoRA adaptation vs finetuning the LM
In this ablation study, we compare LoRA adaptation 33 against full fine-tuning of the language model. LoRA has been employed by existing ALMs to fine-tune language models [36] 34 [3]. Following the ALM literature (36), we use a LoRA configuration with rank 8 and a scaling factor of 16, applied to the projection layers of the key and query in all self-attention blocks. For a 7B- parameter model, this setup introduces about 4.2M additional parameters; for 135M-parameter model, it adds about 0.46M parameters.
Our results show that fully fine-tuning the language model consistently improves performance compared to LoRA adaptation. This is unsurprising, since full fine-tuning utilizes 135M trainable parameters, whereas LoRA adapts only 0.46M on the decoder side. Nevertheless, LoRA enables the model to retain broader world knowledge and the general capabilities of the ALM, thereby mitigating catastrophic forgetting. For example, the LoRA-adapted model can still answer questions such as "Who is the president of the United States?" or "What is the color of the sky?", tasks that may not be audio-related, which the full-finetuned model cannot. Retaining this general knowledge might be expected to improve the model's 's ability to handle new instructions and tasks; indeed, this appears partially true when the tasks are not strictly tied to audio. However, for audio-specift tasks, the LoRA-adapted model shows limited and comparable generalization to the fully fine-tuned LM.
While if one increases the rank and scaling factor, we see consistent improvement in performance. In cases like audio captioning performance, we see the LoRA model beating the full-finetuned model on SPICE metric. This can be attributed to retaining language model information and limitation
26

---

of SPICE metrics discussed in Table 18 The LoRA model also retain world world-knowledge better as previously discussed, however, the improvements in LoRA come with increase in parameter count (167M to 18) Therefore, the decision to use LoRa not, depends on the goal to be achieved with the model. If the model uses a small LM and used mainly for audio tasks, we recommend finetuning the model, while is the goal is a ceneral-purpose assistant with audio capabilities LoRA is better option.
F.6 Synthetic data for training
We generated the ReasonAQA synthetic data using the methodology described in Section 2, referring to this generation method as Type 1. However, synthetic data can created in various ways by modifying prompts and using different language models. To analyze the impact of these factors, we also generated two additional datasets: Type 2 and Type
Data generation. In Type 2 data, we modify the prompt while keeping the LLM fixed as Llama 3 8B. The new prompt incorporates expert-designed questions to guide the LLM in generating questions focused on audio and signal reasoning. The Type 2 prompt is shown in Figure X For Type 3, we retain the same prompt as Type 2 but upgrade the model from Llama Llama 3.1 8B. Qualitative examples comparing all three data types for both MCQ and detailed question-answer pairs are presented in Table an and Table [7 respectively.
{
"syaten.pronpt" prompt": "You are helpful assistant with expert knoxledge about audio, acoustica, and 4 psychoacoustice. Your task is to generation question-anewer pairs based on the captions of audio content using natural language.",
captions. 4 heard: (caption[t]>. Generate 5 question- ansver paira about what you just heard and return them in JSON fornat. Please focus on all the events and sounda occurring in the audio clip. Identify and describe each sound source, euch as objecte, animale, veather, or environmental noises. Include information about the sequence of events and any interactions betveen sound the contert setting it 4 sorda related to their acoustic propartiea, such as thair samantic ralationa, their tenporal characteristicss, frequency, loudness, duration, materiale, interactions, sound sources. Provide ansvera in detail. Sone example questions to take inspiration from what is the sound event present in the clip7, what can be inferred about the envirommett7. Only return the JS0X vhere each dictionary contains question key and anaver key. ",
"user_ "user_prompt_dee :ettipptetdedaii "You are listening to audios instead reading
just
Make
inferred from the sounds. Utiliza
"user_prane piser_proppt.et prompt_ncq": "You listening the audios instead of reading audio captions. You just 4 heard: [caption[i]} Generate 5 Multiple Choicea gueation-ansser pairs about what you juat JSON format. Please focus on all the events and sounds accurring the audio clip. Identify and describe each sound source, auch as objecta, aninala, veather, or enviroumental noises. Include information about the sequence of evente and any interactions
4 and return
between sound sources. Make use of the context setting if it be inferred from the sounds. Utilize vords related to their acoustic proparties, such as their senantic relations, their spectro- temporal characteriatics, frequency, loudneas, duration, naterials, interactions and sound sources. Provide anavera in detail. Sone exanple questions to take inspiration from what the sound event present in clip?, what can inferred the environment? Only return the JSON where each dictionary contains question key, choices with alphabete and an ansver key with alphabets."
Figure 8: LLM system prompt used to generate MCQ and descriptive questions for Type 2 and Type 3 audio- question answer pairs. The "user prompt detail" and "user prompt meq" shows the prompt used to generate the detailed and MCQ audio question-anse answer pairs respectively. For Type 2, LLM, while for Type we use Llama 8B as the LLM. The system, detail and meq prompts are the same for both Type and Type 3. The results of Type and Type are shown in Table 16 and
Llama 8B as the choice of
Data analysis. The statistical analysis of Type 2 and Type 3 data, shown in Table 15 highlights the impact prompt modifications and model choice on question- question-answer answer generation. Type 2 data exhibits lower vocabulary diversity and shorter questions and answers compared to Type 1, indicating that prompt design significantly influences the complexity and richness of the generated question- answer pairs, even when using the same language model. Additionally, Type 3 data shows higher vocabulary diversity and longer questions and answers than Type 2, demonstrating that changes in the LLM can affect the quality of the generated data, independent of the prompt.
27

---

Question
Answer
Type Category Dataset of QA pairs
Length words)
Vocuh. words) of works) of words)
Lengih
Vocab.
AC
113861
11.33
7697
2.28
5411
MCQ
Clotho
95551
11.46
8430
2.35
6009
Overall
209412
11.39
10035
2.31
7356
Type
127661
3,06
3034
25.90
13181
Detail
Clotho
94838
2,95
3443
25.32
13445
Overall
222499
3.01
4295
25.65
16161
AC
242037
13.58
9118
2.51
6600
MCQ
Clotho
13.45
2.51
Overall
337487
13:55
10805
2.51
8041
Type
AC
241701
3.85
3349
20.74
12302
Detail
Clotho
95470
3.77
3315
20,10
10794
Overall
337171
3.83
4453
20.56
14153
AC
239006
13.77
9680
3.07
6984
MCQ
Clotho
95253
15.29
8797
3.02
6325
Overall
334259
15.63
11408
3.06
8445
Type
AC 237772
4.07
3500
26.19
13488
Detail
Clotho
95264
4.00
3622
24.74
11811
Overall
333036
4.05
4764
25.77
15477
Table 15: Statistics of generated data types ReasonAQA
We also explore the qualitative differences in the generated questions and responses between these datasets (Type 2 and Type 3) and the original dataset (Type 1). To do this, we identify words that appear in Type 2 but not in Type 1 to determine unique audio characteristics present in Type 2. Similarly, the same method is applied to examine the distinctions between Type and Type 1. The results of this analysis, shown in Figure reveal notable shifts in word usage, highlighting differences in question formulation and response patterns. Specifically, Type and Type 3 questions show an increased occurrence of words such environment, inferred, and setting, suggesting a stronger emphasis on reasoning about environmental and contextual aspects of the audio. This aligns with the prompt instructions, which include directives such "focus all the events and sounds occurring in the audio clip" and "make use of the context or setting if it can be inferred from the sounds." Similarly, answers in these datasets contain words like likely, possibly, and suggests, indicating greater focus on inference and probabilistic reasoning compared to Type 1. Another key finding is that despite differences in the underlying LLMs. Type 2 and Type 3 data exhibit similar vocabulary patterns, with certain words appearing more frequently in both compared to Type 1. This consistency underscores the importance of prompt design in shaping the generated data, demonstrating that the structure of the prompt significantly influences the nature of the questions and answers, even when different LLMs are used.
F.7 Scaling audio data
In Mellow, our primary objective was enhance the model's inherent reasoning ability through archi- tectural, learning, or post-training techniques rather than by increasing data (audio) coverage-such as expanding knowledge of audio objects and scenes. A natural follow-up question arises: now that we have a recipe for training small audio- audio-language models, does performance improve when we scale the training data? To investigate this, we leverage the WavCaps dataset [57]. Using the same method described in Section F.6, we generate approximately 1.80 million detailed QA pairs and 1.82 million multiple-choice QA pairs for WavCaps, resulting a total of 3.6 million QA pairs. For our experiment, we incorporate these WavCaps QA pairs into the ReasonAQA type dataset.
The results, presented in Table 14, reveal an absolute 1.5% improvement in overall MMAU perfor- mance. Notably, music-related performance increases by an absolute 10%, likely due to WavCaps' heavy reliance on FreeSound (70%), which consists predominantly of music data. Performance on MMAU test-mini sound remains comparable, while speech performance appears random since we do not train on speech data. the case of Audio Captioning, we observe decline performance. A closer analysis of the model's output suggests that this drop is primarily due to limitations in the SPICE metric, as discussed in Section [8] Meanwhile, for Audio Entailment, we see slight improvement, which can be attributed to the model's improved understanding of audio concepts absent in the type I dataset, allowing the use of learned reasoning chains over this new knowledge.
Overall, these results demonstrate that expanding the range of audio concepts in the training data
28

---

200330
08093
138230
123330
40090
Oaspe
Werd
220030
120000
100090
120030
00090
40090
ess
xare
Werd
Wars
Figure 9: Comparison of frequent words in type questions and that present in type 1. Comparison of most frequent words type questions (top left), type answers (top right), type questions (bottom left) and type answers (bottom right) that are not present in type 1
leads to improved model performance. Future work could explore scaling laws for audio, allowing extrapolation of small-scale experiments to predict the effects of training on significantly larger audio datasets.
G Metric SPICE
SPICE 3 is sensitive to wording variations and proves overly stringent for evaluating the audio question-answerta task. As shown in Table [8] SPICE performs well when the predicted response exactly matches the reference (c.g., "reading" and "reading") or is very similar (c.g., "bus" " and "yes, it is bus"). However, when the predicted response is longer phrased differently from the reference, SPICE tends to assign low scores even to correct answers. As responses become more complex, SPICE frequently assigns
score, failing recognize their correctness. To address these limitations, we also incorporate the latest metrics from the audio-language literature [21 (22), such the AURA metric, evaluate the model's performance the audio question-answertg task.
H Instruction following
Audio-Language Models (ALMs) are capable of performing various tasks, including answering open-ended questions. However, many ALMs in the literature are known to struggle with following instructions when these instructions deviate significantly from their training distribution. This issue particularly evident when answering deductive reasoning questions [8] or multiple-choice questions (MCQs) [63]. where the model might only reply with yes or no, select one of the options arbitrarily, provide entirely unrelated answer. In our experiments, observed that Mellow exhibits similar issues with instruction following.
To evaluate instruction following, we selected the MMAU benchmark as our target. Although Mellow can handle MCQ tasks, the audio, questions, and answers MMAU are distr--distriuution relative to Mellow's 's training data. This setup allowed us to assess how often Mellow fails to select an appropriate option in MCQ format, instead producing irrelevant responses that do not adhere to the provided options or the question. Out of approximately 10k questions in MMAU, we found that Mellow failed to pick an appropriate option in roughly 20% of the cases. The reasons for this failure can be categorized follows:
29

---

Caption:
A man talks nearhy as water pours out of a faucet
Type 1
Question: What is the primary sound event present in the clip? a) A person speaking b) Water pouring from a faucet c) Background noise d) combination of both
Answer: d) A combination of both
Question: What the most likely material the object making sound? a) Metal b) Glass c) Plastic Ceramic
Answer: Metal
Question: What is the approximate loudness of the sound? a) Very loud b) Moderately loud c) Soft d) Very soft
Answer: b) Moderately loud
Question: What the most similar sound to sound water pouring from fancet? a) Rainfall b) Ocean waves c) Running water
A babbling brook
Answer: c) Running water
Question: What the purpose of the sound of water pouring from faucet? a) create background noise b) mask other sounds C) To create a sense of activity d) To create sense of calm
Answer: c) To create a sense activity
Type
Question: What is the primary sound event present in the clip? A) A man talking B) Background noise C) Other D) Water pouring out of faucet
Answer: D) Water pouring out of a faucet
Question: What the sound source of the pouring water? A) stream B) bucket C) A pool D) A faucet
Answer: D) faueet
Question: What can he inferred ahout the environment? A) It is an outdoor setting By is private place C) It is public place D) It is
an indoor setting
Answer: D) It an indoor setting
Question: What is the relationship between the man's 's vuice and the pouring water? A) The man's voice is louder than the water B) The before water
man's voice simultancous with water The man's voice softer than the water D) man's
Answer: B) The man's 's voice is simultancous with the water
Question: What is the dominant acoustic property of the sound of the pouring water? A) Frequency B) Duration C) Material D) Loudness
Answer: D) Loudness
Type 3
Question: What is the sound event present in the clip? A) A person talking B) Neither A nor B C) Water pouring out of a faucet D) Both A and B
Answer: D) Both A and B
Question:
can be inferred ahout environment? A) aren C) It is outdoors, near a stream or river D) It indoors, likely in a kitchen or bathroom
public place, as a restaurant or store B) quiet, isolated
Answer: D) is indoors. likely in a kitchen or bathroom
Question: What the material
object making sound of water pouring? A) Metal B) Glass C) Plastic Wood
Answers A) Metal
Question: What is the semantic relation between the sound of the man talking and the sound of water pouring? A) They are competing for allention B) The sound of the water pouring is causing the man talk C) They occurring simullaneoussly and independently D) The sound of the man talking is causing the water to pour
Answer: C) They
occurring simultaneussly and independently
Question: What can be inferred about the loudness the sound of the man talking? A) It is much louder than the sound of water pouring B) roughly equal in loudness the sound water pouring C) much softer than the sound water pouring D)
slightly louder than the sound of water pouring
Answer: D) is slightly louder than the sound of water pouring
Table 16: Comparing ReasonAQA Type 1, Type 2 and Type 3 examples. From both datasets, we sample MCQ questions generated for the same audio event.
Lack of task-specific knowledge. In cases such music chord identification phoneme recognition, Mellow does not possess the necessary knowledge to identify individual chord letters or phonemes. As result, the model produces responses based on related knowledge (c.g., referencing guitar playing) instead of selecting an option from the MCQ list.
nntt---diitribation symbols. When the options contain symbols that were not present in the training data (for example, an underscore or differently formatted list of answers), the model sometimes generates gibberish or random responses. Ideally, the model should still select the correct option or produce a valid English response but fails to do so.
String parsing errors. Occasionally, the model produces response that is valid option but includes extraneous characters, such as an extra space, exclamation mark, or question mark. These errors can cause traditional prring-passng methods to fail. This issue could be mitigated by using more robust evaluation metrics, such as leveraging a language model for parsing instead of relying solely on string matching.
Overall, similar to other existing ALMs, Mellow's 's ability to follow instructions has room for improvement. Some of these errors could be resolved by increasing the training data and incorporating a more diverse set of instructions and answer distributions. However, it is also possible that at compute optimal frontier, larger models are inherently better suited to generalize to novel instructions compared to smaller models.
30

---

Caption:
A man talks nearhy as water pours out of a faucet
Type 1
Question: What the prinary sound event present in the clip?
Answer: prinary sound event present in the clip the sound of a man talking, accompanied by the sound of water pouring out of a
faucet. The talking is a dynamic sound with varying pitch, tone, and volume, while the water pouring sound is a steady, continuous sound with consistent pitch and volume.
Question: What is the acoustic property of the water pouring sound?
Answer: The water pouring sound has a characteristic of a steady, continuous flow, which is likely due to the consistent flow rate of the water. This sound also has gentle, soothing quality, which is often associated with low moderatee frequency range (around 100-500 Hz) and moderate to high volume level.
Question: How would you describe the tone of the man's voice?
Answer: The lone of the man's voice likely conversational and casual, with natural, relased pitch and a moderate volume level. The
tone may also convey a sense familiarity or routine, as the man likely speaking in everyday conversation rather than in a formal or dramatic context.
Question: What
sound most similar to?
Answer: The sound of the talking and the water pouring is most similar typical morning routine, such brushing teeth
washing hands. The combination of the gentle water sound and the conversational tone creates seuse of familiarit and comfort.
Question: What
psychoacoustic effect of this sound?
Answer: The sound of the man lalking and the water pouring may lave calming soothing effect the listener, the gentle water sound the conversational tone create a sense create sense of focuss as the listener is drawn to the gentle, continuous flow of the water.
and familiarity. The sound may also help distract from other stimuli
Type
Question: What sound event is present in the clip?
Answer: The sound event present in the clip a man talking and water pouring out a laancet. The talking is low pitched, steady state sound with moderate loudness, while the water pouring high-pitchod, transient sound with varying loudness.
Question: What can be inferred about the environment?
Answer: From the sound of the water pouring out of a fancet, can be inferred that the environment is likely a indoor setting, such as a the listener.
kitchen or bathnoom, with sink and rumning water. The man's talking suggests that he is likely in the same Question: What is the acoustic property of the man's 's talking?
quality. which suggests that his voice likely deep and resonant. The moderate loudness of his talking indicates that he is speaking at normal volume, not too loud or too soft.
Answer: The man's 's talking has a low- low-pitched, steady-
Question: What is the acoustic property of the water pouring?
Answer: The water pouring has high pitched. transient quality, which suggests that a fast moving fluid with a lot of energy. The
varying loudness of the water pouring indicates that the flow rate of the water is changing, possibly due to the man adjusting the fancet.
Question: What is the interaction between the sound sources?
talking and the water pouring are two separate sound sources are occurring simultameously. The talking is a steady-state sound, while the water pouring is a transient sound, which suggests that the man is likely talking while be is adjusting the fancet or observing the water pouring. There no apparent interaction hetween the sound sources, but they coexist same environment.
Answer: The
Type 3
Question: What the primary sound event present in the clip?
Answer: The primary sound event present the clip man talking nearby, accompanied by sound of water pouring out a faucet.
Question: What can be inferred about the environment based the sounds?
Answer: Based on the sounds, it can be inferred that the environment is likely domestic or residential setting, such as a kitchen or bathroom, where faucet present and person nearby.
Question: What are the acoustic proerties of the sound of water pouring out of fancet?
Answer: The sound of water pouring out of a faucet is a continuous, flowing sound with steady, consistent pitch and moderate to likely broadband noise with frequency range around 20 to 20 kHz, and duration that sustained
high loudness level. throughout the clip.
Question: How do the sound of the man talking and the sound water pouring interact?
Answer: The sound the talking and the sound of water pouring interact creating a of spatial separation between the two sound sources. The talking sound is likely coming from a closer location to the microphone, while the water pauring sound is coming from a farther location. creating sense of depth and distance between the two sound sources.
Question: What can be inferred about the man's inns proximity to the microphone based on the sound of his voice?
Answer: Based on the sound of his voice, can be inferred that the man likely standing relatively close to the microphone. as his voice is loud and clear, with minimal reverberation
echo. This suggests that he is relatively small, enclosed space, such as vnice can he easily heard.
kitchen or bathrom, where sound
Table 17: Comparing ReasonAQA Type 1, Type 2 and Type 3 examples. From both datasets, we sample detail questions generated for the same audio event.
I Hallucination
Language models often struggle with hallucination, generating outputs that are misleading or incorrect. For Audio-Language Models (ALMs), this issue manifests as producing responses not based on actual audio input. For instance, an ALM might invent audio events, misinterpret common sounds like human speech, or completely disregard audio information. This lack of grounding can impair the model's 's ability to answer user's questions faithfully. Therefore, we take inspiration from past work [1). and provide users with a way to detect hallucinations. To achieve this, we use the HTSAT audio encoder, which is part of Mellow, It trained on AudioSet, and can predict 527 audio events and their presence probabilities over time. This enables users to view the detected audio events and their presence across time. Users can then compare the generated descriptions with the audio event presence probabilities to verify the description's 's accuracy. An example of this is shown in Figure
31

---

Question
Reference
Answer
Haman
SPICE
AURA
Answer
What activity is taking place?
reading
reading
1.00
1.00
Can you identify type of vehicle the audio clip?
Yes, it is
hus
1.00
1.00
1.00
hus.
What kind brush the using?
tooth
Based on the sound, sounds like person using loothbrush to brush their teeth. This is the sound of flowing water.
0.22
1.00
What liquid llwwnng sound is this?
water
1.00
0.40
1.00
somcone using
yes
machine ranning.
1.00
machine?
what room would these items likely be found?
kitchen
These items ure typically fourdl kilchen or dining where glasses and silverware commonly used.
1.00
0.00
1.00
Table 18: AURA scores correlate better than SPICE with human scores on open-ended question answers. GitHub
link: httpp///giiuuu.coo//eeee com/satvik dixittaara
10
The top pane displays the audio difference explanation generated by the model. For each audio, the log mel spectrogram and the top three audio event presence probabilities over time are plotted. From the figure,
the detected audio events and compare them to the generated description from Mellow. Though this method requires numan-in-loop and is not fully automated, provides a way detect hallucinations in critical scenarios.
Log Mei Spec trogs Spectrogaam am
20
Mei Spectrogram Audio
400
900
600
1000
10
Events acress Audio 2
Eurepn mnctater
2023
Figure 10: The user audio event presence probabilities from HTSAT detect potential hallucinations in the language model output of Mellow,
Limitations
Despite Mellow's strong performance on multiple downstream tasks, it has several limitations. First, Mellow not general-purpose audio model. While can understand sound, music, and paralinguistics certain extent, it is not trained for speech-content tasks such as transcription or translation. By training on limited data, Mellow has limited knowledge of audio concepts, which can be improved with scaling training pairs. Second, in Mellow, fine-tune the language model
32

---

Model
Response
Non-binary questions
Question: What is the helicopter doing? Reference Answer: flying
Audio Flamingo
Blyng
GAMA
The helicopter seems to be hovering or stationary, as indicated by the constant presence of its sound throughout the audio.
Qwen Audio Chat
The helicopter is flying.
Qwen Audio
The helicopter is flying overhead.
Mellow
The belicopter flying
whooshing whirring sound that andible throughout the clip. The sound is likely due to the rotor blades of the helicopter moving through the air.
background. creating
Question: on fire? Reference Answer: fireplace brake
Audio Flamingo
GAMA
The audio does not provide specilic information about what's on fire, but could be a camplire u lireplace in the context home setting during rainstorm.
Qwen Audio Chat
It sounds like rain falling on various surfaces. pot of something
Owen Audio
fire.
Meelow
The lire is burning wood, which is evident from the crackling and popping sounds heard in the clip. The wood is likely dry and combustible, given intensity and frequency of the crackling and popping sounds.
Questiin: What is the gender the person speaking? Reference Answer: male
Audio Flamingo
male
GAMA
The speaker male, indicated by the andio label and the presence of man's voice in the recording. The gender of the person speaking is male.
Qwen Audio Chat
Owen Audio
The speaker make.
Mellow
the gender of the person speaking is an adult male, as inferred from the deep and resonant tone of his voice.
Binary questions
Question: Are the children playing outslde? Reference Answer: yes Audio Flamingo
yes
GAMA
Yes, the children are likely playing outside based the continuous presence of their voices and the sounds of birds
chirping in the background.
Owen Audio Chat
Yes, the children are Sayingg outside. Yes
Qwen 2 Audio
Mellow
yes, the children are playing outside, as evidenced by sounds of children's 's laughter. footsteps, and possibly
some toys games being played. the sounds of children playing outside are likely to be combination of footstops. laughter. and possibly some toys being moved around.
Question: Is this taking place outside? Reference Amswer: yes
Audio Flamingo
yes
GAMA
The presence of wind sounds suggests that scene is likely ouidoooss possibly near a road. Yes, this is taking place surside. Yes
Qwen Audio Chat
Qwen Audio
Mellow
yes, this sound is taking place outside, the bus moving and the sound is carrying over distance. the sound is not mufiled or dampened, indicating that the bus not enclosed in building or heavily vegetated area.
Question: Are the
swimming?? Reference Auswer:
Audio Haainng
GAMA
No, sounds suggest they likely in a vehicle or hus, as indicated by engine impact noises. No, the men are not swimming. They are working on hus truck..
Qwen Audio Chat
Owen Audio
No. the men are not swimming.
Mellow
no. the men are swimming in the clip. the audio clip appears to be focused solety the sound of the truck driving away, with no additional sounds or ambient noise present.
Table 19: Example of different Langu-Language Model responses to audio-based questions from ClothoAQA
component. This causes the model to forget factual knowledge unrelated to audio, such as information about the capitals of specific countries or historical and present-day facts that not relevant to audio. Third, Mellow is not a chat-based model and cannot be used as a general-purpose assistant like some existing models in the literature I10 [1] Instead, Mellow closely follows the instruction-tuning paradigm seen in models like T5 (61). FLAN (72), SigLIP2 [70]. and PaliGemma series [5] 64 where the goal is to enable a model perform multiple perception tasks through language. Fourth, as the data scale increases, Mellow's 's parameter count will likely need to be scaled as well to maintain a compute-optimal frontier. However, with Mellow, we aim to demonstrate that the compute-optimal frontier for speech and audio tasks may be significantly lower than previously estimated. In summary, we hope this work inspires researchers to explore small langoalan models capable of performing multiple tasks, alongside the existing research on general-purpose purpose audio assistants.
33

---

Reasoning ever context
Comparative Reasoning
materials involved sound? M: sounds, can inferred the bowl and pipe are likely made of coramic or parcalain, as these materials produco a disinolive sound when waler flows through them. The sound of the water draining also suggests that the pipe likeyy made of a smooth material,
G: can inferred
Q: Explain difference
short
M: The two audios ditter sigrificantly thei acoustic proparties, with the first audio tecturing chaolic midure of high fraquency buzzing sounds with verying loudness and duralion, wheress the second audio has structured soundscape with a low-frequency rumble of the vehičle. punctuated by the deer and
PVC when water flows through
produces low-frequency rumble
voices the people speaking.
O: Explain the difference between the two audios detail M: As liston to the two audios, notice dislinct differonces thair acoustic properties. The first aucio, featuring kid screaming at park, indicating high pttched trequency with rising loudness lavel, emotionel outburst. background chatting and lughing crealed gentie hum, adding a sense environent context. In contrast the second audio is truck driingg
Q: What can be inferred from this audio?
M: From this audio, can inter that the woman likey in social setting, possibly with friand family the conversation sharing thoughts someone laughter and conversetion suggest a lighthearted and playful atmosphere.
that she onjoying
people
down the street, has deep. rumbling frequency with steady, constant loudnass level, charactorisiic of powerful engine. The background noise traffic, Insluding homs ravving anginas, creatos cacophonous soundscapo. evoking sense urban chaos. While audios heve high levels of loudnees, the kid's scream is more attention- grabbing, wherees the truck's a engine austained.
G: What can be inferred about the environment? M: Tha anvironment with body water, vegetation, and possibly some buildings structures in the background, based on the sounds of water spleshing and rusting.
natural
such pond take,
Identifying and reasoning over sound events
Deductive Reasoning
Q: How do the two sound events interact with each other?
Q: Based on the audio, what can be said about the hypothesis "A faimmer is giving tour of his ranch while chickens roam nearby"? a) definitaly true b) definitely false o) plausible cannot M: la definitlly falee
M: The baby's a crying sound and the TV sound interact with each other in The TV
way that the crying sound dominates the sudlory
the background.
masked
the crying sound. This suggests the caregiver family member trying to create a distraction or background noise soothe the baby.
a: How do different sounds Interact with each other?
M: The ongine sound is the dominant sound the clip, and it appoars to be the primary secondary
Multiple Choice Questions
attontion. the high pitched serse leneion of urgency to overall The interaction between the two sounds creates seree of dynamic energy and movement, suggesting that the engine mation or under stress.
Q: What is the acoustic property of the wind sound? A) Soft and gentle B) Low frequency and rumbling G) High frequency and loud D) Sharp and piercing
A) Solt gentle
Q: How do the duck calls relate the other sound events in clip?
Q: What is the characteristic the machine's sound? A) Whiny and squealing B) High-pitched and smooth C) Low-pitched and rough D) Sitent
M: The duck calls are Iikeyy being blown by person, and are ueed to interact with the ducks, possibly to ture them or to communicate with them, and are heard conjunction with the humann conversation, suggesting scene of duck hurling observatio.
A) Whiry and squaaling
O: What is quality the audio the clip? High-quality b) Medium-quality c) Low-quality d) Excellent-quallit M: cj Law-quaity
Q: What happons the stiming sound over
M: Tha stiring sound gradually siows down and becomes lass intenso. with a decrease in frequency and loudness. The spoon's motion becomes more deliberste and controlled, resulting lower amplltude sound with frequency range approximetely 100-400 loudness around perouesive sound to a solter, more mulled sound.
What primary sound event present the alip?
cleaner b) chirping birds c) car horn d) clapping M: vacuum cleener
sound traneftins from sharp,
G: Can you identity the specific type of airoraft engine from the audio?
Binary Questions
type engine, sound charactoriatic whooshh or whine' quality, which is typical of jet engines. the engine's pitsh and tone are consistent with a high-perlamence jel engine, poesibly a turbojet or turboprop engine.
M: While difficult pinpoint
Q: there more than one bird squawking in the audio clip? Answer using yes M: yes
Figure 11: Mellow's ' different capabilities and examples.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims are accurately reflected in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
34

---

Gusesdionn Platerence:
eeegpening Re Reforommos buu
beiciig
Audio Planinge:
Audio Planinge: Potio
and to greor is the autlo
coep inttinnoo with the came cound.
indole a out ootther duoho retuond
Mallow: No. them people present the aucio domtrated
Metlow: The atren belong emergeno police ambulares.
energeniy
Figure 12: Mellow is able reason complicated audio events and generate detailed answers shown left. Mellow also has the ability recognize information that cannot be inferred from the audio alone such the origin siren as shown on right.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings. is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the limitations are highlighted Section Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
35

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The manuscript contains experimental results and not new proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: In every result section, describe the dataset and experimental setup used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
36

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, released the ReasonAQA dataset and will subsequently release the training and evaluation code June to reproduce all experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
CodeSuomi ssinnPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Yes, all experimental details are available in Section E in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
The experimental results are averaged over inference runs of the model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the details are available in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the manuscript and work complies with the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we acknowledge this Section Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We manually filter the data to remove any potential harmful examples related to gunshot, violence and surveillance.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we use audio datasets to create ReasonAQA and the dataset and creators are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
39

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: Yes, we provide data statistics and model checkpoints.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: There were human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There were human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
40

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLMs for spell check and grammar correction.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
41


Reviews:
[{"review_id": "UkgpmfSk4E", "reviewer": "Reviewer_n6cA", "summary": "This paper advances the understanding of building efficient ALMs with regard to trade-offs between model size, data constraints, and performance. While authors do not employ novel, groundbreaking techniques, I think this represents solid technical work with a good and important direction for the LLM-reasoning community. Although given the speech performance and limited audio knowledge, the claims regarding immediate applicability require further consideration.", "questions": "Please see the weakness section above. All questions are embedded with the weakness I found.", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nAmidst the trend towards scaling LLMs to billions of parameters, I think this paper is well-motivated towards closing the gap by showing that their 135M model is able to achieve competitive reasoning over its bulkier counterparts.\n\nOverall the paper is well-written and clear to understand. The evaluation methods are well-explained and cover multiple reasoning types {deductive, comparative, ood}\n\nImportant ablations are provided across tuning methods (lora, finetuning, prefix), data scaling effects with wav2caps, and prompting styles.\n\nWeakenss:\nWhile the authors have intentionally constrained to using only Audiocaps+clotho to isolate reasoning improvements from those of scaling parameters, I think this severely limits the model's knowledge of audio concepts and of course generalization. \nWould a model with double Mellow's parameters but trained on the same constrained data still underperform due to insufficient audio concept coverage?\nWould the simple projection and HTSAT encoder scale effectively to larger parameter counts? Is this reasoning ability transferable or clotho-specific pattern matching? \nI think demonstrating reasoning improvements in isolation, may not provide sufficient evidence for the broad claim that small models can maintain competitive performance in practical applications where both larger models AND larger datasets are available\n\nCould you please provide Fig. 3 to compute the drop against only music and sound subsets of MMAU? I think this should help validate “(1) Speech-based task constitute nearly 30% of MMAU … limited speech training ..” In my opinion, this is not just a limitation but a fundamental weakness for an audio understanding model.\n\nWhile 135M parameters are indeed smaller than cloud-scale models, it still requires significant resource (>500 mb;  authors, please confirm this.). Most successful applications for on-device or even always-on are in 1-5M param range - considered truly on-device AI which occasionally trigger server-side computation for main LLMs. Given their param count, currently both Mellow and Qwen2 Audio sit in the same bucket - i.e both are for server side deployment.\nIf a device triggers a late-stage process, it already transmits the audio to the server. Given that the server hypothetically has no compute constraints, why would one opt for Mellow there instead of passing the audio to bulkier QewnAudio for potentially better results?\nI think some clearer articulation of the intended deployment scenarios would immensely help this paper.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our technical contributions, thorough experiments, clear writing, and providing constructive feedback. We address every question and hope that our response resolves your concerns. Any follow-up questions are welcome.\n\n---\n**While the authors have intentionally constrained ... performance in practical applications where both larger models AND larger datasets are available**\\\nThe reviewer raises a valid point regarding the choice of experimental setup and its implications. To reiterate the premise: in audio models, reasoning can improve through two approaches. First, expanding data (audio) coverage (e.g., knowledge of audio objects and scenes), which provides a broader foundation upon which the model can apply its existing reasoning capabilities. Second, enhancing the model’s inherent reasoning ability through architectural, learning, or post-training techniques. By fixing the training audio, one can conduct controlled evaluations of an ALM’s reasoning performance, which can later be scaled via data expansion. Therefore, we break down the reviewer’s question into two main parts: (1) do reasoning improvements achieved in isolation translate when trained on larger datasets? and (2) do models with smaller parameter counts retain competitive performance when larger datasets are available?\n\nFor point (1), we agree with the reviewer that by constraining data, we limit the model's knowledge of audio concepts and its generalization ability. However, we believe that by identifying a minimal recipe on constrained data, the approach can later be scaled (in both data and parameters) to achieve better performance. One empirical piece of evidence supporting this is shown in Table 8 (the \"+ WavCaps\" row) and further discussed in Appendix F7. In this experiment, we utilize the popular WavCaps dataset and, using the same generation method as ReasonAQA, generate additional training data. This data provides broader coverage of audio concepts, and the results show an absolute 1.5\\% improvement in overall MMAU performance. Notably, music-related performance improves by an absolute 10\\%.\n\nFor point (2), the WavCaps experiment provides some indication that improvements hold when the training data is increased. However, how competitive this approach would be compared to larger models trained on larger datasets remains an open question, and this work does not provide evidence to conclusively answer it. Unfortunately, due to resource constraints, we cannot scale the parameter count to conduct the necessary experiments for validation. Nevertheless, we believe this work is an essential step toward building efficient ALMs, which will eventually help close the performance gap between large models trained on massive datasets and smaller ALMs.\n\n**Could you please provide Fig. 3 ... weakness for an audio understanding model.**\\\nWe thank the reviewer for this suggestion. We conducted this experiment for Mellow, where we measured the drop in performance on the music- and sound-only subsets of MMAU. However, due to space restrictions, we did not include it in the paper. The results are shown in the table below:\n| Experiment | Audio Input | Gaussian Noise | Difference |\n|---|---|---|---|\n| Sound | 61.26 | 39.24 | 22.02 |\n| Music | 54.19 | 39.79 | 14.4 |\n| Speech | 29.73 | 28.70 | 1.03 |\n| Average | 48.40 | 35.91 | 12.49 |\n\nThe results show that performance on sound and music drops significantly, while speech performance changes only marginally. This implies that the model actively utilizes audio information when answering sound and music questions but relies mostly on the language model for speech predictions. This behavior is expected since the model was not trained on speech data. Therefore, in the paper, we state: “(1) Speech-based task constitute nearly 30% of MMAU … limited speech training ..”, which explains why the overall average difference is not higher. We will add this discussion to the Appendix.\n\n**While 135M parameters are indeed smaller than cloud-scale models, ... of the intended deployment scenarios would immensely help this paper.**\\\nWe thank the reviewer for raising this point and helping to improve the manuscript. Mellow, while having 167M parameters, still occupies approximately 670 MB of storage. We agree with the reviewer that most successful on-device or always-on applications typically operate in the 1-5M parameter range. However, there are multiple instances where LLaMA-like models are successfully deployed on devices such as iPhones, laptops, and tablets after applying quantization and other optimization techniques.\n\nBy scaling down from 8B to 135M parameters, we can significantly improve the tokens-per-second throughput of these models. This improvement translates to faster responses and better support for older devices on the edge side, while on the cloud side, it results in lower inference costs and reduced latency. Even in server-side deployments, where compute resources can theoretically scale infinitely, minimizing inference costs remains desirable. In the NLP domain, this approach has already proven viable with models such as Phi, SmolLM, and MobileLLM. Similarly, choosing Mellow or other small ALMs offers benefits for both edge and cloud scenarios. For concrete numbers, we evaluated Mellow and existing ALMs on an Nvidia V100 GPU, where each model was tasked with captioning 100 randomly sampled audio files from the AudioCaps test set. Throughput was computed as the number of tokens produced per second. The results are summarized below:\n\n| Model            | Parameters  | Total Time (s) | Total Tokens | Throughput (tokens/second) |\n|------------------|-------------|----------------|--------------|--------------------|\n| Qwen-Audio Chat  | 8.4B        | 52.99          | 3,039        | 17.04              |\n| GAMA             | 7B          | 199.30         | 3,593        | 18.03              |\n| Audio-Flamingo   | 2.2B        | 93.70          | 1,254        | 13.38              |\n| Mellow           | 167M        | 148.40         | 5,224        | 35.21              |\n\nA higher throughput indicates better performance. As shown, Mellow is significantly faster than the representative 7B model. Moreover, models like Qwen-Audio 2 cannot fit on a V100 GPU and require larger memory. We believe that pursuing this research direction will lead to performant yet efficient ALMs."}, {"review_id": "xOGmMRKPF4", "reviewer": "Reviewer_rXRV", "summary": "This paper introduces Mellow, a small audio language model for audio reasoning. The paper also introduces ReasonAQA dataset for training Mellow. ReasonAQA is built based on AudioCaps and Clotho data. Experiments show that training on ReasonAQA enables Mellow to achieve relatively good performance on the MMAU benchmark.", "questions": "- Did the authors try to use ReasonAQA data for fine-tuning audio LLMs such as SALMONN or QwenAudio? Will this approach yield more remarkable results?\n- How does Mellow perform on other audio benchmarks besides MMAU?", "limitations": "See Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The Mellow model with only 167M parameters can achieve promising performance on the MMAU benchmark.\n\nWeaknesses:\n- About the novelty. The model structure seems to merely replace a large language model with a small one, which is somehow trivial, and the model's improvement seems to stem from the ReasonAQA data. It is not surprising to obtain better results through training with higher-quality data. Therefore, I believe the novelty of this paper is limited.\n- Does Mellow understand speech? I believe it cannot understand speech content at all, as the annotations in AudioCaps and Clotho don't include any conversation-related information. If that's the case, the lack of speech understanding ability in Mellow also limits its novelty.\n- Regarding the rationality of the results. I am concerned that many baseline models are underestimated. Most models in Table 4 have scores below 60, but Mellow exceeds 91, which is rather questionable. Could you provide the results of some state-of-the-art (SOTA) models for reference, such as Qwen-Omni or Gemini? In addition, as for the caption performance of Table 4.4, the caption results of some models are lower than what I expected. For example, the SPICE score of SALMONN-13B is less than 10, which is weird.  When testing audio LLMs, did the authors adjust different prompts?  Some audio LLMs are quite sensitive to prompts, and inappropriate prompts could lead to lower results. The authors should adjust the test prompts to avoid underestimating the performance of the tested models.", "was_revised": true, "final_justification": "The authors' rebuttal provides results on the MMAR benchmark to show the robustness of Mellow. Therefore, I will raise my score to 4.", "rebuttal": "We sincerely thank the reviewer for the detailed comments. We reiterate the key contributions of our work: this paper presents Mellow, a small ALM capable of reasoning over audio and text, outperforming much larger models on several benchmarks (MMAU, Entailment, Difference Explanation, Captioning, AQA) while using only ~152 hours of training audio. We also introduce ReasonAQA, a 1M-instance dataset with curated reasoning questions grounded in audio content. Finally, our ablation studies reveal the effectiveness of architectural choices and training strategies in enabling reasoning in compact ALMs, and we present several additional findings. \n\nWe have carefully addressed each of the reviewer’s concerns below. Any follow-up questions are welcome.\n\n**About the novelty. The model structure seems to ... the novelty of this paper is limited.**\\\nWe thank the reviewer for the comment. To provide context from the literature, one of the first ALMs released was Pengi, published at NeurIPS 2023. It was a small ALM that achieved state-of-the-art performance at the time. In subsequent years, the field largely shifted toward increasing parameter count and training data for ALMs, leading to performance improvements and a move from small to large ALMs. However, during this progression, the necessity of large models was rarely questioned. Fundamental questions -- such as what are the limits of small audio-language models? How far can we push their performance? What architectural changes improve small ALMs, and do these changes scale with additional data? -- remain largely unexplored. We believe answering these questions is crucial for understanding both small and large ALMs and for building efficient models. Therefore, we start with Pengi, the only small ALM in the literature, and subsequently improve it while keeping the model size comparable.\n\nWe disagree with the sentence *\"the model’s improvement seems to stem from the ReasonAQA data ... through training with higher-quality data.\"* First, high-quality data is indeed important, and we believe ReasonAQA is a valid, novel contribution to the field, particularly since AQA datasets are currently very limited. Second, data is only one factor contributing to model performance. For example, Pengi -- the only small ALM in the literature -- achieves 3.40 on the MMAU benchmark. To test the effectiveness of ReasonAQA, we trained a Pengi-like model using ReasonAQA data alone. This model achieves a score of 30.19 on MMAU, as shown in Table 8 (line 3). We then made subsequent improvements to the projection layer, pretraining strategy, SLM fine-tuning, audio encoder, and data augmentation techniques, ultimately reaching an updated score of 48.40 on MMAU. This 18-point absolute improvement is solely due to the architectural and training improvements proposed in the paper.\n\n**Does Mellow understand speech? I believe ... speech understanding ability in Mellow also limits its novelty.**\\\nWe thank the reviewer for requesting clarification. Mellow is not trained on ASR data and therefore cannot perform speech recognition tasks. As a result, its ability to understand conversation-related information is limited. However, this is consistent with prior ALM literature, where models primarily focus on audio and music understanding rather than speech content. For example, Pengi (NeurIPS 23), LTU (ICLR 24), and GAMA (EMNLP 24) are all trained solely on audio and music data, resulting in similarly limited speech understanding capabilities. While conversation-related abilities are desirable, our current work aligns with the existing audio literature and provides sufficient evidence to support the claims being made.\n\n**Regarding the rationality of the results ... the test prompts to avoid underestimating the performance of the tested models.**\\\nWe understand the reviewer’s concern and thank them for raising this valid point. The numbers and benchmarks reported in Table 4 are borrowed from the Audio Entailment paper (Table 4), published at EMNLP 2024. These are standard literature-reported results with prompts optimized for each task by the respective paper authors. Since these represent state-of-the-art results, we report them directly without further prompt optimization.\n\n**Did the authors try to use ReasonAQA data for fine-tuning ... more remarkable results?**\\\nWe thank the reviewer for the question. We used ReasonAQA to retrain Pengi, which is the only model with a parameter count comparable to Mellow. The results are shown in Table 8, line 3. The retrained Pengi achieves an MMAU score of 30.19, whereas Mellow achieves a score of 48.40. We did not evaluate QwenAudio or SALMONN in the submitted paper. However, following the reviewer’s suggestion, we fine-tuned QwenAudio on the ReasonAQA dataset using both LoRA fine-tuning and full fine-tuning (with no frozen parameters). The results are shown in the table below:\n| Experiment | MMAU performance | \n|---|---|\n| Pengi | 30.19 |\n| QwenAudio - Zero Shot | 41.86 |\n| QwenAudio - LoRA | 45.90 |\n| QwenAudio - full finetune| 43.72 |\n\nThe full fine-tuning version performs worse than the LoRA fine-tuned model. Upon examining model responses, we see that the fully fine-tuned QwenAudio heavily overfits the ReasonAQA dataset, likely due to the mismatch between the dataset size and QwenAudio’s parameter count. In contrast, LoRA fine-tuning allows the model to adapt to ReasonAQA while retaining its broader world knowledge. Scaling ReasonAQA to better match QwenAudio’s parameter count might improve performance, but this is beyond the scope of the current work.\n\n**How does Mellow perform on other audio benchmarks besides MMAU?**\\\nWe appreciate the reviewer's suggestion and assistance in improving our work. We benchmark Mellow on several other audio benchmarks, including Audio Entailment (deductive reasoning), Audio Difference (comparative reasoning), Audio Captioning, and Audio Question Answering. The results and detailed explanations are provided in Sections 4.2, 4.3, and 4.4, respectively. We specifically focus on MMAU benchmark, which consists of 10k annotated AQA pairs spanning the speech, sound, and music domains. This benchmark evaluates a model’s capabilities across 27 distinct skills involving various reasoning tasks. Notably, Mellow is trained exclusively on AudioCaps and Clotho, whereas MMAU sources its audio from 10 different datasets, with no overlap with Mellow’s training data. This makes MMAU an ideal benchmark for evaluating Mellow’s reasoning performance in an out-of-distribution setting.\n\nHowever, to address concerns about overfitting or in-distribution training, we also evaluate Mellow on the MMAR benchmark. It consists of 1k examples spanning sound, music, speech, and their mixtures, and focuses on multi-step, deep reasoning beyond surface-level understanding. Moreover, MMAR was released after Mellow, making it a completely independent benchmark with no possibility of overfitting. The results are shown below:\n| Models| Size| Sound (%) | Music (%) | Speech (%) | Sound-Music (%) | Sound-Speech (%) | Music-Speech (%) | Sound-Music-Speech (%) | Avg (%) |\n|---|---|---|---|---|----|---|---|---|---|\n| **Audio Flamingo** | 2.2B   | 32.73 | 21.84 | 24.83 | 18.18 | 30.28  | 24.39 | 25.00 | 26.60|\n| **Audio Flamingo 2**| 1.0B| 20.61| 20.39| 24.15| 27.27| 23.85| 26.83| 25.00| 23.00|\n| **Audio Flamingo 2**| 2.5B| 26.67| 20.87| 22.79| 9.09| 22.94| 23.17| 20.83| 22.90|\n| **Audio Flamingo 2** | 4.7B  | 24.85 | 17.48 | 20.75 | 18.18 | 26.61| 23.17| 8.33 | 21.90|\n| **LTU** | 7B| 19.39| 19.90| 13.95| 18.18| 24.77| 21.95| 16.67 | 19.20|\n| **LTU-AS** | 7B| 20.00| 14.08| 19.05| 9.09| 20.64| 28.05| 12.50 | 19.00|\n| **MusiLingo** | 7B| 9.09| 7.28| 4.08| 9.09| 6.88| 7.32| 8.33 | 6.6 |\n| **MU-LLaMA**| 7B| 13.94| 13.59| 14.97| 9.09| 12.39| 14.63| 16.67 | 13.90|\n| **GAMA**| 7B| 29.09| 24.27| 27.89| 27.27| 24.77| 28.05| 20.83 | 26.50|\n| **GAMA-IT** | 7B| 22.42| 16.02| 12.24| 36.36| 22.48| 14.63| 12.50  | 17.40|\n| **Qwen-Audio-Chat**| 8.4B| 27.88| 20.39| 22.11| 9.09| 25.23| 25.61| 20.83 | 23.50|\n| **Qwen2-Audio** | 8.4B| 33.94| 23.30| 32.99| 9.09| 33.03| 26.83 | 33.33 | 30.40|\n| **Qwen2-Audio-Instruct**| 8.4B| 33.33| 24.27| 32.31| 9.09| 31.19| 30.49| 25.00 | 30.00|\n| **SALMONN**| 7B| 30.91| 29.61| 34.35| 9.09| 37.61| 28.05 | 37.50 | 32.80|\n| **SALMONN**| 13B| 30.30| 31.07| 34.69| 9.09| 34.86| 35.37| 41.67 | 33.20|\n| **Mellow** | 167M| 33.33| 26.70| 24.83| 18.18| 37.16| 32.93| 29.17 | 30.00|\n\nMellow performs comparably to state-of-the-art reasoning models in the literature while being only one-tenth their size. Specifically, it outperforms all models except SALMONN, remaining competitive with Qwen2 and showing strong performance on sound- and music questions."}, {"review_id": "csYKIzkvtH", "reviewer": "Reviewer_9EBg", "summary": "The paper introduces Mellow, a small (167M parameter) Audio-Language Model (ALM) designed specifically for reasoning tasks. Unlike prior work that relies on scaling model and data size, Mellow focuses on improving reasoning abilities through architectural and training innovations. It is trained on a new dataset, ReasonAQA, which blends real and synthetically generated audio-grounded question-answer pairs. Despite its small size and limited data (~152 hours of audio), Mellow achieves state-of-the-art performance among small ALMs and outperforms several much larger models on multiple benchmarks, including multimodal reasoning (MMAU), deductive reasoning (Audio Entailment), comparative reasoning (Audio Difference), and audio captioning. The paper also conducts extensive ablation studies, revealing the importance of language model pretraining, fine-tuning strategies, audio encoder quality, and targeted synthetic data in boosting reasoning performance without relying on large-scale data.", "questions": "* The comparison in Table 5 may not be entirely fair. If I understand correctly, the linear probe is clearly not a strong baseline, and the modeling capacity of a language model is naturally expected to perform better.\n* The paper emphasizes the high quality of ReasonAQA and compares it with OpenAQA by analyzing its shortcomings. However, it does not clearly explain how the data generation process for ReasonAQA ensures such high quality. Is controlling the prompts alone truly sufficient to produce significantly better data?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "* Strengths:\n    * The structure of this paper is fairly clear, and the experiments are thorough. Despite the relatively small model size, it still achieves good performance.\n    * The paper conducts ablation studies from multiple perspectives, providing valuable insights for researchers who are new to the field.\n\n* Weaknesses:\n    * My main concern is that the paper lacks sufficient novelty. The methods used have already become relatively common practices in the field. Furthermore, the ablation studies do not introduce particularly novel analytical perspectives, and many of the conclusions drawn are largely aligned with what is already widely recognized by researchers. This paper feels more like a detailed experimental report rather than a research paper.\n    * The paper attempts to justify the development of SALM from the perspective of applications for edge devices. However, it lacks analysis specific to resource-constrained scenarios. As a result, it fails to demonstrate the advantages of the proposed method in such contexts—for example, through comparisons with alternative solutions such as distilling large models into smaller ones or directly quantizing large models.\n    * The fairness of the experimental comparisons in the paper is questionable. Although Mellow shows strong performance on the benchmarks tested, it seems more like an indication that Mellow may be overfitted to these specific tasks rather than a demonstration of general effectiveness. Perhaps showcasing the model’s performance on entirely unseen tasks would better demonstrate the generalization ability of the proposed approach.", "was_revised": true, "final_justification": "Through rebuttal, the authors further elaborated on the performance comparison of Mellow against other models on the MMAU benchmark and clarified the data collection procedure for training. However, I remain concerned about the novelty of the proposed method and its actual applicability in resource-constrained scenarios. I believe that if the paper had introduced some innovations in model architecture design or training strategies, it would have been a strong candidate for acceptance. Therefore, I decide to maintain my initial score.", "rebuttal": "We thank the reviewer for recognizing our contributions and providing constructive feedback. We address every question and hope that our response resolves your concerns. Any follow-up questions are welcome.\n\n**My main concern is that the paper lacks sufficient novelty ... a research paper.**\\\nWe thank the reviewer for the comment. To provide context from the literature, one of the first ALMs released was Pengi, published at NeurIPS 2023. Pengi was a small audio-language model that achieved state-of-the-art performance at the time. In subsequent years, research in the field primarily focused on increasing the parameter count and training data for ALMs, leading to performance improvements and a shift from small to large audio-language models. However, the necessity of large ALMs has rarely been questioned. Fundamental questions -- such as what are the limits of small audio-language models? How far can we push their performance? What architectural changes improve small ALMs? Do these changes scale as more data is added? -- remain largely unexplored. We believe that answering these questions is essential to understanding both small and large ALMs and to building efficient small models. Therefore, we start with Pengi, the only small ALM in the literature, and subsequently improve upon it while keeping the model size comparable.\n\nWe only partially agree with the statement that “many of the conclusions drawn are largely aligned with what is already widely recognized by researchers.” Small Audio-Language Models (ALMs) have not been previously explored prior to this work. Although the mechanisms or architectures we use may be common, our findings are novel and not previously reported in the literature. For example, we found that larger, transformer-based projection layers -- commonly used in prior work -- only improve performance when the language model is frozen. In contrast, with full fine-tuning, simpler projection layers perform better. Our ablation studies reveal a clear performance trend for training the language model backbone: full fine-tuning > LoRA adaptation > frozen LM. This challenges the common practice of using large language models with LoRA adaptation (as done in LTU, Qwen, GAMA, etc.). Moreover, while transformer-based projection layers consistently outperform linear and non-linear projections when the language model is frozen (as in Pengi), this advantage completely disappears when the language model is trainable. In that case, simple linear or non-linear projections achieve comparable performance. This contradicts the widely held assumption that complex projection layers are universally beneficial. Overall, we believe this line of work -- focused on understanding and systematically studying small ALMs -- is as important as traditional notions of novelty that emphasize introducing new components. While some individual ablations may not be entirely unique, it is necessary to study them specifically in the context of small audio-language models before making definitive claims.\n\n**The paper attempts to justify the development of ... smaller ones or directly quantizing large models.**\\\nWe thank the reviewer for bringing up this valid point. We agree that A key motivation for SALM is enabling edge deployment and reducing inference costs. Scaling from 8B to 135M parameters improves tokens-per-second throughput, resulting in faster responses, support for lower-compute devices, and reduced cloud latency and costs. In the NLP domain, this approach has already proven viable with models such as Phi, SmolLM, and MobileLLM. For concrete numbers, we evaluated Mellow and existing ALMs on an Nvidia V100 GPU, where each model was tasked with captioning 100 randomly sampled audio files from the AudioCaps test set. Throughput was computed as the number of tokens produced per second. The results are summarized below:\n\n| Model            | Parameters  | Total Time (s) | Total Tokens | Throughput (tokens/second) |\n|------------------|-------------|----------------|--------------|--------------------|\n| Qwen-Audio Chat  | 8.4B        | 52.99          | 3,039        | 17.04              |\n| GAMA             | 7B          | 199.30         | 3,593        | 18.03              |\n| Audio-Flamingo   | 2.2B        | 93.70          | 1,254        | 13.38              |\n| Mellow           | 167M        | 148.40         | 5,224        | 35.21              |\n\nHigher throughput reflects better performance, with Mellow significantly faster than 7B models. Moreover, models like Qwen-Audio 2 cannot fit on a V100 GPU and require larger memory. We believe that pursuing this research direction will lead to performant yet efficient ALMs.\n\n**The fairness of the experimental comparisons ... ability of the proposed approach.**\\\nWe thank the reviewer for this question. We use the MMAU benchmark, which consists of 10k annotated Audio Question Answering (AQA) pairs spanning the speech, sound, and music domains. This benchmark evaluates a model’s capabilities across 27 distinct skills involving various reasoning tasks. Notably, Mellow is trained exclusively on audio files from AudioCaps and Clotho, whereas MMAU sources its audio from 10 different datasets, with no overlap with Mellow’s training data. This makes MMAU an ideal benchmark for evaluating Mellow’s reasoning performance in an out-of-distribution (OOD) setting. However, we agree with the reviewer that it is still possible to overfit to MMAU.\n\nTo address this concern, we additionally benchmark Mellow on the MMAR benchmark. This benchmark consists of 1,000 examples spanning sound, music, speech, and their mixtures, and it focuses on multi-step, deep reasoning beyond surface-level understanding. Moreover, MMAR was released after Mellow, making it a completely independent benchmark with no possibility of overfitting. The results are shown below:\n| Models| Size| Sound (%) | Music (%) | Speech (%) | Sound-Music (%) | Sound-Speech (%) | Music-Speech (%) | Sound-Music-Speech (%) | Avg (%) |\n|---|---|---|---|---|----|---|---|---|---|\n| **Audio Flamingo** | 2.2B   | 32.73 | 21.84 | 24.83 | 18.18 | 30.28  | 24.39 | 25.00 | 26.60|\n| **Audio Flamingo 2**| 1.0B| 20.61| 20.39| 24.15| 27.27| 23.85| 26.83| 25.00| 23.00|\n| **Audio Flamingo 2**| 2.5B| 26.67| 20.87| 22.79| 9.09| 22.94| 23.17| 20.83| 22.90|\n| **Audio Flamingo 2** | 4.7B  | 24.85 | 17.48 | 20.75 | 18.18 | 26.61| 23.17| 8.33 | 21.90|\n| **LTU** | 7B| 19.39| 19.90| 13.95| 18.18| 24.77| 21.95| 16.67 | 19.20|\n| **LTU-AS** | 7B| 20.00| 14.08| 19.05| 9.09| 20.64| 28.05| 12.50 | 19.00|\n| **MusiLingo** | 7B| 9.09| 7.28| 4.08| 9.09| 6.88| 7.32| 8.33 | 6.6 |\n| **MU-LLaMA**| 7B| 13.94| 13.59| 14.97| 9.09| 12.39| 14.63| 16.67 | 13.90|\n| **GAMA**| 7B| 29.09| 24.27| 27.89| 27.27| 24.77| 28.05| 20.83 | 26.50|\n| **GAMA-IT** | 7B| 22.42| 16.02| 12.24| 36.36| 22.48| 14.63| 12.50  | 17.40|\n| **Qwen-Audio-Chat**| 8.4B| 27.88| 20.39| 22.11| 9.09| 25.23| 25.61| 20.83 | 23.50|\n| **Qwen2-Audio** | 8.4B| 33.94| 23.30| 32.99| 9.09| 33.03| 26.83 | 33.33 | 30.40|\n| **Qwen2-Audio-Instruct**| 8.4B| 33.33| 24.27| 32.31| 9.09| 31.19| 30.49| 25.00 | 30.00|\n| **SALMONN**| 7B| 30.91| 29.61| 34.35| 9.09| 37.61| 28.05 | 37.50 | 32.80|\n| **SALMONN**| 13B| 30.30| 31.07| 34.69| 9.09| 34.86| 35.37| 41.67 | 33.20|\n| **Mellow** | 167M| 33.33| 26.70| 24.83| 18.18| 37.16| 32.93| 29.17 | 30.00|\n\nMellow performs comparably to state-of-the-art reasoning models in the literature while being only one-tenth their size. Specifically, it outperforms all models except SALMONN, remaining competitive with Qwen2 and demonstrating strong performance on sound- and music-based questions despite having only 167M parameters.\n\n**The comparison in Table 5 may not be entirely fair ... expected to perform better.**\\\nWe thank the reviewer for the question. We use linear-probe performance because it is the standard fine-tuning procedure used in the Audio Entailment paper (AAAI 25). We followed the same method to ensure comparability.\n\n**The paper emphasizes the high quality of ReasonAQA and ... sufficient to produce significantly better data?**\\\nWe thank the reviewer for the clarifying question. The full data generation process is described in Appendix Sections C and D; we provide a concise summary here. Below are the some steps we took to ensure high-quality synthetic data generation: \n\n(1) Reduced Question Count: We deliberately reduced the number of generated questions per caption from 10 (as in OpenAQA) to 5, significantly improving question quality. As noted in Section D.1, requiring 10 questions per caption in OpenAQA often led to illogical or poorly structured questions. (2) Varied Question Formats: We generated both detailed and multiple-choice (MCQ) questions, each serving different learning objectives. MCQs enforce precise answer selection and reasoning about incorrect options, while detailed questions encourage rich, descriptive responses. (3) Explicit Audio Grounding: We added specific instructions such as “you are listening to the audio instead of reading the captions” and “generate question-answer pairs about what you just heard” (Figure 5). This discourages world-knowledge-only questions that can be answered without audio (e.g., “Q: What kind of sound is sizzling? A: A hissing sound when frying food”), a common issue observed in OpenAQA (Section D.1, Table 12). (4) Acoustic Property Emphasis: We instructed the model to explicitly use words related to acoustic properties -- semantic relations, spectro-temporal characteristics, frequency, loudness, duration, materials, interactions, and sound sources. We also used few-shot prompting with audio-focused examples.\n\nEvidence of the improved synthetic data quality includes vocabulary analysis (Figure 6), showing richer, more audio-specific terminology, and comparative examples (Table 13), demonstrating more grounded and detailed responses."}, {"review_id": "Ps7qHApr7v", "reviewer": "Reviewer_Ae2v", "summary": "This paper introduces Mellow, a small audio-language model (ALM) designed for reasoning tasks. Mellow achieves competitive or superior performance compared to many larger models, including some with over 50× the parameters and training data. The authors also release ReasonAQA, a new dataset combining synthetic and existing data to support audio-grounded reasoning. Extensive evaluations are conducted across multiple tasks—deductive reasoning, comparative reasoning, audio captioning, and multimodal understanding—along with thorough ablation studies.", "questions": "NA", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Timely and Relevant Contribution: Addresses an underexplored but important niche: reasoning with small ALMs, which is valuable for edge devices.\n\n- Strong Empirical Results: Mellow outperforms all other small ALMs and even some large ALMs (e.g., Qwen-Audio) on reasoning tasks like MMAU and Audio Entailment.\n\n- Novel Dataset (ReasonAQA): The construction of ReasonAQA demonstrates thoughtful design and use of LLMs for QA generation grounded in audio.\n\n- Comprehensive Evaluation: Covers a wide spectrum of reasoning capabilities (deductive, comparative, binary QA, captioning), including OOD evaluation.\n\n- Thorough Ablation Studies: Provides valuable insights on design choices (projection layers, LM architecture, audio encoder) and training strategies (fine-tuning vs. LoRA, etc.).\n\n- Resource-Efficient: Only 167M parameters and trained on ~152 hours of audio, making it suitable for practical deployment.\n\nWeakness:\n\n- Speech Reasoning Gap: Mellow performs poorly on speech-based tasks due to lack of speech data in training. This limits generalizability.\n\n- Synthetic Data Dominance: 70% of ReasonAQA is synthetic; concerns may arise around the diversity and generalizability of such data.\n\n- Comparison with SOTA Models: While results are impressive, the evaluation would benefit from comparison with recent speech-capable models on more speech-focused tasks.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing our contribution and novelty! We hope that our response resolves your concerns. Any follow-up questions are welcome.\n\n**Mellow performs poorly on speech-based tasks ... in training. This limits generalizability.** \\\nWe thank the reviewer for requesting clarification. Mellow is not trained on ASR data and therefore cannot perform speech recognition tasks. As a result, its ability to understand conversation-related information is limited. However, this is consistent with prior ALM literature, where models have focused primarily on audio and music understanding rather than speech content. For example, Pengi (NeurIPS 2023), LTU (ICLR 2024), and GAMA (EMNLP 2024) were all trained solely on audio and music data and thus exhibit similar limitations in speech understanding. While speech-based tasks are important, our current work is aligned with the audio-focused research direction and provides sufficient evidence for the claims made.\n\n**70% of ReasonAQA is synthetic; concerns may ... and generalizability of such data**\\\nWe appreciate this observation and emphasize that we took deliberate steps to ensure the quality and diversity of synthetic data. Specifically, we: (1) reduced the number of generated questions per caption to improve quality; (2) generated both detailed and MCQ formats to cover varied reasoning skills; (3) explicitly enforced audio grounding in prompts to avoid world-knowledge-only questions; and (4) instructed models to incorporate acoustic properties for richer audio-specific reasoning. Evidence of improved quality includes vocabulary analysis and grounded responses (Appendix Figures 6 and Table 13). Furthermore, 30% of ReasonAQA consists of curated reasoning datasets from the literature, ensuring diversity. The effectiveness of this dataset is shown by strong OOD performance on MMAU and MMAR benchmarks, therefore showing good generalization.\n\n**While results are impressive, the evaluation would benefit ... on more speech-focused tasks**\\\nWe acknowledge this point. Our primary focus is on non-speech audio reasoning, consistent with prior ALMs, and our evaluations therefore emphasize sound and music reasoning benchmarks. Nonetheless, following the reviewer's suggestion, we additionally fine-tuned QwenAudio -- a speech-capable model -- on ReasonAQA and report results below:\n| Experiment | MMAU performance | \n|---|---|\n| Pengi | 30.19 |\n| QwenAudio - Zero Shot | 41.86\n| QwenAudio - LoRA | 45.90 |\n| QwenAudio - full finetune| 43.72 |\n\nEven after fine-tuning, Mellow remains competitive and, in many cases, outperforms QwenAudio despite being over 50× smaller. Evaluating against speech-focused reasoning benchmarks is a promising direction for future work as we explore adding speech reasoning capabilities to Mellow."}]

Author rebuttal:

Meta-review: This work reverses the trend in audio-LMs, by proposing a "small" (~170M params, compared to 8B in sota Qwen2) model which is performant comparable to sota models. This is achieved using: 
(i) a new dataset comprising "reasoning" related question/answers obtained from LLMs on audio-text pairs from audio captioning datasets.
(ii) a number of training techniques, e.g., full fine-tuning instead of lora, transformer-based projection layers, better lm, representation learning using ssl. 

All the reviewers appreciated the clear presentation, performant small ALMs, and thorough ablations.

Concerns were raised regarding: (i) lacking speech abilities, (ii) being task or data specific.
However, the authors fine-tuned sota models (e.g., Qwen) on the proposed dataset and showed they still underperform the proposed method, and, also showed good performance on new tasks (not included in training), and new datasets (e.g., MMAR released after the proposed method).

Given the above, this work may be valuable to the community to build on small models, especially as the authors promise to share their models and code. Hence, I am inclined to recommend this work for acceptance.
Decision: Accept (poster)