MATS: An Audio Language Model under Text-only Supervision
Authors: Wen Wang
RuiBing Hou
Hong Chang
Shiguang Shan
Xilin Chen
Venue: ICML 2025 poster (2025)
Keywords: Large Audio-Language Models
Large Language Models
Audio-Language Contrastive Models
Primary area: deep_learning->large_language_models
TL;DR: We propose MATS, an audio-language multimodal LLM designed to handle Multiple Audio task using solely Text-only Supervision.

Abstract: Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. 
However, the training of LALMs demands a large corpus of audio-language pairs, which requires substantial costs in both data collection and training resources. In this paper, we propose **MATS**, an audio-language multimodal LLM designed to handle **M**ultiple **A**udio task using solely **T**ext-only **S**upervision. By leveraging pre-trained audio-language alignment models such as CLAP, we develop a text-only training strategy that projects the shared  audio-language latent space into LLM latent space, endowing the LLM with audio comprehension capabilities without relying on audio data during training. To further bridge the modality gap between audio and language embeddings within CLAP, we propose the **S**trongly-rel**a**ted **n**oisy **t**ext with **a**udio (**Santa**) mechanism. Santa maps audio embeddings into CLAP language embedding space while preserving essential information from the audio input. Extensive experiments demonstrate that MATS, despite being trained exclusively on text data, achieves competitive performance compared to recent LALMs trained on large-scale audio-language pairs. The code is publicly available in [https://github.com/wangwen-banban/MATS](https://github.com/wangwen-banban/MATS)

MATS: An Audio Language Model under Text-only Supervision
Wen Wang's Ruibing Hou"
Hong Chang"
Shiguang Shan 12 Xilin Chen
Abstract
Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. However, the train- ing of LALMs demands a large corpus of audio- language pairs, which requires substantial costs in both data collection and training resources. In this paper, we propose MATS, an timodal LLM designed to handle Multiple Audio task using solely Text-only Supervision. By leveraging pre-trained audio-language alignment models such as CLAP, we develop a text-only training strategy that projects the shared audio- language latent space into LLM latent space, en- dowing the LLM with audio comprehension ca- pabilities without relying on audio data during training. To further bridge the modality gap be- tween audio and language embeddings within CLAP,, we propose the Strongly-related noisy text with audio (Santa) mechanism. Santa maps au- dio embeddings into CLAP language embedding space while preserving essential information from the audio input. Extensive experiments demon- strate that MATS, despite being trained exclu- sively on text data, achieves competitive perfor- mance compared to recent LALMSs trained on large-scale audio-language pairs. The code is publicly available in https//gitnub.com/waageenn banban/MATS.
SCAP
Nists
1116
1000
152506
DCASE
V00
Accundey
AutaCaps
14
HaioCape
Acousay
mul-
Figure I. Performance of MATS compared to the previous SOTA (Chu al., 2023; Kong al., 2024; Deshmukh et al., 2023) under zero-shot setting for close-ended audio tasks.
of audio crucial role in understanding the physical world. Audio, encompassing sound, music, and other au- ditory elements, plays a crucial role in enabling intelligent agents to assist humans and navigate the complexities of the real world. Consequently, developing models capable of effectively interpreting these audio modalities is vital for advancing munnan- interaction and fostering deeper understanding of our surroundings.
In the field of audio-language multimodal learning, con- trastive learning models, such as Contrastive Language- Audio Pretraining (CLAP) (Elizalde et al., 2023), have demonstrated remarkable zero-shot abilities across various audio tasks. However, these models primarily specialize in audio discriminative tasks, lacking decoder support open-ended audio question-answering (QA) capabilities. To overcome this limitation, recent studies have integrated
1. Introduction
LLMs as the text generator within multimodal framework, capitalizing on LLMs' inherent instruction following and generating capabilities. A pioneering effort, Pengi (Desh- mukh al., 2023), integrates CLAP audio encoder and GPT2 (Radford et al., 2019), employing transformer pro- jection module to align audio embedding with LLMs' text embedding space. Building on this foundation, recent mod- els (Gong et al., 2024: Tang et al., 2024; Kong et al., 2024; Ghosh et al., 2024) utilize advanced audio encoder and more powerful LLMs, constructing advanced LALMs to address increasingly complex audio relevant tasks.
Recent advancements multimodal models for audio pro- cessing have progressed rapidly, driven by the recognition
'Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China University of Chinese Academy of Sciences, China. Correspondence to: Ruibing Hou <houruib- ing mng.cctac.cn>>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MATS: An Audio Language Model under Text-only Supervision
However, training large audio-language model that ef- fectively bridges the entirely decoupled audio embedding space and LLM text embedding space is challenging. Pre- vious models (Gong et al., 2024; Tang et al., 2024; Kong et al., 2024; Ghosh et al., 2024) rely heavily on extensive audio-language QA pairs for training. This requires anno- tators to carefully listen to each audio and then construct corresponding questions and answers, labor-intenssie and time-consuming process. As a result, the cost of data col- lection becomes prohibitively high, leading to significant resource demands for training LALLM.
propose a modal-transfer method that integrates noise in- jection during training and employs Strongly-related noisy text with audio (Santa) during inference. And the Santa mechanism employs a k- basan memory alongside a balancing strategy. This design effectively balances audio embeddings and semantically strongly related augmented language embeddings, effectively mitigating the modality gap and enhancing MATS's s generalization ability.
We conduct comprehensive evaluation of our model on both close-ended and open-ended tasks. Remarkably, de- spite being trained on text-only data, MATS demonstrates performance comparable to, and oven surpassing, some LALMs trained on extensive audio-language pairs. For close-ended tasks, MATS demonstrates significant zero-shot performance improvements over previous sfate-of-the-an (SOTAs) (Deshmukh et al., 2023; Tang et al., 2024; Doh et al., 2023), achieving gains of 12%, 7%, and 9.9% on DCASE (Mesaros et al., 2017), VGG (Chen et al., 2020), and MusicCaps (Agostinelli et al., 2023) benchmarks. re- spectively, as shown Figure 1. For open-ended tasks, MATS (7B) achieves the second-best performance on MMAU benchmark (Sakshi al., 2024), surpassing SALMONN (13B) (Tang al., 2024) by 11.7% with significantly smaller model size.
To reduce resource demands, researchers have proposed text-only audio captioning frameworks (Deshmukh et al., 2024; Kouzelis & Katsouros, 2023; Zhang et al., 2024; Li et al., 2024) which generate audio captions without re- lying on costly audio- audio-language pairs for training. These frameworks leverage CLAPP, which have established well- aligned audio-language embedding space, enabling LLMS to interpret audio embedding by linking CLAP language em- bedding space with LLM semantic space. However, these methods have certain limitations. First, they are narrowly tailored to specific audio type (sound) and constrained to single audio captioning task, limiting their applicability to boarder audio open-ended ceesstion-answrinng scenarios. Second, to bridge the audio-language modality gap within CLAP, these methods either rely solely on noise injection during training or exclusively utilize a memory bank during inference. However, using only noise injection is highly random and sensitive, while relying solely on memory bank during inference not only sacrifices audio embedding information but also increases inference time, ultimately compromising overall effectiveness.
2. Related work
Audio Language Models. In the field of audio-language models, CLAP models (Wu et al., 2023; Elizalde et al., 2023) have demonstrated remarkable capabilities in audio discriminative tasks. However, the lack of decoder limits their applicability in open-ended QA scenarios. Recently, with the rapid advancements in LLMs, researchers have started integrating audio understanding into LLMs. For ex- ample, Pengi (Deshmukh et al., 2023) combines the CLAP audio encoder with GPT2 (Radford et al., 2019), and em- ploys a transformer-based mapper for multimodal fusion, achieving strong performance on close-ended tasks. Sim- ilarly, LTU (Gong et al., 2024) incorporates a more ad- vanced LLM, LLaMA (Touvron et al., 2023), and demon- strates emerging audio comprehension and reasoning abil- ities. SALMONN (Tang et al., 2024) utilizes a dual audio encoder consisting of a Whisper speech encoder model (Rad- ford al., 2023) and a BEATs (Chen et al., 2023) audio encoder to handle speech and non-speech audio tasks effec- tively. Other studies have expanded instruction fine-tuning data and explored more advanced audio encoder architec- ture, such as GAMA (Ghosh et al., 2024), Qwen2-Audio (Chu et al., 2024), yielding commendable performance.
In this paper, we propose MATS, an audio-language mul- timodal LLM designed to address Multiple Audio tasks, encompassing both close-ended and open-ended scenarios, using solely Text-only Supervision. MATS builds upon re- cent advancements in LALMs (Deshmukh al., 2023; Tang et al., 2024), incorporating two key enhancements. First, recognizing that the joint audio-language embedding space of pre-trained CLAP is shared across both modalities, the language embeddings extracted by CLAP language encoder can serve as effective representations of audio semantics. During training, MATS integrates CLAP language encoder, a Mapper and a LLM, where the CLAP language encoder extracts cross-moot modality embeddings of audio descriptions within the shared audioge-audio embedding space. During inference, the LLM can generate responses based on audio embedding derived from input audios, processed through CLAP audio encoder and the Mapper. Second, we derive the generalization error bound for text-only supervised mul- timodal models, revealing that the inherent modality gap in contrastive audio-languae representation learning can amplify the generalization error bound. To this end, we
However, these LALMS heavily rely on large-scale audio- language pairs for training, which pose significant chal- lenges in term of data collection and training costs. Differ- ently, we adopt highly cost-efficient approach for training
2

---

MATS: An Audio Language Model under Text-only Supervision
LALMs using text-only data, which substantially reduces the data collection and training overhead while maintaining comparable performance.
Test reepeee
Noai--ranie Melsal
fur
Etuer
Text-only Supervised Multimodal LLMs. To alleviate resource demands, researchers have proposed zero-shot cap- tioning frameworks, aimed at generating image/audio cap- tions through text-only training. In the visual domain, large- scale pre-trained contrastive models like CLIP (Radford et al., 2021) align images and language into shared vision- language embedding space. Building on CLIP, CapDec (Nukrai et al., 2022) trains a decoder to reconstruct text from its corresponding CLIP language embedding, which is then used to decode CLIP image embeddings at inference. To mitigate the vision-language modality gap, CapDec injects noise into language embedding during training. Differently, DeCap (Li et al., 2023) leverages memory to store CLIP language embeddings, which is subsequently used to project visual embedding into CLIP language embedding space at inference. In the audio domain, models such as NoAudio- Captioning (Deshmukh et al., 2024), WSAC (Kouzelis & Katsouros, 2023), PromptAAC (Zhang et al., 2024), and DRCap (Li et al., 2024) adopt similar strategies, replacing CLIP with CLAP develop zero-shot audio captioning frameworks. However, these works focus on audio caption- ing, lacking the flexibility to handle a diverse range of audio tasks sinultaneously, and they are limited to processing only a single type of audio (sound).
Mappor fowe
Instruction
1 Santa
Noiso lnjatton
Santa forrse
Nakso lejotion
TRAIN
TEST
Tout Enookr E
Audio Enoxla
Text eabed Fy(T)
Augassttdd crubed 5
+
Audio cabeel 5264) Saats
Audio Captise
- -
Augurared andis anhted Frsae(4)
TBAIN
TEST
X
Guandan Nake
Figure 2. The architecture of proposed MATS.
text response.
Train Phase. During training, only text data is utilized, Formally, the input text T, which provides the descriptions of audio contents, is encoded by CLAP language encoder Er. To mitigate the audio-language modality gap, zero- mean Gaussian noise is added to the extracted language embedding m (T). The augmented language embedding is subsequently projected through the mapper Smmp to gen- erate embeddings within the LLM latent space. These em- beddings are concatenated with the instruction embeddings Eisttr extracted by LLM's word embedding layer, forming the prefix Strain. which serves as the input to the LLM for autoregressive learning. The process is formulated as:
Strain concat Suap (Er (T) + c), Emmu]
(1)
3. Methodology
where noise e ~ NN o) with variance o. Given the prefix Strain and corresponding text response R, the LLM pre- dicts the probability distribution of potential next token at each step, with the Cross-Entrope loss function used for optimization:
In this section, we introduce MATS, an audio-language multimodal LLM designed to tackle multiple audio tasks using text-only supervision. We begin by presenting the formulation of MATS in Section 1. followed by an outline of its overall framework Section 3.2. Section 3.3 provides the theoretical foundation, demonstrating that reducing the feature distribution gap between the CLAP's audio and lan- guage modalities can effectively reduce the generalization error of MATS. Section 3.4 presents the proposed Santa mechanism to bridge the audio-language modality gap. Fi- nally, the training pipeline is detailed in Section 3.5.
L c= CE (JIIM (Strain: REt) R²),
(2)
where CE denotes the cross-entropy function.
Inference Phase. At inference, the input data includes both audio files and textual instructions. The key struc- ture difference is that the augmented language embedding (T) + is replaced by audio embedding (A),, encoded by the audio encoder of CLAP. Notably, the Santa mecha- nism is then applied the audio embedding to mitigate the audio audio-language modality gap. Finally, the LLM predicts the responses Rpred in an autoregressive manner, as:
3.1. Formulations
As shown in Figure 2, MATS consists of a language en- coder er and an audio encoder ε from CLAP (Elizalde et al., 2023). During training, the language encoder E pro- cesses text data T, while during inference, the audio encoder EA processes audio data A. To bridge the audio languaee modality gap, the noise injection is used during training and the Santa mechanism fsanta is employed inference. Then, a transformer- based mapper Smap serves as the connection module, integrating its output sequence with the text instruc- tion prompt, which is then fed into the LLM to generate the
Stest = concat [map (fsanta (EA (A))) Emnst] (3)
Rpred = fU. (Stest).
3.2. Model Architecture
The architecture of the MATS model is depicted in Figure 2.
3

---

MATS: An Audio Language Model under Text-only Supervision
CLAP Encoder. To obtain an aligned audio-language embedding space, we adpot CLAP (Elizalde et al., 2023), which integrates the HTSAT (Chen et al., 2022) audio en- coder ΣA and the GPT2 (Radford et al., 2019) language encoder Er. CLAP is trained to connect language and audio by corresponding encoder and bring them into a joint multi- modal space using contrastive learning. Both encoders are kept frozen throughout the entire process. Notably, during training. the language encoder Er is utilized, whereas the audio encoder A is employed during inference.
T by the CLAP language encoder, and 2 stands for the embedding space induced from A by the CLAP audio en- coder. Additionally, let h denote the prediction function mapping from the CLAP embedding space to output space, Le. h 20/20 + y
To simplify the analysis, we focus on a specific audio event classification task and disregard the influence of instructions. In this case, we are given a training set consisting of text- label paired examples, D'r = {2', y 3 where (2), 31) ~ Pr (z',y) Our goal is to learn a target model h 2 + y by fitting on Di.. with the smallest generalization risk on the test distribution P.A (2" y) where (2", y) denotes audio- label pairs. Formally, the empirical error on training set and the generalization risk on test set are computed as:
Modality Transfer. To mitigate the audio-language modal- ity gap within CLAP and enhance the model's generaliza- tion, we add zero-meat Guassian noise to the language em- bedding during training as &r (7) + 6, where ~ .V(0, The variance a is determined by calculating the infinity norm between audio and language embeddings over set of 30 randomly selected samples following (Deshmukh et al., 2024). During inference, the augmented language embed- ding &r (T) + is replaced by the audio embedding EΛ (A), and the Santa mechanism is applied to the audio embedding to further reduce the audio-language modality gap within CLAP, formulating as /Santa (EA (A)). The Santa mecha- nism is detailed in Section 3.4.
N
Rer (h) = N
1
Σ
(21) - Vil.
(4)
Rue (h) = [E x))pp( w) // (2*) - yl.
where |-| is L1 norm. With these definitions, we can derive the generalization risk bound of the text-only supervised audio model, leading to following theorem.
The mapper module consists of a 8- layer Transformer (Vaswani, 2017) followed by linear layer. Specifically, we introduce a fixed set of learnable query embeddings, which are concatenated with the em- beddings produced by the CLAP encoder and then fed into the Transformer. Through the self-attention mechanism in transformer, these queries interact with the frozen CLAP embeddings, enabling effective integration of their informa- tion. Finally, the output query embeddings are projected via a linear layer to align with the dimensionality of the L.LM's text embeddings.
Mapper Module.
Theorem 3.1. Let H. be a hypothesis space of Natarajan- dimension d. For classification with V classes, let Der be text-only training set drawn from distribution pT (z',y). As for test set with audio files draw from distribution PAA (2", y). assuming that the class distribution of training and test data is consistent, for any target classifier l E H. δ E (0. 1), with probability at least 1 - 8:
Rie (h) S Rer (h)
+ max (discz, (pa (z"10) 1P PT (z'lu))
(5)
Large Language Model. To generate textual responses, we utilize a pretrained autoregressive causal language model. In this study, we evaluate two large language models: the smaller version employs GPT2 with 125M parameter, while the larger version leverages the LLaMA-7B model fine-tuned with Vicuna instruction- following capabilities. These two LLMs are selected to balance computational effi- ciency with performance, enabling a comparative analysis of lightweight and advanced language generation capabilities.
+VN
2dlog V2NV + log
N
Here, disc denotes the Discrepancy Distance (Definition in (Mansour et al., 2009)).
The proof is provided in Appendix A. Theorem 3.1 reveals that the generalization risk can be bounded by the sum of three components: (1) Empirical Risk: Rey (h), represent- ing the empirical risk on training set Dtr. which can be reduced during training. (2) Modality Discrepancy: The second term denotes the discrepancy between CLAP au- dio embeddings and language embeddings. (3) Complexity Term: The last term is only relevant with the choice of the hypothesis space H. and magnitude of N. In general, this term vanishes when the training data is sufficiently large. These observations highlight a crucial factor in reducing the generalization error bound: minimizing the audio- langull modality gap within CLAP.
3.3. Theoretical Analysis on Generalization
In this subsection, we study the generalization error bound of the text-only supervised audio models. Formally, let A and T be input audio and corresponding textual descrip- tions space respectively, and y be output one-hot class space, where the one-hot vector E RVV is used to rep- resent the label and V denotes the LLM's vocabulary size. Similarly, z' stands for the embedding space induced from
4

---

MATS: An Audio Language Model under Text-only Supervision
resentational capabilities. Denote the cluster centers as {cx) we identify the most relevant cluster Selosest by
calculating the distance between the audio embedding z" and the cluster centers:
Merocry-taseed
Figure 3. t-SNE visualizations for various methods on 350 ran- domly selected language embeddings (blue) and paired audio em- beddings (red) from training dataset of MATS-Audh in Table S9. CLAP b) CLAP with Gaussian Noise (Deshmukh et al., 2024) CLAP with memory bank (Li et al., 2024) d) CLAP with Santa.
closest arg min || cx112.
(6)
Next, we select L augmented language embeddings from the cluster Sclosest. denoting as { [mflosest, ..... m mflooeat J. with the top L closest distances to the audio embedding. Finally, we compute weighted sum of these L augmented language embeddings and balance its contribution with the audio embedding. The weights of these language embeddings are derived by calculating the dot-product similarity between z" and each m mllosest, normalized via a softmax function. This overall Santa process is formalized
3.4. Santa Mechanism
According to theoretical analysis above, reducing Modality Discrepancy is critical for improving MATS's generaliza- tion. However, CLAP suffers from an inherent modality gap: As shown in Figure 3 a). CLAP's audio and language em- beddings occupy entirely separate regions. To address this issue, (Deshmukh et al., 2024) injects noise into CLAP's lan- guage embeddings during training. However, relying solely on noise injection is highly stochastic and sensitive, limit- ing its robustness. Another work (Li al., 2024) utilizes memory bank that integrates similar language embeddings to represent audio embeddings during inference. However, this approach completely discard original audio embeddings, resulting in a significant loss of audio information. In this work, we propose modality transfer, which combines noise injection during training with the Santa mechanism during inference. The Santa mechanism utilizes a k-means-baas based memory alongside a balancing strategy. This design inte- grates audio embeddings with semantically relevant aug- mented language embeddings, more effectively bridging the modality gap, as shown in Figure 3 d).
Sssanta (=") (1 -1)==11- 2)
wi * m
closest
((E-a)
exp
m
closest /T)
where, n1
プド= exp
((34)
mj moloorstt
closest
/T)
where is temperature hyperparameter and Á is a balane- ing hyperparameter.
3.5. Training Pipeline
AudioTIA-5M Dataset. MATS is designed to handle a wide range of audio tasks relying solely on text supervi- sion, which makes existing audio datasets unsuitable for direct use in its training. In this work, we construct a new dataset AudioTIA-5M, where each training sample is formatted as (text, instruct ion, answer) tuple. Audi OTTA-5M encompasses both close-ended and open- ended tasks, employing four audio-task templates in Ta- ble S1 I of Appendix. For the classification task (CLS), we leverage ChatGPT (Achiam et al., 2023) to generate a sub- stantial amount of textual data. Specifically, by utilizing ChatGPT's instructionnfolloviii and text generation capa- bilities, we create diverse set of audio descriptions (used as text) based given event labels (used as answer). Further details are provided in Appendix B.4. For the audio captioning task (CAP), we directly use audio cap- tion annotations from public datasets, including Clotho-v2 (Drossos et al., 2020), AudioCaps (Kim et al., 2019), Music- Caps (Agostinelli et al., 2023), WavCaps (Mei et al., 2024) and Macs -Martín-Morató -Moraóó & Mesaros, 2021). as text and answer. To further enhance model's understanding and reasoning capabilities for audio, incorporate two open- ended datasets: OpenAQA (Gong et al., 2024) and MusicQA (Liu et al., 2024a). In OpenAQA, we replace raw audio files with their audio captions as model inputs. For MusicQA, where music captions are unavailable, we generate corre- sponding captions using I.P-MusicCaps (Doh et al., 2023).
To obtain the CLAP language embedding space augmented with noise injection, we randomly select M texts from the text-only training set, denoted as {T and construct memory M = {mi where m; = E9 (T;) + e. At inference, the goal is to generate an answer for a given audio A and associated instruction. Specifically, given the audio embedding ' = Ea (A), we obtain its representation in language embedding space by combing =" with a weighted sum of semantically relevant embeddings in memory M. A straightforward strategy to obtain the weights of these language embeddings is to compute their similarity with the audio embedding. However, due to the limited repre- sentational power of individual language embedding, this strategy is prone to retrieving the texts with insufficient semantic relevance, thereby affecting the effectiveness of audio-language modality alignment.
M
To this end, we apply k-means clustering to M and obtain a clustered memory S = { where Sy stores the augmented language embeddings assigned to this cluster. By aggregating semantically similar augmented language embeddings, each cluster exhibits enhanced semantic rep-
5

---

MATS: An Audio Language Model under Text-only Supervision
Overall, AudioTIa 5M comprises two subsets: 1.5 M close-ended question subset, and a 3.8 M open-ended ques- tion subset. The detailed statistics of AudioTIA-55 5M are presented in Table S8 of Appendix.
with a different training set as outlined in Appendix B.2.
4.2. Main Results
We evaluate the close-ended and open-ended capabilities of MATS using 16 downstream tasks. Table S10 in Appendix summarizes all test benchmarks and corresponding evalua- tion metrics. Notably, for classification task, we adopt the text- eoxt-matching setup following (Gong et al., 2024), where the CLIP language encoder is used to encode both the answer set and model's predictions. The answer with the highest similarity to the prediction is selected as the final result.
Training. To maintain alignment between language and audio embeddings within the CLAP model while leveraging the extensive knowledge and generative capacities of LLM, we freeze both CLAP and LLM model, training only the mapper module and LLM's LoRA adapters (Hu et al., 2022). Our model is trained on AudioTIA 5M integrates both close-ended and open-ended tasks. The close-ended tasks equip the model with foundational audio perception capabilities, while the open-ended tasks enhance its advanced reasoning abilities. Training is conducted in an autoregressive manner using Equation 2.
5M dataset, which
4.2.1. CLOSE-ENDED RESULTS
Zero-shot Audio Classification. Since MATS performs audio classification solely based on GPT-generated descrip- tions, its performance is evaluated under zero-shot setting, Table 1 summarizes the comparative results across both sound and music types. As shown in Table 1, MATS achieves SOTA performance in the zero-shot setting on FSD50K, DCASE, TUT, VGG and BJO benchmarks, with improvements of 1%, 10%, 14%, 6% and 3%. Remarkably, MATS performs comparably with SALMONN (Tang et al., 2024) and Qwen-Audio Chat (Bai et al., 2023), both of which are trained on a significant amount of audio-language pairs. These results validate the efficacy of our approach for audio classification, highlighting the excellent audio perception capabilities of MATS in a zero-shot scenario.
4. Experiment
In this section, we conduct extensive experiments on both close-ended and open-ended tasks. The details about train- ing and test benchmarks are provided in Appendix B. Addi- tional extensive hyperparameter analysis and visualizations are provided in Appendix C.
4.1. Experimental Setup
MATS-GPT2. In the smaller version, we employ GPT2 with 125M parameters as the LLM. We use CLAP's audio encoder and language encoder. The audio is sampled at 44.1 kHz and converted into a log Mel spectrograms with 64 Mel bins, a hop size of 320 ms, and window size of 1024 ms. All audio files are randomly truncated to 7 seconds in length with CLAP audio encoder. The mapping module uses a 8-layer transformer with a prefix length of 40. Training conducted using the AdamW optimizer with learning rate of 5 x 10- a linear learning rate scheduler with 2000 warmup steps. The batch size is set to 128. For the LoRA configuration, we set the rank to 8, the scaling factor to 4, and the dropout rate to 0.1. MATS-GPT2 underwent about 25 hours of training over 90,000 iterations on 2 A100 GPUs The hyperparameters are configured as follows: a = 0.015, K- 100, L- 32, 7 0.1 and A = 0.3 (Equation 7).
Audio Captioning. Table 2 summarizes the comparative results for audio captioning tasks on AudioCaps, Clotho, and MusicCaps benchmarks. We can observe that: (I) Compared to text-only supervised audio captioning mod- els, which are specifically tailored for audio captioning and restricted to the Sound type, our model demonstrates en- hanced versatility by effectively addressing a broader spee- trum of audio-relevant tasks across both Sound and Music type. Despite its broader applicability, our model achieves superior CIDEr and SPIDEr scores on AudioCaps and a higher SPICE score on Clotho. The performance gains can be largely attributed to the proposed Santa mechanism, which effectively mitigates the audio-language modality gap within CLAP. (2) Compared to aupio-supervised models, MATS-LLaMA exhibits comparable performance despite being training only on text data. Also, MATS -PTT2 sur- passes the music captioning model I.P-MusicCaps (Doh et al., 2023) by 9.9% % on the ROUGH-L metric. These re- sults underscore the effective audio captioning capabilities of MATS under text-only supervision.
MATS-LLaMA. In the larger version, we employ the LLaMA-7B model as the LLM. We use AdamW optimizer with learning rate of x 10-5 cosine annealing learning rate scheduler with 3000 warmup steps. The batch size is set to 96. MATS-LLaMA underwent about 35 hours of training over 360,000 iterations. All other settings remain consistent with those of MATS-GPT2.
Simple AQA. The ClothoAQA benchmark primarily con- sists of responses limited to yes or no answers (evaluated using the B-ACC metric), framing the task largely as classi- fication problem. As shown in Table 1, MATS-LLaMA out- performs the recent model Qwen- (Chu et al., 2023)
MATS-Audio. We explore an audio version of MATS, supervised by audio. MATS-Audio consists of the CLAP audio encoder, the mapper module and LLaMA-7B. It em- ploys the same training configurations as MATA,
6

---

MATS: An Audio Language Model under Text-only Supervision
Table Comparision results on audio classification benchmarks and ClothoAQA benchmark. 25- and indicate weakly zero-shot and supervised seltings, respectively. Results without any annotation represent zero-shot setting. Note: the results of SALMONN and Qwen-Audio-Chal are from GAMA (Ghosh et al., 2024).
Audio Type
Sound
Music
ESC-50 (ACC)
FSDS0K (mAP)
DCASE (ACC)
TUT (ACC)
VGG (ACC)
US8K (ACC)
ClothoAQA (ACC|B-ACC)
GTZAN
BJO
Model
(ACC)
(ACC)
CLAP (Elizalde et al., 2023)
0.83
0.30
0.30
0,30
0.73
0.25
0.30
Pengi (Deshmukh et al., 2023)
0.92
0.47
0,33
0.35
0.72
0.65t
0.35
0.62
LTU (Gong et al., 2024)
0.832s:
0.461
0.46
0.33
0.50
0.70
LTUUAS (Gong et al., 2024)
0.812s.
0.50
GAMA (Ghosh et al., 2024)
0.83
0.48
0.38
0,22
0.52
0.721
0.14
0.70
Qwen-Audio (Chu al., 2023)
0.651
0.581
0.751
Audio Flamingo (Kong et al., 2024)
0.70
0.75
0.871
0.68
SALMONN (Tang et al., 2024)
0.16
0.22
0.18
0.08
0.23
0.10
0.25
Qwen-Audio Chat (Bai et al., 2023)
0.72
0.40
0.32
0.17
0.18
0.42
0.51
MATS-GPT2 (Ours)
0.86
0.48
0.43
0,49
0.29
0.67
0.56
0.66
0.58
0.70
MATS-LLaMA (Ours)
0.88
0.45
0.43
0,47
0.29
0.67
0.66
0.76
0.63
0.73
MATS-Audio
0.70
0.43
0.55
0.61
0.42
0.39
0.67!
0.761
0.29
0.46
Table Comparision results for audio captioning tasks on AudioCaps, Clotho and MusicCaps benchmarks.
AudioCaps SPICE
Clotho SPICE
MusicCaps
Method
CIDEr
SPIDEr
CIDEr
SPIDEr
ROUGHIL
BLUE4
Pengi (Deshmukh al., 2023)
0.752
0,182
0.467
0.416
0.126
0.271
LTU (Gong et al., 2024)
0.170
0.119
GAMA (Ghosh et al., 2024)
0.185
0.135
Audio Supervision (AS)
LTU-AS (Gong et al., 2024)
0.150
SALMONN (Tang el al., 2024)
0.485
21.5
Audio Flamingo (Kong et al., 2024)
0.502
0.465
I.P. JP-MusicCaps (Doh al., 2023)
13.0
0.7
NoAudioCaptioning (Deshmukh al., 2024)
0.697
0.178
0.437
0.379
0.132
0.256
Text Supervision (TS)
PromptAAC (Zhang et al., 2024)
0.644
0.156
0.400
0,403
0.119
0.261
DRCap (Li al., 2024)
0.718
0,186
0.452
0.438
0.133
0,285
MATS-GPT2 (Ours)
0.676
0.164
0.420
0.413
0.124
0.269
22.9
4.5
MATS-LLaMA (Ours)
0.735
0.171
0.453
0.431
0.134
0.282
18.7
3.2
AS
MATS-Audio
0.704
0.171
0.438
0.448
0.137
0.293
21.7
5.4
on ClothoAQA, achieving 8% higher ACC score. This result highlights the enhanced audio comprehensive capabilities of MATS-LLaMA after scaling up. However, MATS shows slightly lower performance on the B-ACC metric compared to Audio Flamingo (Kong et al., 2024). Notably, Audio Flamingo leverages audio supervision during training and employs a sliding window to extract audio features. While this enhances its performance, it also introduces signifi- cant computational overhead. In contrast, MATS-LLaMA operates under text-only supervision, offering a more com- putationally efficient alternative. This trade-off between efficiency and performance underscores the practical consid- erations in model design for real-world applications, where computational cost is often a critical factor.
are presented in Table 4 and Table (1) On AIR-Bench. under text-only supervision, MATS-LLaMA achieves performance comparable Qwen-Audio (Chu et al., 2023), ranking third-highest on Sound type and fourth-highest on Music type. We argue that CLAP struggles to extract fine- fine-grained audio information, such as the exact number of occurrences. This limitation makes MATS-LLaMA more prone to errors when responding to fine-grained questions on AIR-Bench. (2) For the MMAU, the QA format requires LALMs to select from set of provided options. We use the following prompt: (question) Select one option: {options} Respond with exactly one of the options above. Compared to AIR-Bench, this option-based format offers additional contextual cues, partially mitigating CLAPP' limitations in extracting detailed information. Consequently, our model outperforms Qwen-Audio-Chan (Bai et al., 2023) by about 3% in the sound type and ranks second only to the Qwen2-Audio- Instruiinntttttion (Chu et al., 2024) in other audio types. Notably, our model even surpasses larger-scale models like Aiien-Audio-Caa and SALMONN.
4.2.2. OPEN-ENDED RESULTS
To further evaluate our model's understanding and reason- ing capabilities in complex audio QA tasks, we conducted experiments on AIR-Bench Chat (Yang et al., 2024) and MMAU (Sakshi et al., 2024) benchmarks. The results
7

---

MATS: An Audio Language Model under Text-only Supervision
Table 3. The results of our model on the MMAU datasets across the sound and music categories.
Sound Test-mini
Music Test-mini
Avg
Method
Size
Test
Test
Test-mini
Test
Swee2-Audio-Instruccee (Chu al., 2024)
8.4B
55.0
45.9
51.0
53.3
49.2
52.5
Owen-Audio-Chal (Bai et al., 2023)
8.4B
55.3
56.7
44.0
40.9
43.1
41.9
SALMONN (Tang ct al., 2024)
13B
41.0
40.3
34.8
33.8
33.7
32.8
GAMA (Ghosh et al., 2024)
7B
41.4
45.4
32.3
30.8
30.9
31.8
AS
MuLLaMa (Liu al., 2024b)
7B
40.8
44.8
32.6
30.6
31.9
30.7
GAMA-IT (Ghosh al., 2024)
43.2
43.2
28.4
28.0
30.2
29.0
LTU-AS (Gong et al., 2024)
7B
23.4
25.0
9.1
10.5
17.7
18.9
LTU (Gong et al., 2024)
7B
22.5
25.9
9.7
12.8
16.9
18.5
Pengi (Deshmukh al., 2023)
323M
6.1
8.0
2.9
3.1
3,4
4.2
MATS-GPT2 (Ours)
370M
7.8
7.2
14.1
11.7
10.5
9.8
TS
MATS-LLaMA (Ours)
7B
52.3
59.8
44.9
42.9
43.3
44.5
AS
MATS-Audio
7B
51.7
57.4
43.7
41.5
42.2
43.8
Furthermore, compared MATS-Audio, MATS-LLaMA achieves comparable results open-ended tasks, further validating the effectiveness of the eext-super strategy for training LALMs.
for open-ended tasks.
MATS-LLaMA vs MATS-Audio. As shown in Ta- ble 1, MATS-LLAMA achieves performance comparable to MATS-Audio in the supervised setting. However, under the significantly outperforms MATS-Audio, highlighting the heavy reliance of audio- supervised models on audio-language pairs for achieving superior performance. Moreover, shown in Table 4 and Table 3, MATS- achieves performance par with MATSS in open-ended scenarios, further highlighting the cost-effectiveness and efficiency of our approach.
Table The results of our model the AIR-Bench Chat Bench- mark across the sound and music categories.
zero-shot setting,
Method
Sound
Music
Qwen2-Audio (Chu el al., 2024)
6.99
6.79
Qwen- OwennAAudo--e Audio-Turbo (Bai et al., 2023)
6.59
5,98
SALMONN (Tang et al., 2024)
6,28
5.95
Qwen-Audio (Chu et al., 2023)
6.95
5.52
Gemini-1.5-pp -ro (Team et al., 2024)
5,49
5,06
Santa. Table presents the ablation study our pro- posed Santa mechanism. The study evaluates the impact of excluding various components, including k-means (w/o K-means), memory bank (w/o MB), audio embedding (w/o AE) and Gaussian noise (w/o GN). As shown in Table 6, removing any component of Santa results in performance drop, highlighting their collective effectiveness. Notably, the absence of Gaussian noise (w/o GN) during training and the exclusion of audio embeddings (w/o AE) during inference lead to significant performance decline. These results emphasize the critical role of incorporating Gaussian noise during training and retaining audio embeddings during inference to achieve robust performance.
AS
BI.SP (Wang al., 2023)
5.55
5.08
Pandagpt (Su et al., 2023)
5.46
5.06
Next-gpt (Wu ul., 2024)
4.76
4.18
SpeechGPT (Zhang et al., 2023)
0.95
0.95
Macaw-LLan (Lyu et al., 2023)
1.01
0.91
MATS-GPT2 (Ours)
4.33
3.31
TS
MATS-LLaMA (Ours)
6.43
5.76
AS
MATS-Audio
6.25
5.77
4.3. Ablation
In this subsection, we conduct ablation studies validate the effectiveness of our method. Further analysis of the hy- perparameters in Equation (7), including K, L, M, T, X, and the number of mapper layers, is provided in Appendix C.2
Santa vs other modal-transfer mechanism. To demon- strate the superior effectiveness of the Santa mechanism, we compare Santa with other modal-transfer methods, in- cluding the memory-based mechanism DRCap (Li et al., 2024), the noise-based mechanism in NoAudioCaptioning (Deshmukh al., 2024) and PromptAAC (Zhang al., 2024). As shown in Table 5, Santa consistently outperforms the competing methods, validating its strength in bridging the audio languae modality gap. In DRCap, the original au- dio embeddings are completely discarded during inference, making its performance highly dependent on the alignment between the memory bank and the test benchmark. However, in multi task setting, the memory bank must accommo-
MATS-GPT2 vs MATS-LLaMA As shown in Table 1 and Table 2, MATS-GPT achieves performance comparable to MATS-LLaMA on close-ended tasks. However, as shown in Table and Table 3. MATS-GPT2 exhibits significantly weak performance on open-ended tasks. These results indi- cate that smaller LLM like GPT2 is sufficient for handling basic audio perception tasks, but it struggles with complex open-ended tasks that require advanced audio understanding and reasoning. In contrast, larger LLMs are better equipped to capture the depth and complexity necessary for audio reasoning and comprehension, making them indispensable
8

---

MATS: An Audio Language Model under Text-only Supervision
Table 5. Performance Comparison between Santa and Other Modality Transfer Methods Close-Ender and Open-Ended Benchmarks
Benchmark
ESC-50 ACC
AudioCaps CIDEr
AIRBench-Souni
AlRBench-Music
Metric
GPT4
GPT4
WAATS-NoAuioCaaphie
0.88
0.698
6.15
5.71
MATS- MATS-PPomptAAC
0.77
0,593
6.07
5.28
MATS-DRCap
0.84
0.619
5.83
5.29
MATS-LLaMA
0.88
0.735
6.43
5.76
date diverse range of benchmarks rather than single one, which introduces noise into the retrieval process and projects the audio embeddings into less relevant textual spaces, lead- ing to performance drop. No.AudioCatiioning on the other hand, employs a pure noise-injection strategy. While simple, this approach fails to effectively reduce the modality gap within CLAP, as analyzed detail in Appendix C.I, result- ing in limited performance gains. PromptAAC adopts an augmentation- approach that involves injecting noise and substituting similar language inputs. It retrieves audio events by matching audio embeddings with language embed- dings derived from 527 predefined audio labels in AudioSet. However, the limited variety of audio events restricts the diversity of the retrieved information, resulting in inferior performance compared Santa.
5. Conclusion
In this paper, we propose MATS, an audio-language mul- timodal LLM tuned solely on text data. Leveraging an efficient text-only training strategy, MATS establishes the semantic connection between the LLM and audio-language contrastive mode, substantially reducing data and train- ing costs while improving zero-shot performance. To ad- dress the modality gap issue, we further introduce modal- transfer strategy, Santa, which effectively balances audio embeddings and augmented language embeddings. Compre- hensive experiments show that MATS achieves performance comparable to, and in cases surpassing. LALMs trained on large-scale audio-language paired datasets.
Acknowledgements
The impact of Gaussian noise variance Table 7 inves- tigates the impact of the variance of injected noise during training. MATS achieves the best performance at a 0.015. We argue that a small variance fails to effectively mitigate the Ladio-language modality gap within CLAP, while an excessively large variance impairs the model's 's ability to capture useful input information.
This work is partially supported by National Natural Science Foundation of China (NSFC): 62376259 and 62306301, National Postdoctoral Program for Innovative Talents under Grant BX20220310.
Impact Statement
Table Ablation Study of Santa AudioCaps benchmark.
This paper presents work whose goal is to advance the field of Large Audio-Language Models. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Method
CIDEr
SPICE
SPIDEr
MATS-LLaMA
0.735
0.171
0.453
w/o K-means
0.703
0.171
0.437
w/o MB
0.698
0.170
0.434
References
w/o AE
0.343
0.114
0.229
w/o GN
0.384
0.121
0.253
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint urXiv:2303.08774 2023,
Agostinelli, A., Denk, I., Borsos, Z., Engel, J. H., Verzetti, M., Caillon, A., Huang, Q.. Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., and Frank, C. Н. Musiclm: Generating music from text. CoRR, abs/2301.11325. 2023.
0.65
0.60
0.55
0.50
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309, 16609, 2023.
0.45
0.40
0.00
0.01
0.02
0.03
0.04
0.05
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced
Table 7. The impact of a on AudioCaps henchmark.

---

MATS: An Audio Language Model under Text-only Supervision
emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4):377-390 2014.
Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 30:829-888 852, 2021.
Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 721-725. IEEE, 2020.
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A.. Lawrence, W., Moore, R. C.,, Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 776-780, 2017.
Chen, K., Du, X., Zhu, B., Ma, Z., King-KKikpatrick, T., and Dubnov, S. Hts-at: hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing, pp. 646-650. IEEE, 2022.
Ghosh, S., Kumar, S., Seth, A., Evuru, C. K. R., Tyagi, U., Sakshi, S., Nieto, O., Duraiswami, R., and Manocha, D. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, Miami, FL. USA, November 12-16, 2024, pp. 6288-6311 Association for Computational Linguistics, 2024.
Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z. and Wei, F. Beats: Audio pre-training with acous- tic tokenizers. In International Conference on Machine Learning, 23-29 July 2023, Honolulu, Hawaii, USA, vol- ume 202 of Proceedings of Machine Learning Research, pp. 1193.. PMLR, 2023.
Chu, Y., Xu, J., Zhou, X., Yang, Q.. Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models. arXiv preprint arXiv:2311.07919, 2023.
Gong, Y., Yu, J., and Glass, Vocalsound: dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing, pp. 151-155. IEEE, 2022.
Chu, Y., Xu, J., Yang, Q. Wei, H.,, Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., al. Qwen2-audio technical report. arXiv preprint arXiv:2407. 10759, 2024.
Gong, Y., Luo, H., Liu, H., Karlinsky, I.., and Glass, J. R. Listen, think, and understand. In The Twelfth Interna- tional Conference on Learning Representations, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Deshmukh, S., Elizalde, B., Singh, R., and Wang, H. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:18090- 18108, 2023.
Hu, E, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, Virtual Event, April 25-29, 2022. OpenReview.ne,, 2022.
Deshmukh, S., Elizalde, B., Emmanouilidou, D., Raj. B., Singh. R., and Wang, H. Training audio captioning mod- els without audio. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing, pp. 371-375 IEEE, 2024.
Kim, C. D., Kim, B., Lee, H., and Kim, G. AudioCaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pp. 119-132, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
Doh, S., Choi, K., Lee, J., and Nam, J. Lp-musiccaps: Llm-based pseudo music captioning. In Proceedings of the 24th International Society for Music Information Retrieval Conference, Milan, Italy, November 5-9, 2023, pp. 409-416, 2023.
Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In Forty-first International Conference on Machine Learn- ing. Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
Drossos, K., Lipping, S., and Virtanen, Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference Acoustics, Speech and Signal Processing, pp. 736-740. IEEE, 2020.
Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. Clap learning audio concepts from natural language su- pervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5. IEEE, 2023.
Kouzelis, T. and Katsouros, V. Weakly auto- mated audio captioning via text only training. arXiv preprint arXiv:2309. /2242, 2023.
10

---

MATS: An Audio Language Model under Text-only Supervision
Martín-Morató, I. and Mesaros, A. Diversity and bias in Li, W., Zhu, L.,, Wen, L.,, and Yang. Y. Decap: Decoding CLIP latents for zero-shot captioning via text-only train- ing. In The Eleventh International Conference Learn- ing Representations, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
audio captioning datasets. In Proceedings of the 6th Workshop Detection and Classification of Acoustic Scenes and Events 2021, Online, November 15-19, 2021, pp. 90-94, 2021.
Li, X., Chen, W., Ma, Z., Xu, X., Liang, Y., Zheng, Z., Kong, Q. and Chen, X. Drcap: Decoding clap latents with retrieval- -atmmeeel- generation for zero-shot audio captioning, arXiv preprint 24014:2410.04472. 2024.
Mci, X., Meng, C., Liu, H., Kong, Q. Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Waveaps: A chatgpt-assisted weakly-lahelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing. 32:3339- 3354., 2024. ISSN 2329-9304.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung. S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi-moda contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35: 17612-17625, 2022.
Mesaros, A.. Heittola, T., and Virtanen, T. Tut database for acoustic scene classification and sound event detection. In 2016 24th European Signal Processing Conference, pp. 1128-1122 132. IEEE, 2016.
Lipping, S., Sudarsanam, P., Drossos, K., and Virtanen, T. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference, pp. 1140-1144. IEEE, 2022.
Mesaros, A., Heittola, T., Diment, A., Elizalde, B., Shah, A., Vincent, E., Raj. B., and Virtanen, T. DCASE 2017 Challenge setup: Tasks, datasets and baseline system. In DCASE 2017 Workshop on Detection and Classification of Acoustic Scenes and Events, Munich, Germany, 2017.
Liu, S., Hussain, A. S., Sun, C., and Shan, Y. Music un- derstanding Ilama: Advancing text-to-music generation with question answering and captioning. In ICASSP 2024 2024 IEEE International Conference Acoustics, Speech and Signal Processing, pp. 280-290, 2024a.
Nukrai, D., Mokady, R., and Globerson, A. Text-only train- ing for image captioning using noise-injected CLIP. In Goldberg, Y., Kozareva, Z.,, and Zhang, Y. (eds.), Find- ings of the Association for Computational Linguisties: EMNLP 2022, Abu Dhabi, United Arab Emirates, Decem- ber 7-11, 2022, pp. 4055-4065 Association for Compu- tational Linguistics, 2022.
Liu, S., Hussain, A. S., Sun, C., and Shan, Y. Music un- derstanding Ilama: Advancing text-to-music generation with question answering and captioning. In ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing, pp. 286-290. IEEE, 2024b.
Piczak, K. J. Esc: Dataset for environmental sound classi- fication. In Proceedings of the 23rrd ACM international conference Multimedia, pp. 1015-1018, 2015.
Livingstone, S. R. and Russo, F. A. The ryerson audio- visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PLOS ONE, 136)),9016339, 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 1748-8763, PMLR, 2021.
Lyu, C., Wu, M., Wang, L., Huang, X., Liu, B., Du, Z., Shi, S., and Tu, Z. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.0900 2023.
Manco, I., Week, B., Doh, S., Won, M., Zhang. Y., Bog- danov, D., Wu, Y., Chen, K., Tovstogan, P., Benetos, E., et al. The song describer dataset: corpus of audio cap- tions for mussic-and-languaage evaluation. arXiv preprint arXiv:2311.10057 2023.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In International Conference on Machine Learning, pp. 28492-28518 PMLR, 2023.
Sakshi, S. Tyagi, U., Kumar, S., Seth, A., Selvakumar, R., Nieto, O.,, Duraiswami, R., Ghosh, S., and Manocha, D. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168 2024.
Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation: Learning bounds and algorithms. In The 22nd Conference on Learning Theory, Montreal, Quebec, Canada, June 18-21, 2009, 2009.

---

MATS: An Audio Language Model under Text-only Supervision
Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxon- omy for urban sound research. In Proceedings the the 22nd ACM International Conference on Multimedia, MM 14, pp. 1041-1044, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 97814 158433333333
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatreen T., and Dubnov, S. Large-scale contrastive language-audin pretraining with feature fusion and augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5, IEEE, 2023.
Spadini, T. Sound events for surveillance applications (1.0.0) [data set]. 2019.
Yang. Q. Xu, J., Liu, W., Chu, Y., Jiang, Z., Zhou, X., Leng, Y., Lv, Y.. Zhao, Z., Zhou, C., and Zhou, J. Air-bench: Benchmarking large audio-language models via genera- tive comprehension. In Ku, L., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), ACI. 2024, Bangkok, Thailand, August 11- 16, 2024, pp. 1979-1998. Association for Computational Linguistics, 2024.
Sturm, B. L. The state of the art ten years after a state of the art: Future research in music information retrieval. Journal of new music research, 43(2):147-172, 2014.
Su, Y. Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instruction-follow them all. arXiv preprint urXiv:2305.16355 2023.
Tang. C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. SALMONN: towards generic hearing abilities for large language models. The Twelfth International Conference on Learning Representations, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguis- tics. Singapore, December 6-10, 2023, pp. 15757-15773. Association for Computational Linguistics, 2023.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Zhang, Y., Xu, X., Du, R., Liu, H., Dong, Y., Tan, Z.-.., Wang, W., and Ma, Z. Zero-shot audio captioning using soft and hard prompts. arXiv preprint arXiv:2406.06295. 2024.
Tian, M., Srinivasamurthy, A., Sandler, M., and Serra, X. A study of instrument-wise onset detection in beijing opera percussion ensembles. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2159-2163, 2014.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. CoRR, abs/2302.1339 13971, 2023.
Vapnik, V. Statistical learning theory. John Wiley & Sons google schola, 2:831-842, 1998.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Wang, C., Liao, M., Huang, Z.. Lu, J.. Wu, J.. Liu, Y., Zong, C., and Zhang, J. Blsp: Bootstrapping language-speech pre-training via behavior alignment of continuation writ- ing. arXiv preprint arXiv:2309.000
2023.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T. Next-gpt: Any-to-any multimodal LLM. In Forty-first International Conference on Machine Learning, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
12

---

MATS: An Audio Language Model under Text-only Supervision
A. Proof of Theorem 3.1
In this section, we provide a proof for Theorem 3.1.
Proof. Let Ree(h) = [2 (s)())))) [h(z") yl denote the generalization error on the train set. According to triangle inequality, we can drive that:
(h) Rie(h) (ex(hi ((r(h) R1r(h) (h)
(8)
S Rio(h) (ar(h)) [Rr((h) (rr(h)) (h)
Bho - Per
[(s= 1)((2) -vl -(( [M(=) - wll
(9)
Σn Palu) ((a)))aa)))) - -yl -l E. pr()) (ee))>pp()))))((() vll
Assuming that the class distributions for training and testing consistent, we can obtain pa(μ) pT()), then we can achieve that:
(ho(h) Per (h))
Σn pA(U) (B(==)=pa(t* w)lh(==) - vl -Bst) =/)
(( (h) (10)))((
max V
(10)
max dixci (p)
(h) - Rer(h)) follows by standard application Cherii-Chervonenkis (Vapnik, 1998) theory bound the true Rar (h) by its empirical estimate Ruz(h). Namely, if the Drr is N-size i.i.d. samples, then with probability exceeding - ō,
(h) Rgr(1))
2d 2d V2NV log
(11)
N
At the last, we can achieve that:
8 Rie (h) S Rer (h) + max (discz, (PA (z)lb) ,rr (z'lo))) +
2 8 2) 2d V2NV + log
(12)
V N
B. Dataset
B.1. The train dataset MATS-GPT2 and MATS-LLaMA
Table S8 provides detailed overview the training dataset, AudioTIA. this dataset, the inputs are primarily composed of descriptions of audio contents and textual instrucrions. AudioTIA is divided into close-ended and open-ended tasks, focusing two audio types: Sound and Musie. Specifically, the close-ended dataset, consisting of about 1.5M samples, primarily includes classification tasks (6),7%), captioning tasks (40%), and the remaining portion comprises simple audio question-answert tasks. In contrast, the open-ended dataset, comprising about 3.8M samples, focuses on complex audio question-answeringg answering tasks. The close-ended dataset is designed to establish the model's ' foundational audio perception capabilities, and the open-ended dataset focuses on enhancing the model's 's capacity for advanced audio understanding and reasoning.
13

---

MATS: An Audio Language Model under Text-only Supervision
Table S8. The staties of the Audi o'l A dataset.
Close-Ended Audio Text Instruction Answering Data (~1.5M)
Audio Type
Task
Datasets
#Samples
Percentage
AudioSet (Gemmeke ct al., 2017), FSD50k (Fonseca et al., 2021),
VGGSound (VGG) (Chen et al., 2020), DCASE (Mesaros al., 2017),
ECS-50 (Piczak, 2015), UrbanSSundSK(USSK (Salamon et al., 2014)
CLS
TUT Acoustic Scenes 2017(TUT)) (Mesaros et al., 2016),
3885K
~25.6%
Sound Events for Surveillance Coplications(SESAA (Spadini, 2019)
Sound
VS (Gong ct al., 2022), CREMA-D (Cao ct al., 2014)
RAVDESS (Livingstone & Russo, 2018)
WavCaps (Mei et al., 2024), Macs (Martín-Moratóó & Mesaros, 2021),
CAP
5118
~34.5%
AudioCaps (Kim et al., 2019), Clotho-v2 (Drossos et al., 2020)
AQA
Clotho-AQA (Lipping et al., 2022)
~00
1.33
GTZAN (Sturm, 2014), Beijing Opera (BJO) (Tian al., 2014),
CLS
4497K
33.1%
OpenAQA (Ghosh et al., 2024)
Music
OpenAQA(Ghosh al., 2024), MusicCaps (Agostinelli et al., 2023),
CAP
330K
5555
Song Describer dataset (SDD) (Manco al., 2023)
Open-Ended Audio Text Instruction Answering Data (~3.8M)
Sound
AQA
OpenAQA (Gong et al., 2024)
2547K
~6..8
Music
AQA
MusicQA (Liu et al., 2024a)
1207K
333.2%
B.2. The train dataset of MATS-Audio
Table S9 presents the training dataset of MATS-Audio, referred to AudioATA. The data format for AudioAIA is (audio, instruction ion, answer), which differs from the input format of AudioTIA with (text, instruction answer). The AudioAIA IA dataset is derived from AudioTIA, excluding any subsets used exclusively as test data. Overall, the Audion oATA AA dataset comprises approximately 1.5M samples for close-ended tasks and 2.6M samples for open-ended tasks.
Table S9. The statics of the Audi oAIA dataset.
Close-Ended Audio Instruction Answering Data (~1.5M
Audio Type
Task
Datasets
#Samples
Percentage
AudioSet (Gemmeke al., 2017), FSD50k (Fonseca al., 2021).
VGGSound (VGG) (Chen al.. 2020), DCASE (Mesaros al., 2017).
CLSS
<31K
21.9%
TUT Acoustic Scenes 2017(TUT) (Mesaros et al., 2016),
Sound
Sound Events for Surveillance (pplications(EESA)) (Spadini. 2019)
WavCaps (Mei et al., 2024), Maes (Martín-Moratt
Mesaros, 2021),
CAP
517K
344.2%
AudioCaps (Kim et al., 2019), Clotho-v2 v2 (Drossos et al., 2020)
AQA
Clotho-AQQ (Lipping et al., 2022)
~..
00.4
CL.S
OpenAQA (Ghosh et al., 2024)
633K
41..7
Music
CAP
OphhAQAsGGhoh et al., 2024), MusicCaps (Agostinelli et al., 2023),
28.3K
1.99
Open-Ender Audio Instruction Answering Data (~2.6M)
Sound
AQA
OpenAQA (Gong al., 2024)
~4477K
~95.4%
Music
AQA
MusicQA (Liu et al., 2024a)
118K
44.6%
B.3. The test benchmark of MATS
Table S10 provides a detailed overview of the test benchmark and corresponding evaluation metrics. The benchmark covers a range of audio event classification tasks, such as ESC-50 (Piczak, 2015) (single-label classification), FSDSOK (Fonseca et al., 2021) (multi-label classification), DCASE (Mesaros et al., 2017), TUT (Mesaros et al., 2016), VGG (Chen et al., 2020), US8K (Salamon et al., 2014), GTZAN (Sturm, 2014) and BJO (Tian et al., 2014), as well as audio captionning
14

---

MATS: An Audio Language Model under Text-only Supervision
tasks like AudioCaps (Kim al., 2019) and Clotho-v2 (Drossos al., 2020). It also encompasses music captioning tasks including MusicCaps (Agostinelli al., 2023), simple QA tasks ClothoAQA, and complex QA tasks such as AlR-Bench Chat (Yang et al., 2024) and MMAU (Sakshi et al., 2024) benchmarks.
Table S10. The test benchmarks and evaluation metrics are presented. B-ACC represents binary accuracy. Note: All evaluation metrics are designed such that higher values indicate better performance.
Glose-Ended Audio Instruction Answering Test Data
Audio Type
Task
Benchmark
Metric t
CLS
ECS-50 (Piczak, 2015)
ACC
CL.S
FSD50K (Fonseca et al., 2021)
mAP
CLS
DCASE (Mesaros et al., 2017)
ACC
CLS
TUT (Mesaros al., 2016)
ACC
Sound
CLS
VGG (Chen et al., 2020)
ACC
CLS
US8K (Salamon et al., 2014)
ACC
CAP
AudioCaps (Kim ct al., 2019)
SIDEESSPICE/SPDEER
CAP
Clotho-v2 (Drossos ct al., 2020)
(ICEPSSPIE
AQA
Clotho AQA (Lipping ct al., 2022)
ACC|B-ACC
CLS
GTZAN (Sturm, 2014)
ACC
Music
CLS
BJO (Tian et al., 2014)
ACC
CAP
MusicCaps (Agostinelli et al.. 2023)
BLUGHI
Open-Ended Audio Instruction Answering Test Data
AQA
AIR-Bendh Chat (Yang al., 2024)
GPT4
Sound
AQA
MMAU (Sakshi et al., 2024)
ACC
AQA
AIR URR-Bench Chat (Yang al., 2024)
GPT4
Music
AQA
MMAU (Sakshi et al., 2024)
ACC
B.4. The design for generating audio descriptions in close-ended tasks.
In this part, we describe the pipeline and prompt used with ChatGPT (Achiam et al., 2023) to generate audio descriptions for various classification task datasets, including ESC-50 (Piczak, 2015), FSD50K (Fonseca et al., 2021), VGG (Chen et al., 2020), ECS-50 (Piczak, 2015), US8K (Salamon et al., 2014), TUT (Mesaros et al., 2016), SESA (Spadini, 2019), VS (Gong et al., 2022), CREMA-D (Cao et al., 2014), and RAVDESS (Livingstone Russo, 2018) for sound classification, and GTZAN (Sturm, 2014) and BJO (Tian et al., 2014) for music classification. Figure S4 provides overview of the pipeline. The pipeline involves supplying GPT-4o-mini with set of example audio descriptions corresponding to few labels, followed by prompt to guide the generation process. GPT-4o-mini then produces descriptions for all provided labels. The final set of generated descriptions, paired with their corresponding labels, serves as input for MATS.
B.5. The instruction templates for training MATS
Recent studies (Deshmukh et al., 2023; Tang et al., 2024) have demonstrated that instruction fine-tuning enhances the instruction-followingg abilities and comprehension capabilities of audio-language multimodal models. Building on this insight, we adopt instruction fine-tuning when training MATS. Table S11 presents the instruction templates designed for all tasks in our study.
C. More Experimental Results
C.1. Visualization Analysis on Audio-Language Modality Gap
Figure 3 presents visualization analysis of the audio-language modality gap. As shown in Figure 3 a), a noticeable gap
(Liang et al., 2022) remains between audio embeddings and language embeddings of CLAP. In Figure 3 b). introducing Gaussian noise reduces the embedding gap to some extent; however, the language and audio embeddings remain distributed near opposite ends. Similarly, in Figure 3 c), the nemorry-bsse method shows a comparable effect to b). As previously noted, this method completely replaces audio embeddings with language embeddings, resulting in the processed audio
15

---

MATS: An Audio Language Model under Text-only Supervision
1. A solo male vocal performance, deep and resonant, carries
the emotional weight of the song.
male, vocal
apt-4o-mini
2. powerful male voice delivers poignant lyrics with a robust vocal range.
intense male vocalist commands track with dynamic strength.
You are an Al assistant tasked with describing the sound of given music instrument.
I will you a music instrument. Write 50 diverse and concise sentences that describe the sound of this instrument.
Requirements for the generated captions:
The caption should less than words.
Make caption mainly
the given sound
Make you are using grammatical subject-verb-oojjet sentences.
4. Directly describe the sound and avoid using the word "heard".
USER: Clapper-drum
ASSISTANT:[1. The clapper-drum produces a sharp, crisp beat., The clapper-drum emits a bright, percussive crack.,
The drum emits rapid, rhythmic snap.,
...]
Figure S4. Pipeline And Prompt of generating audio descriptions.
Table S11. The instruction template of every task. CLS: Audio Event Classification; CAP: Audio Captioning: AQA: Audio Question Answer.
Task
Input text
Input instruction format
Output answer format
CLS
GPT generated description
"Classify events in the audio clip."
label a, caption
Sound CAP
caption
"Generate audio caption."
Music CAP
caption
"Generate music caption."
caption
AQA
generated caption
Question
answer
features losing audio-specific information and becoming overly concentrated, thereby reducing the model's generalization ability. In contrast, Figure 3 d) illustrates that our Santa further integrates the two modalities into a more unified and cohesive distribution.
C.2. Hyperparameter Analysis
The impact of the different temperature т. In Table S12, we analyze the impact of different 7 values on the model's performance. Our model achieves the best performance when r = 0.1. When 7 is too small, our algorithm tends to select only the most similar embedding, which limits the representational power of language embeddings and leads to a decrease in model performance. Conversely, when large, the model may capture irrelevant augmented language embeddings with low relevance.
Table S12. performance of different on AudioCaps.
Table S13. The performance different on AudioCaps.
1/150
1/80
1/40
1/20
1/10
L
8
16
32
64
CIDEr
0.731
0.734
0.732
0.734
0.735
0.731
CIDEr
0.728
0.730
0.729
0.735
0.734
The impact of different top L. Table S13 presents the performance of our method with different values of L. When too large, the model may capture irrelevant augmented language embeddings with low relevance. Conversely, as approaches 1, the model loses the benefits of multiple language representations. We set to strike a balance.
16

---

MATS: An Audio Language Model under Text-only Supervision
The impact of the different balancing parameter X. In this part, conducted experiments with different values of X, as shown in Figure S5. The results clearly demonstrate that the best performance is achieved when both language and audio are integrated simultaneously, and the performance is relatively insensitive to the variations in A. However, when audio is entirely removed, the model's performance drops significantly, further validating the effectiveness of the balancing strategy within the Santa mechanism.
Effect of MATS'S Performance the AudioCaps Dataset
0.55
0.30
5.0
CBO
60
91
93
0.5
3.6
37
3.5 3.0
Figure The effect of different on AudioCaps.
aver-Variations in LoRA. Table S14 and Table S15 present the performance variations of MATS on the AudioCaps test benchmark under different LoRA ranks and scaling factors. When the rank scaling factor is small, the model undergoes insufficient updates, leading to underfitting. Conversely, excessively large ranks scaling factors result in overly aggressive updates, increasing model complexity and causing overfitting.
Table S14. performance of MATS different ranks LoRA Table $15. The performance MATS with different scaling factors on AudioCaps.
of LoRA on AudioCaps.
Rank 4
8
16
4
Rank
CIDEr
0.706
0.735
0,702
CIDEr
0.674
0.735
0.682
The impact of the different number of M. Table S16 shows that the performance of our model initially increases and then decreases as the size of the memory bank M grows. The initial improvement is attributed to the increasing richness of the provided language embedding information. However, as the memory bank continues to expand, the fixed value of limits the model's ability to effectively utilize diverse language embeddings.
Table S16. The performance of MATS with the different number of memory bank M on AudioCaps.
M
1000
2000
3000
3500
4000
4500
5000
5172
5500
6000
7000
CIDEr
0.710
0.700
0.714
0.716
0.719
0.705
0.708
0.735
0.705
0.700
0.700
The impact of k-means cluster size K. Table S17 investigates the impact of the k-means cluster size in the Santa mechanism. As shown, regardless of the cluster size, the model consistently outperforms noise-based and memory-based methods, indicating that our Santa mechanism is robust to the choice of the k-means cluster number. We K 100 in this work.
The number of Mapper'layers. We further evaluated the impact of varying the number of Transformer layers on the model's ss performance on AudioCaps test set. As shown in Table S18, the model achieves optimal performance with 8 layers. When the number of layers is fewer than 8, the model's 's capacity is insufficient for effective learning, leading to underfitting. Conversely, increasing the number of layers beyond 8 results in overfitting, thereby reducing generalization ability. Based on these findings, we adopt an 8-layer Transformer in this work.
17

---

MATS: An Audio Language Model under Text-only Supervision
Table S17. Ablation study of MATS-LLaMM the cluster numher K in the Santa mechanism, evaluated on AudioCaps benchmark.
Table S18. Ablation Study the number Transformer layers on the AudioCaps benchmark.
K
CIDEr
SPICE
SPIDEr
10
0.731
0.170
0.450
number
CIDEr
SPICE
SPIDEr
30
0.733
0.170
0.452
0.696
0.166
0.431
60
0.734
0.171
0.452
0.735
0.171
0.453
100
0.735
0.171
0.453
16
0.653
0.174
0.414
150
0.731
0.170
0.450
C.3. Some Instances of MATS-LLaMA
In Figure S6, Figure S7, Figure S8, Figure S9, and Figure S10, we present examples MATS-LLaMA and Qwen2-Audio- Instruct (Chu et al., 2024) applied to tasks including complex QA, audio classification, general audio captioning, simple QA, and music captioning. In the AIR-Bench Chat and MusicCaps benchmark examples, Qwen2-Audio-Insinn significant hallucination issues, whereas MATS-LLaME while only partially covering the answers, does not generate incorrect or nonsensical responses. For audio event classification tasks (FSD50K, BJO, and US8K). Qwen2-Audio Instruct struggles to generate specific responses, often providing only broad and general answers. For audio captioning examples (AudioCaps and Clotho), both @wee2-AAudooInsttrr and MATS-LLaMA can correctly generate appropriate captions. Additionally, in the ClothoAQA example, the correct answer is 'BIRD', While Qwen2-Audio-Instrune response ('owl') refers bird, the sound does not correspond to that of owl. These cases highlight MATS-LLaMA'S strong audio understanding and reasoning capabilities.
exhibits
+
=|[|=|||
speaking on radio music plays in the background.
These Electric guitar; Guitar; Plucked string instrument; Musical instrument; Music being
-
-
played.
What is the primary sound in the audio?
Identify the sounds in the audio clip, only display audio tags.
USER
The primary sound in the audio is man speaking.
Electric guitar; guitar; plucked string instrument; bass; music
MATS
MATS
The primary sound is of truck horn honking continuously.
[(sounn'; 'music"), ('sound': 'singing", 'type': 'vocal"), ('sound': 'respiratory sounds', 'type': 'breathing']
Qwenn Audio- Instruct
Owen2-Audio- Instruct
Man speaking
Electric guitar; Guitar; Plucked string instrument; Musical instrument; Music
a)
Figure $6. The example of different tasks, Complex audio QA tasks AIR-Bench Chat Benchmark. b) Audio Classification tasks FSD50K.
18

---

MATS: An Audio Language Model under Text-only Supervision
A thunder storm is quietly rolling in the background.
+
-|||||||
A sound of Clapper-drum
Identify the sounds in the audio clip, only display audio tags.
Describe the following audio a caption.
Clapper-drum
The sound in the background thunder.
MATS
[['sound: 'music)), ('sound': 'respiratory sounds, breathing"
It's the sound of thunder rolling and rain falling.
Qwan2-Audic- Instruct
Clapper-drum
A thunder storm.
Answeer
Answer
Figure The example of different tasks. a) Complex audio QA tasks on AIR LIBBBench Chat Benchmark. b) Audio Classification tasks on BJO.
Children are talking to each other and birds are chirping.
-[]|m[||
sound of gun shot
Identify the sounds in the audio clip, only display audio tags.
Describe the following audio caption.
USER
USER
Birds chirping and child is yelling the background.
Gun shot
MATS
[['sound: 'explosion', 'start time': 0.05, 'end time': 0.68), l'sound': 'fieworks", 'start time': 0.05, 'end time': 0.68)]
variety of birds chirping with people talking in the background.
Qwen2-Audio- Instruct
Children are talking to each other and birds are chirping.
gun shot
Answer
b)
Figure S8. The example of different tasks. a) Audio Captioning tasks Clotho. b) Audio Classification tasks UrbanSound8K.
19

---

MATS: An Audio Language Model under Text-only Supervision
A baby cries and woman speaks followed by some light rustling and someone speaking faintly over television.
[4 -[]|-[||
A flock of birds screaming.
What type of animal is making the light sound in the background?
Describe the following audio a caption.
USER
Bird
A baby crying followed by woman talking.
MATS
A baby cries briefly and then an adult female speaks and laughs.
An owl.
Qwen2 Audio Instruct
Qwand Audio- Instruct
A baby cries and woman speaks followed by some light rustling and someone speaking faintly over television.
BIRD
Answer
Amawer
b)
Figure S9. The example of different tasks. a) Audio Captioning tasks AudioCaps. Simple audio QA tasks ClothoAQA.
The low quality recording features cover of rock song that consists of an electric guitar solo melody played over playback instrumental. The instrumental features punchy kick and hits, soft cymbals and smooth bass. It sounds passionate, even though the recording is muffled and noisy, as it was probubly recorded with phone.
There are wind sound
+ "[[[i|
What is the most noticeable sound event in the audio? Select one option: v'Continuous wind,, 'Eruption', 'Mau speaking'. Ticking noise' "nisee''nRespond with exactly one of the options ahove.
Listen to this music clip and describe the music.
The low quality recording features a cover rock song and consists of an electric guitar solo melody, the recording noisy and mono, as it was probably recorded with a phone.
Continuous wind
recording a cover of a rock song. There is a distorted e-guitar solo melody being played over playback instrumental music which consists acoustic drums, an e-bass and a piano melody. The piece has grovv feel to it. This seng may be playing pencticing at home headphones.
This music is
MATS
Enittion
Qwent-Audio- Instruct
Qwen
The low quality recording features a cover of a rock song that consists of electric guitar melody played over playback instrumental. The instrumental Reatures punchy kick and suare hits, soft cymbals and smooth bass. It sounds passionate, even though recording muffled and noisy, recorded with a phone.
Continuous wind
Answer
probably
a)
b)
Figure $10. The example of different tasks. a) Music Captioning tasks MusicCaps. h) Complex audio QA tasks MMAU Benchmark.
20


Reviews:
[{"review_id": "rwDTpD1Qkt", "reviewer": "Reviewer_72QC", "summary": "This paper proposes MATS, an audio-language multimodal large language model (LALM) that is trained solely on text data while achieving strong performance on various audio comprehension tasks. Unlike conventional LALMs, which require a large corpus of audio-language pairs, MATS leverages CLAP (Contrastive Language-Audio Pretraining) to align audio and language modalities without audio supervision.  \n\nDuring training, MATS only uses textual data, where CLAP's language encoder extracts text embeddings, which are further processed using a Transformer-based mapper before being fed into the LLM. To mitigate the modality gap between CLAP’s audio and text embeddings, a Gaussian noise injection strategy is applied to text embeddings during training.  \n\nAt inference time, audio inputs are encoded using CLAP’s audio encoder, and the Santa mechanism is introduced to bridge the modality gap. Santa retrieves semantically related caption embeddings from a clustered database and balances them with the input audio embedding. The final input to the LLM consists of both the audio embedding and Santa's retrieved text embedding, effectively improving generalization.  \n\nExtensive zero-shot evaluations demonstrate that MATS achieves performance comparable to state-of-the-art audio-supervised models on multiple benchmarks, including audio classification, captioning, and open-ended question answering. Notably, MATS surpasses SALMONN and Qwen-Audio-Chat on the MMAU benchmark while being trained only on text data, showcasing its ability to learn audio semantics without direct audio supervision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: Training an ablated system that replaces Santa with mechanisms of prior works (PromptAAC and DRCap)**. \n\nFollowing your suggestion, we replace Santa with the modality-gap reduction mechanism of PromptAAC and DRCap, referred to MATS-PromptAAC and MATS-DRCap. As shown in Table 1, Santa achieves the best performance on both closed-ended and open-ended tasks, which validate that Santa outperforms previous text-only methods.\n\n- DRCap enhances audio captioning performance by leveraging a benchmark-specific memory bank, fully mapping the audio embedding to weighted language embeddings. But DRCap discards the original audio embedding, making its performance heavily dependent on the relevance between the memory bank and the test benchmark. \n  However, in **multi-task setting**, the memory bank is no longer tailored to a single benchmark but instead aggregates information from multiple benchmarks. As a result, the mapping process may introduce unintended noise, projecting the audio embedding into a less relevant textual embedding space, leading to performance drop.\n\n- PromptAAC adopts an augmentation-based approach that involves injecting noise and substituting similar language inputs. It retrieves audio events by matching audio embeddings with language embeddings derived from 527 predefined audio labels in AudioSet. However, the limited variety of audio events restricts the diversity of the retrieved information, resulting in inferior performance compared to Santa.\n\n\n| Benchmark| ESC-50 (ACC) | AudioCaps (CIDEr) | AIRBench-Sound (GPT-4) | AIRBench-Music (GPT-4) |\n| - | - | - | - | - |\n| MATS-PromptAAC| 0.77| 0.593| 6.07| 5.28 |\n| MATS-DRCap | 0.84 | 0.619 | 5.83 | 5.29 |\n| **MATS-LLaMA (Ours)** | **0.88** | **0.735** | **6.43** | **5.76** |\n\n_Table 1: Comparison results on CLS, CAP, and AQA benchmarks._\n\n### **W2: What is the difference between the mechanism in the first column of Table 4 and DRCap?**\n\n- DRCap introduces the Retrieval-Augmented Generation (RAG) and Projection-Based Decoding (PD) strategies. In Table 4 of main paper, we only use PD strategy (denoted as **Memory-based**). \n- We further report the performance of DRCap (RAG+PD), where we replace Santa with DRCap in our framework. As shown in Table 2, it still underperforms the performance of DRCap (reported in original paper) in single-task setting . \n- This is because DRcap fully discards original audio embedding during inference, making its performance heavily dependent on the relevance between the memory bank and test benchmark. \n    However, in multi-task setting, the memory bank is no longer tailored to a specific benchmark but instead integrates information from multiple benchmarks. This broader integration can introduce unintended noise during the mapping process, projecting the audio embedding into a less relevant textual embedding space. As a result, MATS-DRCap, trained in a multi-task setting, experiences a performance drop compared to DRCap.\n  \n| Method| CIDEr | SPICE | SPIDEr |\n|-|-|-|-|\n| Memory-based (Only PD) | 0.234 | 0.094 | 0.164  |\n| MATS-DRCap (PD+RAG) | 0.619 | 0.175 | 0.397 |\n|DRCap (single-task setting reported in original paper) | 0.718 | **0.186** | 0.452\n| **MATS-LLaMA** | **0.735** | 0.171 | **0.453** |\n\n_Table 2: Ablation Study on AudioCaps._\n\n### **W3: Improve Figure 2.**\n\nThanks for your suggestion. We will update the figure in next version to better illustrate the \"Modal-Transfer module\" and its connection to noise injection/Santa.\n\n### **Q1: Could you elaborate or provide evidence explaining why individual CLAP text embeddings have limited representational power? What factors lead to insufficient semantic relevance?**\n\n1. To validate it, we perform a retrieval task between CLAP audio embeddings and CLAP text embeddings on Clotho test set. Specifically, we compare the error rates of two strategies: top-K retrieval; K-means clustering followed by top-K retrieval. As shown in Table 3, the K-means method achieves a lower error rate in capturing semantically relevant captions, effectively mitigating the impact of irrelevant textual information caused by the limited representational capacity of individual language embeddings.\n\n2. It may be attributed to the  CLAP text encoder compressing textual information into a 1024-dimensional embedding space. Such aggressive dimensionality reduction leads to a loss of fine-grained semantic details, resulting in insufficient representational capacity of individual text embedding.  \n\n|Method|Error Rate@5|\n| - | - |\n|K-means-based | 18.3% |\n|TopK | 23.3% |\n\n_Table 3: Retrieving Error Rate@5 on the Clotho Test Set_\n\n### **Q2: How exactly is noise variance determined?**\n\nThe variance is treated as a hyperparameter. As suggested by [1], the optimal value roughly aligns with the used strategy (calculating infinity norm between audio and text embeddings over 30 randomly selected samples), also shown in Figure 4 of the main paper.\n\n[1] Training audio captioning models without audio.", "claims_and_evidence": "The paper presents experimental results comparing MATS with audio-supervised models, demonstrating that the proposed text-only training method achieves comparable performance. Additionally, the authors claim that the Santa mechanism effectively mitigates the modality gap and outperforms previous text-only audio LLM approaches (as shown in Table 2 and Table 4).  \n\nHowever, I have concerns regarding the justification of the latter claim. From Table 2, MATS appears to perform similarly to previous text-supervised models, and these models are not compared in tasks beyond audio captioning. While I acknowledge that previous text-supervised models primarily focus on captioning, a stronger justification for Santa's superiority is needed. Specifically, since Santa is the key architectural difference from previous text-supervised approaches, a more thorough comparison would be beneficial. This could be done by training an ablated system that replaces Santa with mechanisms proposed in prior works, evaluating it on the broader set of tasks used in this paper. Such an experiment would provide clearer evidence of Santa’s advantage over prior approaches.  \n\nAdditionally, I am unclear about the discrepancy between the first column of Table 4 and DRCap. What is the exact difference between the mechanism in the first column of Table 4 and DRCap? The performance gap between the first column of Table 4 and DRCap in Table 2 is quite large, and it would be helpful to clarify why this occurs.", "methods_and_evaluation_criteria": "Please see \"Claims And Evidence\" section.", "theoretical_claims": "I read through the theoretical claims in 3.3. I think it is correct but I am not very certain.", "experimental_designs_or_analyses": "Please see \"Claims And Evidence\" section.", "relation_to_broader_scientific_literature": "The key contribution of this paper is proposing a method to train a text-only supervised audio LLM that generalizes to a broader range of audio-related tasks by constructing a more extensive dataset, as well as introducing an improved inference-time mechanism (Santa) to reduce the modality gap between audio and text embeddings. The text-only training framework leverages the recent CLAP model (Elizalde et al., 2023), building upon prior works such as Pengi (Deshmukh et al., 2023) and LTU (Gong et al., 2024). The Santa mechanism further enhances previous memory-based or noise-based methods (e.g., DRCap by Li et al., 2024; NoAudioCaptioning by Deshmukh et al., 2024) by integrating clustering and weighted embedding retrieval, explicitly addressing limitations in existing methods regarding the preservation of audio semantic information.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "I would recommend the authors improve Figure 2 to better highlight the correspondence between the upper-right \"Modal-Transfer Method\" block and the rest of the figure. Currently, there is no clear segmentation between the Santa mechanism and the noise injection component, making it difficult to distinguish these parts. Additionally, clarifying the connection between the label \"modal-transfer method\" and the Santa/noise injection block would improve the figure's readability.", "questions_for_authors": "1. In line 252:  \n   > \"However, due to the limited representational power of individual language embedding, this strategy is prone to retrieving the texts with insufficient semantic relevance, thereby affecting the effectiveness of audio-language modality alignment.\"  \n\n   Could you further elaborate or provide evidence explaining **why** individual language embeddings from CLAP have limited representational power? What factors lead to insufficient semantic relevance in this context?\n\n2. In line 405, the authors state that the variance is a hyperparameter and searched for the optimal value. However, in line 169, the author introduces variance as determined by calculating the infinity norm between audio and language embeddings over a set of 30 randomly selected samples. How exactly is noise variance determined?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "USCMBfkRdk", "reviewer": "Reviewer_pS8x", "summary": "This paper proposes a text-only supervision method that closes the gap between the text embedding space and the audio embedding space via a mechanism called santa.\n\n## Update after rebuttal\nI deeply appreciate authors providing additional results. It resolves my other concerns except for this one: \"connection between the bound derived and the proposed \"santa\" method is not clear\". I have read the rebuttal about this but still feel not that directly related and felt the derivation a bit disjoint with the main theme of the paper.\n\nTherefore, I decided to maintain my score as it is.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: The proposed Santa does not directly use the disc_L1 metric.** \n\n1. In our design, we only have access to text-only data during training,  making it impractical to directly use the disc_L1 metric to reduce the modality gap. Instead, as shown in Figure 3 of main paper, our Santa achieves a similar effect in effectively reducing the distance between audio and text embeddings.\n2. Further, Table 1 presents relevant statistical information. Specifically, we randomly select 350 samples from `AudioAIA` dataset and generate their language embeddings $Z_T$ and audio embeddings $Z_A$ using the CLAP encoder. Santa is then applied to the audio embeddings, denoted as $f_{\\mathrm{Santa}}(Z_A)$. \nNext, we calculate the L1 distance between the prototype of audio embeddings $Z_A/f_{\\mathrm{Santa}}(Z_A)$ and language embeddings $Z_T$. As shown in Table 1, Santa effectively reduces the distance between audio and text embeddings, validating its effectiveness in bridging the two spaces closer.  \n\n\n| Method      | L1 distance |\n| ----------- | ----------- |\n| Origin CLAP | 18.35       |\n| Santa       | 10.48       |\n\n_Table 1: The statistics on the modality gap within CLAP. **Note:** We randomly select 350 samples to calculate the L1 distance._\n\n\n\n### **Q2: Lines 165-177 on the left column of page 4 seem to be a repetition of stuff on page 3 line 264: identity -> identify**  \nThanks for your reminder. We will update it in the next version.\n\n### **Q3: Why do you need 5M text samples? Is the description space that large?**\n\n1. Using 5M text samples aims to improve the generalization of MATS. With approximately 7B parameters, MATS-LLaMA requires a substantial amount of training data to effectively scale its capacity. \nAs shown in Table 2, most existing LALMs of similar size (around 7B parameters) are trained on over 5M audio-text pairs. \n\n2. Furthermore, we conducted an ablation study to assess the performance of MATS-LLaMA with different training dataset sizes. As shown in Table 3,  the model's performance improves as the dataset size increases, especially in open-ended scenarios. This result validates the necessity of using 5M text samples.\n\n| Model          | Capacity | #Sample |\n| -------------- | -------- | ------- |\n| Audio Flamingo | 2.2B     | 5.9M    |\n| GAMA           | 7B       | 8.7M    |\n| LTU            | 7B       | 5.6M    |\n| LTU-AS         | 7B       | 9.6M    |\n| SALMONN        | 7B       | 5M      |\n| MATS-LLaMA     | 7B       | 5M      |\n\n_Table 2: The number of train dataset from current LALMs_\n\n| Ratio (%) | AudioCaps (CIDEr) | AIRBench-Sound (GPT-4) | MusicCaps (ROUGH_L) | ESC-50 (ACC) |\n| --------- | ----------------- | ---------------------- | ------------------- | ------------ |\n| 50%        | 0.697             | 6.25                   | 16.4                | 0.87         |\n| 75%        | 0.705             | 6.30                   | 17.7                | 0.88         |\n| 100%       | **0.735**         | **6.43**               | **18.7**            | **0.88**     |\n\n_Table 3: Performance of MATS-LLaMA under different training data ratios_", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. The proof of theorem 3.1 is convincing to me provided that the assumption of P_A(y) = P_T(y) is true. This might limit the application scenario to more general tasks (e.g. speech) but is a reasonable assumption for the experiments they conduct.", "experimental_designs_or_analyses": "Yes. The experiments are sound to me.", "relation_to_broader_scientific_literature": "This paper has compared to a wide range of existing work that can perform audio understanding.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Other Weaknesses:\n1. I found the connection between the bound derived and the proposed \"santa\" method not very clear. It reads to me as if the authors used a lot of maths to prove a bound, only to conclude that we need to bridge the two spaces closer and propose a method that does not directly use the disc_L1 metric. Please clarify.", "other_comments_or_suggestions": "Lines 165-177 on left column of page 4 seem to be a repetition of stuff on page 3\nline 264: identity -> identify", "questions_for_authors": "1. Why do you need 5M text samples? Is the description space that large?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UFBvrIOoyr", "reviewer": "Reviewer_LCoU", "summary": "The authors propose to use pre-trained audio-text contrastive models such as CLAP to achieve text only supervision, with a strongly-related noisy text with audio mechanism to introduce robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We sincerely appreciate your time and effort in reviewing our manuscript. Your positive evaluation is highly encouraging. Thank you for your valuable feedback.", "claims_and_evidence": "The authors compare proposed methods to several other audio large language models and show that proposed methods can perform similarly to those trained with the need of audio data during training phase. This provide evidence that proposed method works to some degree.", "methods_and_evaluation_criteria": "The authors leverage various and diverse audio tasks and datasets for evaluation, this shows more robust and generalizable results.", "theoretical_claims": "The authors provide a theoretical analysis on generalization and take the modality discrepancy into consideration, this helps to provide different perspectives to the problem other than based on heuristics.", "experimental_designs_or_analyses": "The authors also provide MATS-Audio to show the modality gap of approximate with CLAP models. This is a good additional contribution to the community.", "relation_to_broader_scientific_literature": "Multimodal large language model is a current popular research, this work explore the modality gap in multimodal encoders and how it affects when trained with large language model, which is a good contribution to the community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The comparisons with various open-ended audio tasks such as AIR-Bench and MMAU provide thorough understanding of the current landscape.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes MATS, a text-only trained audio-language model that achieves performance comparable to audio-supervised LALMs.
It introduces SANTA (Strongly-related noisy text with audio), a novel method that bridges the modality gap between CLAP’s audio and text embeddings during inference. Reviewers agree this is a solid contribution

MATS represents a clear step forward in training multimodal models under modality-limited supervision, which is both a practical and theoretically interesting direction. The paper introduces a novel method (SANTA), demonstrates competitive empirical results, and shows thoughtful evaluation across multiple dimensions.
Decision: Accept (poster)