Venue: NAACL 2025

# PAT: Parameter-Free Audio-Text Aligner to Boost Zero-Shot Audio Classification

Ashish Seth, Ramaneswaran Selvakumar, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha

University of Maryland, College Park

{aseth125, ramans, sonalkum, sreyang, dmanocha}@umd.edu

###### Abstract

Audio-Language Models (ALMs) have demonstrated remarkable performance in zero-shot audio classification. In this paper, we introduce PaT (**P**arameter free **A**udio-**T**ext aligner), a simple and _training-free_ method aimed at boosting zero-shot audio classification performance of CLAP-like ALMs. To achieve this, we propose to improve the cross-modal interaction between audio and language modalities by enhancing the representations for both modalities using mutual feedback. Precisely, to enhance textual representations, we propose a prompt ensemble algorithm that automatically selects and combines the most relevant prompts from a datastore with a large pool of handcrafted prompts and weighs them according to their relevance to the audio. On the other hand, to enhance audio representations, we reweigh the frame-level audio features based on the enhanced textual information. Our proposed method does not require any additional modules or parameters and can be used with any existing CLAP-like ALM to improve zero-shot audio classification performance. We experiment across 18 diverse benchmark datasets and 6 ALMs and show that the PaT outperforms vanilla zero-shot evaluation with significant margins of 0.42%-27.0%. Additionally, we demonstrate that PaT maintains robust performance even when input audio is degraded by varying levels of noise. We make our code publicly available 1.

Footnote 1: Code: [https://github.com/cs20s30/PAT.git](https://github.com/cs20s30/PAT.git)

## 1 Introduction

Advancements in multimodal language models (MLMs) have significantly improved performance across various modalities and applications Turian et al. (2022); Yang et al. (2021). Audio-Language Models Ghosh et al. (2024b), specifically Audio-Language Encoders (ALEs) like CLAP Guzhov et al. (2022); Wu* et al. (2023), are a distinct type of MLMs that learn a shared representation space between the audio and language modalities. Trained on large-scale audio-caption pairs, these models acquire diverse audio concepts during pre-training, enabling them to generalize to new, unseen audio categories. This capability allows ALEs to excel in zero-shot audio classification, accurately categorizing any set of classes described with natural language during inference. Such flexibility is essential for ALEs to effectively adapt to dynamic environments with diverse and unknown sounds.

Significant progress has been made to enhance the zero-shot performance of ALEs across a wide range of audio classification tasks. For instance, Contrastive Language Audio Pretraining (CLAP) Wu* et al. (2023) was one of the first ALE models to achieve notable zero-shot improvements by utilizing descriptive audio captions, as opposed to the class labels used in earlier models like Wav2CLIPWu et al. (2022) and Audio-CLIP Guznov et al. (2022). Additionally, ALEs such as LAION-CLAP Wu* et al. (2023) and MS

Figure 1: Comparison of zero-shot audio classification performance of ALEs (LAION CLAP Wu* et al. (2023) and MS CLAP Elizalde et al. (2023a)) with and without PaT on MUSDB Bertin-Mahieux et al. (2011) (music genre classification). Our proposed training-free method significantly enhances zero-shot performance, even in low-resource domains where the ALEs have limited training data.



CLAP Elizalde et al. (2023) employ vast collections of audio-text pairs, while models like CompA-CLAP Ghosh et al. (2023) leverage complex learning objectives during pretraining to show zero-shot performance gains across various audio classification tasks. While these models show zero-shot improvement, it comes at the additional cost of pre-training them with either more refined learning objectives or by improving both the quality and quantity of audio-text pairs. To overcome these challenges, researchers have attempted to improve ALEs' audio classification capabilities through parameter-efficient transfer learning. Inspired by prompt learning Brown (2020) and adapter fine-tuning approaches Houlsby et al. (2019), Treff Adapter Liang et al. (2023) and Audio Prompt Learner Liang et al. (2023) incorporate learnable prompts or lightweight adapters to adapt both textual and audio features. Although these methods result in performance improvements, they introduce learnable parameters and require additional training phases with few-shot labelled data. On the other hand, training-free methods can utilize the zero-shot capabilities of ALEs, making them more scalable and efficient for real-world applications which can contain diverse and unknown sounds. To the best of our knowledge, there exists no _training-free_ method to improve the zero-shot classification capabilities of such models.

**Our Contribution.** To this end, we propose **PaT**: Parameter-free \({\bf Aud}\)io \({\bf Text}\) Aligner, a simple, parameter-free, and training-free approach for boosting zero-shot audio classification performance in ALEs. We first identify two major issues with current ALEs: 1) Current ALEs rely on simplistic prompts like "The sound of <label>" during zero-shot evaluation, leading to suboptimal performance. 2) During zero-shot transfer, when we align audio and text representations through cosine-similarity, audio representations are typically average pooled, which results in information loss and reduced discriminative power Ruderman et al. (2018); Springenberg et al. (2014). Inspired by our findings, we propose two novel components: _1) Weighted Prompt Ensemble:_ To enhance zeroshot performance for ALEs, we enrich the textual embeddings by creating a task-agnostic prompt datastore comprising 400 unique prompts. These prompts are specifically designed to reduce the distribution shift between the zero-shot setting and the training data, where audio captions typically contain more than a single-word description, unlike previous approaches. Each prompt is generated from handcrafted templates, capable of accommodating unique sound labels while ensuring semantic coherence. Moreover, we show that naively using all the prompts for various downstream applications may not consistently yield the desired performance improvements. To address this, we introduce _weighted prompt ensemble_, a method which computes a score for each prompt using the ALEs' response as feedback without the need for additional labeled data or training. These scores are then used as weights to perform a weighted ensemble of the text embedding associated with a prompt and a class label. _2) Cross Modal Alligner:_ We introduce a _cross-modal aligner_ that enhances audio representations with text-guided information. Specifically, we first compute an attention map using parameter-free attention mechanisms between the frame-level audio representations and the enriched textual representations associated with each label. Next, we use this attention map to perform weighted pooling on the frame-level audio representations. Finally, we compute the cosine similarity between the enriched audio and text representations to perform zero-shot classification. To summarize, our main contributions are:

* We propose PaT, a novel approach to improve zero-shot audio classification performance in a _training-free_ fashion. PaT introduces a cross-modal interaction approach aimed at improving audio-text alignment by enhancing both audio and textual representations in a zero-shot setting.
* We evaluate PaT across multiple ALEs on 18 audio classification datasets and show that PaT achieves 0.42%-27.0% improvement over our baselines.
* We further investigate PaT's robustness to noisy audio to show that PaT consistently outperforms our baselines under varied noise augmentation settings.

## 2 Related Work

**Audio Language Encoders (ALEs).** Previous explorations on developing multimodal encoders to learn shared representations across different modalities have shown significant promise. For example, building on contrastive pre-training techniques from vision-language models like CLIP Radford et al. (2021), audio-language encoders (ALEs) have 

made advances in audio-language understanding, achieving state-of-the-art zero-shot performance across various audio classification tasks. Early efforts, such as Wav2Clip Wu et al. (2022) and AudioCLIP Guzhov et al. (2022), focused primarily on aligning audio representations with labels. More recent approaches, like CLAP Wu* et al. (2023); Elizalde et al. (2023a) have shifted towards explicitly mapping audio representations to textual descriptions, leading to significant improvements in zero-shot performance across diverse audio classification tasks. However, most prior ALEs have relied on compute-intensive strategies, such as refining existing multimodal alignment objectives during pre-training Ghosh et al. (2023); Kim et al. (2024); Ghosh et al. (2024) or increasing the number of parameters and training data Elizalde et al. (2023a). In contrast, training-free approaches to improve zero-shot performance for ALEs remain largely underexplored.

**Downstream Adaptation of ALEs for Audio Classification.** Recent methods have employed various parameter-efficient transfer learning approaches, primarily inspired by prompt learning Brown (2020) and adapter finetuning Houlsby et al. (2019). For instance, Treff Adapters Liang et al. (2023) introduce an additional learnable cross-attention module to enhance cross-modal interaction in few-shot settings. Other approaches, such as Audio Prompt Learners Li et al. (2024) and PALM Hanif et al. (2024), use soft prompting to append learnable prompt tokens to textual representations. Although these methods show improvements in audio classification tasks, they fail to leverage the core advantage of ALEs, which is zero-shot transfer.

## 3 Preliminaries

### Zero-shot transfer in ALEs

Let \(D\) be a downstream audio classification dataset with \(M\) class labels, \(L\leftarrow\{l_{1},\dots,l_{M}\}\). First, we insert these labels into a fixed prompt template \(p\), such as "The sound of <label>," where the "clabel>" token is replaced with each unique class label. This creates \(M\) textual inputs: \(p(L)\gets p(l_{1}),\dots,p(l_{M})\). Next, we use text encoders \(T(\cdot)\) to transform these textual inputs into latent representations: \(T_{p}\gets T_{p}(l_{1}),\dots,T_{p}(l_{M})\). Similarly, for each audio input \(a_{i}\), we use audio encoders \(A(\cdot)\) to obtain frame-level audio representations \(A^{s}_{i}\in\mathcal{R}^{c\times d}\), which we then pool to produce compact representations \(A_{i}\in\mathcal{R}^{1\times d}\). To ensure consis

Figure 2: Illustration of PAT. PAT improves the zero-shot capabilities of ALEs by enriching audio-text representations in a parameter and training-free fashion. PAT consists of two major components: 1_Weighted prompt ensemble_ that first utilizes an in-house generic prompt datastore to transform class labels into diverse textual descriptions, which are then encoded by a text encoder. Further, each prompt is assigned a unique score based on the level of uncertainty it introduces during zero-shot prediction (less uncertainty results in a higher score). A weighted average is then performed to generate task-specific, semantically rich textual representations. 2� Next, the enriched textual representations are used to guide the enhancement of audio representations using a novel zero-shot _cross model alignment_. Precisely, frame-level audio representations are paired with enhanced textual representations to compute a parameter-free attention map, which is used in performing audio and text-guided transformations. Finally, the transformed frame-level audio representations are pooled, and the audio-text-guided information is added to the original logit space, boosting the ALE’s zero-shot prediction capabilities.

 

tent dimensionality between audio and text representations, we pass both through fully connected layers of the same size. Finally, we compare the features using cosine similarity to generate the classification logits: \(\mathcal{L}_{\text{pred}}(a_{i})\gets A_{i}\cdot T^{T}_{p}\)

### Prompt Ensemble

Let \(P\leftarrow\{p_{1},...,p_{N}\}\) be a collection of unique prompts. We first encode each prompt \(p_{i}\) with class labels \(l_{j}\) using a text encoder and then average the textual representations of similar labels: \(T_{\text{avg}}\leftarrow\{\sum_{i=i}^{N}T_{p_{i}}(l_{1})\},...,\sum_{i=i}^{N}T _{p_{i}}(l_{m})\}\). We recognize that one key challenge in applying prompt ensembles to Audio-Language Encoders (ALEs) is the lack of a collection of diverse prompts specifically tailored for zero-shot audio classification.

## 4 Methodology

Fig.2 demonstrates our proposed method PAT. We propose two simple training-free extensions for existing zero-shot transfer learning in ALEs. To enhance textual representation, we introduce a weighted prompt ensemble that selectively identifies and scores prompts which are more relevant to an unseen downstream task. To enhance audio representation, we introduce a cross-modal alignment that utilizes a parameter-free attention mechanism to align frame-level audio representations with textual descriptions in a zero-shot setting. In upcoming subsections, we explain PAT in detail.

### Prompt Datastore

We adopt prompt ensemble for zero-shot audio classification tasks by developing a robust prompt datastore for ALEs. Specifically, our prompt datastore consists of 400 semantically and linguistically diverse prompts. These prompts are designed to minimize the distribution shift between the zero-shot setting and training data, where audio captions typically contain more than a single-word description. We first use GPT-4o [3] to generate 600 prompts by providing diverse templates and examples to guide generation. To ensure semantic diversity in our prompts, we design a variety of prompt-generation templates to avoid vague or overly open-ended prompts, such as "How does the sound <label> occur?". Furthermore, to ensure linguistic diversity, we ask GPT-4o to generate a paraphrased version of the initial set of generated prompts. Finally, we conduct a manual filtering process, selecting 400 prompts by discarding those that are inaccurate or repetitive. Fig. 3 shows a few examples of prompt templates (See Appendix 4 for more examples of prompts generated).

### Weighted Prompt Ensemble

**Motivation.** Through our experiments, we discover that despite the vanilla prompt ensemble method consistently outperforming single prompts like "The sound of <label>', not all prompts in a large prompt datastore \(P\) are equally relevant for unseen downstream tasks. To address this in VLEs, prior work has utilized prompt engineering to create task-specific prompts by analyzing output labels for unseen tasks [5]. However, we argue that such an approach requires additional manual effort and is not scalable across diverse downstream tasks.

In response to this, we propose a Weighted Prompt Ensemble (WPE) algorithm in PAT to adapt prompt ensembles across unseen downstream tasks by scoring individual prompts automatically without requiring additional parameters or training. Our approach is guided by a simple intuition: _Given a set of audio data and class labels, prompts that introduce greater uncertainty in label predictions should receive lower scores, and vice versa_. In line with the prior works [7], we quantify uncertainty in model predictions using _max prediction logits_. As shown in Fig. 2, for a given prompt \(p\) in prompt datastore \(P\), audio representation \(A\), and textual representation

Figure 3: Examples of prompt templates and their paraphrased versions used in the prompt datastore.

 

\(T_{p}\in\mathbb{R}^{M\times d}\), where \(M\), \(d\) are the number of class labels and embedding dimensions respectively, we first compute the prediction logits \(\mathcal{L}_{p}\gets A\cdot T_{p}^{T}\). The prompt score \(w_{p}\) is then determined by calculating a cumulative sum of maximum prediction logits. We formally define this as:

\[w_{p}=\sum_{i=1}^{|D|}\text{max}_{j}\mathcal{L}_{\text{pred}}[i,j],\forall j\in \left\{1,\dots,M\right\} \tag{1}\]

where \(|D|\) is the number of samples in an unseen downstream task \(D\). Finally, we apply softmax normalization to the prompt scores and perform a weighted averaging of the prompt-specific textual representations to obtain enriched textual representation \(T_{\text{avg}}\), as shown below:

\[T_{\text{avg}}=\sum_{p=1}^{N}w_{p}*T_{p} \tag{2}\]

where \(N\) is the number of unique prompts. We summarize the complete process of our weighted prompt ensemble in Algorithm 1

```
Downstream task \(D\); Audio Representations \(A\); Textual Representation, \(T\) for \(N\) prompts with \(M\) class labels; Prompt scores \(w\leftarrow\left\{w_{1},\dots,w_{N}\right\}\); \(N\) no of prompts: \(p\in\left\{p_{1},p_{2},...,p_{\text{N}}\right\}\)for\(p\leq N\)do  Compute prediction logits \(\mathcal{L}_{p}\gets A\cdot T_{p}^{T}\)  Compute max prediction logits: \(\mathcal{L}_{p}^{max}\leftarrow max_{j}\mathcal{L}_{p}[i,j]\).  Compute prompt score \(w_{p}\leftarrow\sum_{i=1}^{|D|}\mathcal{L}_{p,d}^{max}\)  Update prompt score \(w\gets w\cup\left\{w_{p}\right\}\) endfor  Normalize prompt scores \(w\leftarrow\text{Softmax}(w)\)  Perform weighted prompt ensemble: \(T_{\text{avg}}\leftarrow\sum_{p=1}^{N}w_{p}\times T_{p}\)
```

**Algorithm 1** Weighted Prompt Ensemble

### Cross-Modal Alignment Motivation.

We observe that ALEs encode audio and text modalities independently, preventing them from utilizing cross-modal interactions before making final predictions. Additionally, audio representations are typically average-pooled, leading to a loss of fine-grained information in the audio.

Inspired by our findings, we propose parameter-free cross-modal alignment in PAT to improve the audio-text alignment in ALEs in a zero-shot fashion. As shown in Fig. 2, we utilize the intermediate frame-level audio representations \(A^{s}\in\mathbb{R}^{c\times d}\) and enriched textual representations \(T_{avg}\in\mathbb{R}^{M\times d}\) to compute parameter free attention weights \(c_{i,j}\in C\) that denotes the correlation between individual audio frames and class labels. We formulate this below:

\[C=A^{s}\cdot T_{avg}^{T}\in\mathbb{R}^{c\times M} \tag{3}\]

Next, we utilize our parameter-free attention weights \(C\) to enhance cross-modal alignment between the audio and textual representations. To achieve this, we project the original audio representation \(A^{s}\) and the average textual representation \(T_{\text{avg}}\) using \(C\) as follows:

\[\tilde{A^{s}} =\text{Softmax}(C)\cdot T_{\text{avg}}\in\mathbb{R}^{c\times d} \tag{4}\] \[\tilde{T}_{\text{avg}} =\text{Softmax}(C^{T})\cdot A^{s}\in\mathbb{R}^{M\times d} \tag{5}\]

where \(\tilde{A^{s}}\) represents the audio-guided representations that amplify the information of labels strongly associated with the audio, and \(\tilde{T}_{\text{avg}}\) represents the text-guided representations that enhance the information of the audio frames strongly correlated with specific labels. We then pool the transformed frame-level audio representation \(\tilde{A}^{s}\) as follows: \(\tilde{A}\leftarrow\text{AvgPool}(\tilde{A}^{s})\). The transformed representations \(\tilde{A}\) and \(\tilde{T}_{\text{avg}}\) are then used to create audio-guided and text guide logits, \(\mathcal{L}_{\text{audio}}\leftarrow\tilde{A}\cdot T_{\text{avg}}^{T}\) and \(\mathcal{L}_{\text{text}}\gets A\cdot\tilde{T}_{\text{avg}}^{T}\) respectively. Finally, we combine the audio and text-guided logits with the original logits \(\mathcal{L}_{\text{pred}}\gets A\cdot T_{\text{avg}}^{T}\). We formulate this below:

\[\mathcal{L}_{\text{combine}}=\mathcal{L}_{\text{pred}}+\beta_{1}\mathcal{L}_{ \text{audio}}+\beta_{2}\mathcal{L}_{\text{text}} \tag{6}\]

where \(\beta_{1}\) and \(\beta_{2}\) are the hyper-parameter weights for audio and text-guided logits, respectively. Additionally, we provide details on hyper-parameter tuning in Appendix E

## 5 Experimental Setup

### Evaluation Datasets and Metric

For zero-shot evaluation, we utilized 18 open-source audio classification datasets covering a broad range of musical and non-verbal audio types. Specifically, for music-related downstream tasks such as instrument and genre classification, we present results on eight widely-used musical datasets, including NSynth (NSynth Inst/Src) Engel et al. (2017b), Beijing Opera (Beijing Op) Tian et al. (2014), MedleyDB Bittner et al. (2014), 

MUSDB (Rafii et al., 2017), GTZAN (Tzanetakis et al., 2001), and Mridangam Stroke/Tonic (Mri St/ Mi. To) (Turian et al., 2022). Additionally, we report results on non-verbal sound classification tasks using datasets like ESC-50 (Piczak), UrbanSound8K (USD8K)(Salamon et al., 2014), TUT-Urban (TUT)Mesaros et al., 2018), VocalSound (VS)(Gong et al., 2022), SESA(Spadini, 2019), CochlScene (Jeong and Park, 2022), DCASE Task 4 (Mesaros et al., 2017), and Gunshot Triangulation (GT) (Turian et al., 2022). We also conduct an evaluation on multi-label datasets like FSD50K (Fonseca et al., 2021) and AudioSet (Gemmeke et al., 2017). For zero-shot evaluation, we report the mean average precision (mAP) for AudioSet and FSD50K and accuracy for the other datasets, which averaged over 5 runs. Appendix C provides additional dataset details.

### Baselines

We demonstrate the scalability and robustness of the PaT across five open-source audio-language encoders (ALEs). Specifically, we integrate PAT with Wav2CLIP (Wu et al., 2022), trained with 200k audio-label pairs and various CLAP models, including LAION CLAP (L-CLAP) (Wu* et al., 2023), trained on over 633K audio-text pairs; CompA CLAP, which is further trained on 60K compositionally aware audio-text pairs; LAION CLAP MUSC (LM-CLAP), which is further trained on music datasets; MS CLAP-22 (Elizalde et al., 2023b), trained on 128K+ audio-text pairs; and MS CLAP-23 (Elizalde et al., 2023a), which is trained on 4.6M pairs covering diverse audio types such as speech, music, and non-verbal sounds. We re-evaluate the zero-shot accuracy of publicly available ALEs under similar compute settings using A6000 GPUs.

### Audio Augmentations

We further evaluate PaT's robustness in noisy settings by augmenting audio with various kinds of audio augmentations. Specifically, we apply _gaussian noise_ to simulate real-world background noise, _pitch shift_ to test frequency variation, and _polarity inversion_ to check the model's sensitivity to phase changes. Additionally, we use _delay_ to introduce echo effects, _gain_ to assess performance under varying volume levels, and both _low pass_ and _high pass filters_ to evaluate the PaT ability to handle reduced frequency ranges. Lastly, we use _reverb_ to simulate different reverberant acoustic environments. Appendix D provides additional details on various audio augmentations.

## 6 Results and Result Analysis

### Main Results

We summarize the results of PaT applied to 6 different ALEs across 18 audio classification datasets, including 10 non-verbal speech and non-speech

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**Model \(\rightarrow\)** & \multicolumn{2}{c}{**L-CLAP**} & \multicolumn{2}{c}{**LM-CLAP**} & \multicolumn{2}{c}{**MSCLAP-22**} & \multicolumn{2}{c}{**MSCLAP-23**} & \multicolumn{2}{c}{**Wav2CLIP**} & \multicolumn{2}{c}{**CompA**} \\ \cline{2-11}
**Dataset \(\downarrow\)** & ZS & Pat & ZS & Pat & ZS & Pat & ZS & Pat & ZS & Pat \\ \hline  & \multicolumn{11}{c}{ Sound} & & & & & & & & & & \\ \hline EESC-50 & 89.00 & **93.00\({}_{-1.001}\)** & 85.60 & **92.65\({}_{-1.014}\)** & 76.95 & **78.35\({}_{-0.017}\)** & 91.80 & **94.80\({}_{-1.003}\)** & 24.85 & **31.60\({}_{-0.014}\)** & 91.35 & **93.20\({}_{-1.015}\)** \\ USD-8K & 76.00 & **80.00\({}_{-0.000}\)** & 28.09 & **39.93\({}_{-1.173}\)** & 72.54 & **74.80\({}_{-1.027}\)** & 77.70 & **82.50\({}_{-1.000}\)** & 20.97 & **22.69\({}_{-1.775}\)** & 73.53 & **78.32\({}_{-1.975}\)** \\ TUT & 36.00 & **39.00\({}_{-0.000}\)** & 28.09 & **39.93\({}_{-1.113}\)** & 24.44 & **25.61\({}_{-1.014}\)** & 45.00 & **47.00\({}_{-0.000}\)** & 11.54 & **15.18\({}_{-1.014}\)** & 40.21 & **46.26\({}_{-0.018}\)** \\ VS & 78.20 & **80.00\({}_{-0.000}\)** & 74.46 & 78.91\({}_{-1.014}\)** & 43.78 & **54.94\({}_{-1.000}\)** & 79.00 & **79.60\({}_{-0.000}\)** & 22.72 & **24.06\({}_{-1.016}\)** & 65.22 & **71.26\({}_{-0.000}\)** \\ DCASE & 44.88 & **50.81\({}_{-0.000}\)** & **56.76** & 59.44\({}_{-0.000}\)** & 13.93 & **23.77\({}_{-0.000}\)** & 45.90 & **45.96\({}_{-0.000}\)** & 09.63 & **17.21\({}_{-1.015}\)** & 33.20 & **34.29\({}_{-1.015}\)** \\ Gunshot Tri & 10.23 & **22.72\({}_{-0.000}\)** & 13.64 & **29.52\({}_{-1.000}\)** & 17.05 & **23.86\({}_{-0.000}\)** & 25.00 & **25.00\({}_{-0.000}\)** & **25.00\({}_{-0.000}\)** & 25.00 & **26.15\({}_{-1.153}\)** \\ SESA & 67.72 & **44.28\({}_{-0.000}\)** & 72.38 & **79.04\({}_{-0.000}\)** & 66.67 & **68.47\({}_{-0.000}\)** & 70.48 & **71.61\({}_{-1.012}\)** & 29.52 & **56.10\({}_{-0.000}\)** & 64.76 & **69.42\({}_{-0.000}\)** \\ AudioSet & 31.88 & **36.98\({}_{-0.000}\)** & 33.32 & 38.21\({}_{-0.000}\)** & 16.10 & **17.81\({}_{-1.012}\)** & 25.33 & **28.73\({}_{-1.000}\)** & 18.03 & **20.12\({}_{-1.013}\)** & 33.34 & **35.12\({}_{-1.015}\)** \\ PSD50K & 46.45 & **48.76\({}_{-1.014}\)** & 47.12 & **49.10\({}_{-1.014}\)** & 32.50 & **33.80\({}_{-1.000}\)** & 44.49 & **45.52\({}_{-1.003}\)** & 42.31 & **44.14\({}_{-1.014}\)** & 42.18 & **43.22\({}_{-1.016}\)** \\ Cochlscen & 38.56 & **48.66\({}_{-0.000}\)** & 50.66 & 55.35\({}_{-0.000}\)** & 25.94 & **33.51\({}_{-1.000}\)** & 85.00 & **85.22\({}_{-0.000}\)** & 13.09 & **16.11\({}_{-1.015}\)** & 31.95 & **38.21\({}_{-1.015}\)** \\ \hline  & \multicolumn{11}{c}{ Music} & & & & & & & & & & & \\ \hline Beijing Op. & 45.34 & **68.64\({}_{-0.000}\)** & 75.00 & **75.42\({}_{-0.000}\)** & 54.24 & **73.72\({}_{-1.000}\)** & 71.19 & **71.61\({}_{-0.000}\)** & 26.69 & **34.32\({}_{-1.000}\)** & 61.86 & **63.21\({}_{-1.000}\)** \\ GTZAN & 43.40 & **54.20\({}_{-0.000}\)** & 63.92 & **63.93\({}_{-0.000}\)** & 19.19 & **20.75\({}_{-1.000}\)** & 56.24 & **58.56 

sound datasets and 8 musical datasets, in Table 1. Our key finding can be summarized as follows: 1) PAT consistently outperforms the vanilla zero-shot (ZS) approach across all baselines and datasets, achieving an absolute improvement of 0.42%-27%. This underlines PAT strength to generalize across diverse audio classification tasks and scale with different ALEs in a training-free fashion. 2) PAT shows a remarkable performance boost even for less-pretrained ALEs. For instance, LAION-CLAP Wu* et al. (2023) pre-trained on 0.6M audio-text pairs, gains an 11% boost on MUSDB Rafii et al. (2017b) when combined with PAT, even surpassing MS CLAP Elizalde et al. (2023a) by 5%, which has been pre-trained on 4.6M pairs. We attribute such gains to our cross-modal aligner that utilizes enriched textual representations to reweigh audio representations. 3) PAT shows a slight performance degradation on ALEs like Wav2CLIP Wu et al. (2022). As mentioned by Elizalde et al. (2023a), these models are pretrained solely on audio labels rather than textual descriptions, thus limiting them to interpret prompts.

### Ablation on various PAT Components

We conduct extensive ablations on individual Pat components, as shown in Table 2 and Table 3. For all ablation studies, we use the best-performing zero-shot model, MS-CLAP, unless stated otherwise. In Table 2, we compare the performance gains from using Prompt Ensemble (PE) versus Weighted Prompt Ensemble (WPE), which is employed in Pat, averaging the zero-shot accuracy across all datasets. Overall, both PE and WPE outperform the baseline, with WPE consistently outperforming PE even with or without Cross-Modal Alignment (CMA). With this, we show that selectively scoring prompts that reduce uncertainty in model predictions, as opposed to using uniform scoring, leads to improved zero-shot performance. Table 3 further explores CMA by ablating different combinations of original, audio-, and text-guided logits. We show that combining all the logits gives the biggest improvement in zero-shot performance.

### Ablation on Number of Prompts

Fig. 4 shows the effect of increasing the number of prompts when applying the weighted prompt ensemble in Pat. The key findings are: 1) While adding more prompts to the prompt datastore improves zero-shot performance in Pat, the performance of MSCLAP-23 plateaus after reaching 400 prompts. This suggests that naively increasing the number of prompts may not lead to further gains

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Dataset** & **Prompt** & **Score** \\ \hline ESC-50 & The sound of \multicolumn{1}{l}{‘clab’s coming from a cliff edge.} & 0.0035 \\  & A sound of a \multicolumn{1}{l}{‘clab’s coming from a parking lot’} & 0.0033 \\ \hline NSyth Inst & \multicolumn{1}{l}{‘A major sound of a \multicolumn{1}{l}{‘clab’s}} & 0.0038 \\  & A minimal sound of a \multicolumn{1}{l}{‘dabc’s} & 0.0037 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Top two highest scoring prompt by PAT for MSCLAP-23 on ESC-50 and Nsynth-Inst

Figure 4: Measuring MSCLAP-23 zero-shot performance with the prompt ensemble (PE) vs. PAT on NSynth-Src by varying the prompt count. Generally, PAT outperforms PE across different prompt counts.

 

in zero-shot performance for audio classification tasks. 2) The Weighted Prompt Ensemble (WPE) used in PaT consistently surpasses the standard Prompt Ensemble (PE), even with fewer prompts. Table 4 lists the top 2 highest-scoring prompts generated by MSCLAP-23 using PaT on both sound datasets (e.g., ESC-50) and music datasets (e.g., NSynth Inst). We observe that, for sound datasets, prompts associating sounds to random sources like "parking lot" or "cliff edge" receive higher scores. In contrast, for music datasets, prompts describing random audio attributes like "minimal" or "loud" tend to score higher. We extend this analysis in Appendix G, where we show the top 10 best-scoring prompts for each downstream task.

### Zero-Shot Evaluation under Noisy Setting

Table 5 shows the performance improvement of MSCLAP-23 using PaT across 16 audio classification tasks under noisy conditions, compared to the baseline zero-shot scores. In particular, individual audio samples in each task are subjected to five different types of augmentation, which modify key audio features such as pitch, frequency, amplitude, and phase. Under noisy settings, PaT is able to show promising results by achieving an absolute improvement of 0.10-11.15% when compared with simple zero-shot evaluation. This highlights the robustness of PaT to classify out-of-distribution audio samples without the need for additional few-shot data or training. Additionally, we provide results on 3 other audio augmentations: delay, low pass and reverb in Appendix D

## 7 Conclusion

We introduce PaT, a parameter- and training-free approach to improve zero-shot audio classification for audio-language encoders (ALEs). Our studies indicate that current methods that guarantee audio classification improvements are not capable of leveraging the core strength of ALEs, which is zero-shot transfer and often require extensive training or additional parameters to adapt audio and textual representations for an unseen task. To overcome this limitation, PaT offers two key contributions: 1) a novel prompt scoring method, called weight prompt ensemble, which adapts prompt ensembling for unseen audio classification tasks in a training-free fashion, and 2) a cross-modal alignment framework, which uses parameter-free attention to better align audio and textual representation. PaT significantly boosts zero-shot performance in ALEs across various audio classification datasets.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline Dataset & \multicolumn{2}{c}{Gaussian Noise} & \multicolumn{2}{c}{Pitch Shift} & Polarity Inversion & Gain & High Pass \\ \cline{2-9}  & ZS & PAT & ZS & PAT & ZS & PAT & ZS & PAT & ZS & PAT \\ \hline \multicolumn{8}{c}{ Sound} \\ \hline ESC-50 & 91.80 & **94.20\({}_{+.299}\)** & 78.05 & **80.10\({}_{+.209}\)** & 91.85 & **94.40\({}_{+.259}\)** & 92.05 & **94.85\({}_{+.209}\)** & 82.35 & **86.15\({}_{+.399}\)** \\ USD8K & 77.26 & **82.70\({}_{+.449}\)** & 63.61 & **70.31\({}_{+.499}\)** & 77.43 & **82.69\({}_{+.526}\)** & 77.08 & **82.67\({}_{+.599}\)** & 71.12 & **76.77\({}_{+.569}\)** \\ TUT & 44.94 & **45.74\({}_{+.099}\)** & **26.05** & 26.04\({}_{+.099}\)** & 45.68 & **47.34\({}_{+.166}\)** & 38.95 & **41.97\({}_{+.099}\)** & **35.80** & 35.00\({}_{+.098}\) \\ VS & **81.31** & 77.86\({}_{+.549}\)** & **76.61** & 69.64\({}_{+.499}\)** & **78.98** & 78.00\({}_{+.098}\) & 79.00 & **79.44\({}_{+.049}\)** & 74.07 & **76.16\({}_{+.209}\)** \\ DCASE & 38.32 & **42.21\({}_{+.399}\)** & 31.76 & **34.01\({}_{+.229}\)** & 38.93 & **45.69\({}_{+.676}\)** & 43.24 & **45.28\({}_{+.209}\)** & 33.40 & **37.70\({}_{+.398}\)** \\ Gunshot Tri. & **25.00** & **25.00\({}_{+.099}\)** & **25.00\({}_{+.099}\)** & **25.00\({}_{+.099}\)** & **25.00\({}_{+.099}\)** & **25.00\({}_{+.099}\)** & **25.00\({}_{+.099}\)** & **25.00\({}_{+.099}\)** & 19.32 & **22.72\({}_{+.399}\)** \\ SESA & 67.62 & **69.52\({}_{+.109}\)** & 62.86 & **64.76\({}_{+.66}\)** & 67.62 & **69.52** & 68.57 & **69.52\({}_{+.099}\)** & 48.57 & **58.10\({}_{+.099}\)** \\ AudioSet & 30.40 & **31.15\({}_{+.599}\)** & 22.37 & **23.06\({}_{+.099}\)** & **30.40** & 29.22\({}_{+.109}\)** & 28.78 & **30.23\({}_{+.109}\)** & 23.89 & **24.38\({}_{+.099}\)** \\ FSD50K & 44.54 & **45.74\({}_{+.125}\)** & 37.16 & **43.87\({}_{+.019}\)** & 44.39 & **44.96\({}_{+.079}\)** & **44.56** & 43.79\({}_{+.079}\)** & 37.94 & **43.73\({}_{+.399}\)** \\ Cochlscene & **85.07** & 84.36\({}_{+.078}\) & 60.18 & **61.42\({}_{+.126}\)** & 85.07 & **85.17\({}_{+.096}\)** & 81.97 & **82.09\({}_{+.027}\)** & 73.34 & **75.15\({}_{+.197}\)** \\ \hline \multicolumn{8}{c}{Music} \\ \hline Beijing Op. & 70.34 & **70.62\({}_{+.209}\)** & 61.02 & **62.74\({}_{+.1270}\)** & 71.19 & **71.61\({}_{+.025}\)** & 69.49 & **69.61\({}_{+.025}\)** & **65.68** & 64.86\({}_{+.098}\)** \\ GTZAN & 55.77 & **58.53\({}_{+.299}\)** & 47.56 & **50.38\({}_{+.245}\)** & 56.43 & **58.26\({}_{+.197}\)** & 55.20 & **57.79\({}_{+.399}\)** & 47.36 & **50.54\({}_{+.198}\)** \\ MUSDB & **63.60** & 53.60\({}_{+.109}\)** & 58.00 & **61.60\({}_{+.099}\)** & **68.00** & 56.80\({}_{+.109}\)** & **68.00** & 58.40\({}_{+.099}\)** & 46.80 & **55.20\({}_{+.099}\)** \\ Medley & **96.61** & 95.96\({}_{+.099}\)** & 92.09 & **92.46\({}_{+.099}\)** & 95.98 & **96.42\({}_{+.098}\)** & 95.97 & **96.53\({}_{+.099}\)** & **93.37** & 90.94\({}_{+.109}\) \\ Mri. St. & 42.63 & **48.93\({}_{+.099}\)** & 33.15 & **44.94\({}_{+.179}\)** & 44.09 & **47.12\({}_{+.109}\)** & 41.31 & **44.67\({}_{+.366}\)** & 34.73 & **37.30\({}_{+.579}\)** \\ Mri. To. & 24.97 & **26.61\({}_{+.169}\)** & 13.54 & **17.12\({}_{+.356}\)** & 22.02 & **26.18\({}_{+.169}\)** & 19.08 & **26.78\({}_{+.209}\)** & **17.24** & 16.59\({}_{+.065}\)** \\ NSynth Inst & 52.86 & **53.85\({}_{+.499}\)** & 60.11 & **64.77\({}_{+.469}\)** & 63.89 & **66.33\({}_{+.246}\)** 

Acknowledgements

This project is supported in part by NSF#1910940.

## 9 Limitation and Future Work

In this section, we highlight a few limitations and potential future direction of our proposed method, PAT

* Due to compute constraints, we did not evaluate PaT across speech classification tasks. In future, we plan to release PAT evaluation scores against various speech-related downstream tasks such as Keyword spotting, Emotion Recognition, etc.
* We acknowledge that PAT increases the space complexity of existing zero-shot evaluation algorithms by introducing a prompt datastore.
* While PAT shows significant improvements, it is limited by ALE's existing knowledge space to classify unknown sounds. In future, we plan to integrate parameter-efficient methods such as soft prompting to make PaT robust towards the new evolving sounds.

## References

* Anantapadmanabhan et al. (2014) Akshay Anantapadmanabhan, Ashwin Bellur, and Hema A. Murthy. 2014. Mridangam stroke dataset (1.0). 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada. Data set.
* Bertin-Mahieux et al. (2011) Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. 2011. The million song dataset. In _Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011)_.
* Bittner et al. (2014) Rachel M Bittner, Justin Salamon, Mike Tierney, Matthias Mauch, Chris Cannam, and Juan Pablo Bello. 2014. Medlecydb: A multitrack dataset for annotation-intensive mir research. In _ISMIR_, volume 14, pages 155-160.
* Brown (2020) Tom B Brown. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Elizalde et al. (2023a) Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023a. Clap learning audio concepts from natural language supervision. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 1-5. IEEE.
* Elizalde et al. (2023b) Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023b. Clap learning audio concepts from natural language supervision. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 1-5. IEEE.
* Engel et al. (2017a) Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. 2017a. Neural audio synthesis of musical notes with wavenet autoencoders.
* Engel et al. (2017b) Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. 2017b. Neural audio synthesis of musical notes with wavenet autoencoders. In _International Conference on Machine Learning_, pages 1068-1077. PMLR.
* Fonseca et al. (2021) Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. 2021. Fsd50k: an open dataset of human-labeled sound events. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 30:829-852.
* Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_, pages 776-780. IEEE.
* Ghosh et al. (2024a) Breyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024a. Reclap: Improving zero shot audio classification by describing sounds. _arXiv preprint arXiv:2409.09213_.
* Ghosh et al. (2024b) Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024b. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. _arXiv preprint arXiv:2406.11768_.
* Ghosh et al. (2023) Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S Ramaneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2023. Compa: Addressing the gap in compositional reasoning in audio-language models. _arXiv preprint arXiv:2310.08753_.
* 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 151-155.
* Guzhov et al. (2022) Andrey Guzhov, Federico Raue, Jorn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 976-980. IEEE 

Asif Hanif, Maha Tufail Agro, Mohammad Areeb Qazi, and Hanan Aldarmaki. 2024. Palm: Few-shot prompt learning for audio language models. _arXiv preprint arXiv:2409.19806_.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larousilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In _International conference on machine learning_, pages 2790-2799. PMLR.
* Jeong and Park (2022) Il-Young Jeong and Jeongsoo Park. 2022. Cochlscene: Acquisition of acoustic scene data using crowdsourcing. In _2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_, pages 17-21. IEEE.
* Kim et al. (2024) Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. 2024. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 6735-6739. IEEE.
* 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 491-495.
* Liang et al. (2023) Jinhua Liang, Xubo Liu, Haohe Liu, Huy Phan, Emmanouil Benetos, Mark D Plumbley, and Wenwu Wang. 2023. Adapting language-audio models as few-shot audio learners. _arXiv preprint arXiv:2305.17719_.
* Mesaros et al. (2017a) Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. 2017a. Dense 2017 challenge setup: Tasks, datasets and baseline system. In _DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events_.
* Mesaros et al. (2017b) Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. 2017b. Tut acoustic scenes 2017, development dataset. Data set.
* Mesaros et al. (2018) Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. 2018. A multi-device dataset for urban acoustic scene classification. In _Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018)_, pages 9-13.
* OpenAI et al. (2024) OpenAI, Josh Achiam, and Others. 2024. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Piczak (2018) Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In _Proceedings of the 23rd Annual ACM Conference on Multimedia_, pages 1015-1018. ACM Press.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _International conference on machine learning_, pages 8748-8763. PMLR.
* Rafii et al. (2017a) Zafar Rafii, Antoine Liutkus, Fabian-Robert Stoter, Stylianos Ioannis Mimilakis, and Rachel Bittner. 2017a. The MUSDB18 corpus for music separation.
* Rafii et al. (2017b) Zafar Rafii, Antoine Liutkus, Fabian-Robert Stoter, Stylianos Ioannis Mimilakis, and Rachel Bittner. 2017b. The musdb18 corpus for music separation.
* Ruderman et al. (2018) Avraham Ruderman, Neil C Rabinowitz, Ari S Morcos, and Daniel Zoran. 2018. Pooling is neither necessary nor sufficient for appropriate deformation stability in cnns. _arXiv preprint arXiv:1804.04438_.
* Salamon et al. (2014) J. Salamon, C. Jacoby, and J. P. Bello. 2014. A dataset and taxonomy for urban sound research. In _22nd ACM International Conference on Multimedia (ACM-MM'14)_, pages 1041-1044, Orlando, FL, USA.
* Spadini (2019) Tito Spadini. 2019. Sound events for surveillance applications (1.0.0). Data set.
* Springenberg et al. (2014) Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. _arXiv preprint arXiv:1412.6806_.
* Tian et al. (2014) Mi Tian, Ajay Srinivasamurthy, Mark Sandler, and Xavier Serra. 2014. A study of instrument-wise onset detection in beijing opera percussion ensembles. In _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 2159-2163.
* Turian et al. (2022) Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Bjorn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, et al. 2022. Hear: Holistic evaluation of audio representations. In _NeurlPS 2021 Competitions and Demonstrations Track_, pages 125-145. PMLR.
* Tzanetakis et al. (2001) George Tzanetakis, Georg Essl, and Perry Cook. 2001. Automatic musical genre classification of audio signals.
* Wu et al. (2022) Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. 2022. wav2clip: Learning robust audio representations from clip. In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 4563-4567. IEEE.
* Wu* et al. (2023) Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Bilomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP_.
* Yang et al. (2021) Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kwshal Lakhotia, Yiat Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. 2021. Superb: Speech processing universal performance benchmark. _arXiv preprint arXiv:2105.01051_ 

Zihan Zhang and Xiang Xiang. 2023. Decoupling maxlogit for out-of-distribution detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 3388-3397.
* Zhou et al. (2022) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision-language models. _International Journal of Computer Vision_, 130(9):2337-2348.

## Appendix A Appendix

In the appendix, we provide:

1. Section B: Baseline Details
2. Section C: Dataset Details
3. Section D: Audio Augmentations
4. Section E: Hyper-Parameter tuning
5. Section F: Additional Details
6. Section G: Filtering Criteria For Prompt Selection

## Appendix B Baseline Details

**LAION-CLAP**Wu* et al. (2023). This is a contrastive language-audio pretraining (CLAP) model from LAION-AI trained on LAION-Audio-630K Wu* et al. (2023), a large collection of 633,526 audio-text pairs from different data sources. To improve the model's ability to handle audio inputs of variable lengths and boost overall performance, it integrates a feature fusion mechanism and keyword-to-caption augmentation. This enables the model to effectively align and process both audio and text data for enhanced learning.

**LIAION-CLAP Music**Wu* et al. (2023). This is a music-specific version of the LAION-CLAP model. This version is trained both on audio and music, with the LAION-Audio-630K dataset contributing a major portion of its training data. The details of the music-text data being used for training are not specified.

**MS-CLAP 22**Elizalde et al. (2023b). This is a contrastive language-audio pretraining (CLAP) model from Microsoft. This version is trained on 128k audio and text pairs.

**MS-CLAP 23**Elizalde et al. (2023a). This is a follow-up to the MS-CLAP 22, from Microsoft. This version of CLAP uses two innovative encoders and is trained on massive 4.6M audio-text pairs. To learn audio representations, the authors trained an audio encoder on 22 audio tasks instead of the standard training of sound event classification. To learn language representations, they trained an autoregressive decoder-only model instead of the standard encoder-only models.

**Wav2CLIP**Wu et al. (2022). This is a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP) Radford et al. (2021). Wav2CLIP is a model that maps audio into a shared embedding space alongside images and text, enabling multi-modal tasks like zero-shot classification and cross-modal retrieval. It achieves competitive performance on downstream tasks with only about 10% of the data needed by fully supervised models. Additionally, Wav2CLIP is more efficient in pretraining, as it focuses solely on the audio modality and does not require joint training of visual and auditory models, unlike some competing methods.

**CompPA**Ghosh et al. (2023). This is a CLAP model that is trained specifically to enhance its compositional reasoning abilities. The authors introduce improvements to contrastive training by incorporating composition-aware hard negatives, allowing for more precise and focused training. Additionally, they propose a modular contrastive loss designed to help the model learn fine-grained compositional understanding.

## Appendix C Dataset Details

**ESC-50:2**Piczak) The ESC-50 dataset is a labelled collection of 2,000 environmental audio recordings, every 5 seconds in length. The dataset is designed for sound classification tasks and contains recordings organized into 50 semantically distinct classes, with 40 examples per class. These classes are further grouped into 5 major categories, which include Animals, Natural soundscapes & water sounds, Humans, non-speech sounds, Interior/domestic sounds, and Exterior/urban noises.

Footnote 2: [https://github.com/karolpiczak/ESC-50](https://github.com/karolpiczak/ESC-50)

**USD-8K:3**Salamon et al. (2014) The Urban-Sound8K dataset is an audio collection that contains 8,732 labelled sound excerpts, each with a duration of up to 4 seconds. The dataset is designed to represent various urban sound environments, with recordings organized into 10 distinct classes: air\(\_\)conditioner, car\(\_\)horn, children\(\_\)playing, dog\(\_\) 

gun_shot, jackhammer, siren, and street_music. These classes are derived from the urban sound taxonomy

**TUT Urban Acoustic Scenes (TUT):4**Mesaros et al. (2017) The TUT Acoustic Scenes 2019 is a large-scale collection of environmental recordings from various urban sound environments. It consists of 10 classes, each representing a different acoustic scene, such as Airport, Metro station, Park, and Residential area. Each recording is 10 seconds long.

Footnote 4: [https://zenodo.org/records/2589280](https://zenodo.org/records/2589280)

**Vocal Sound (VS):5**Gong et al. (2022) The VocalSound dataset consists of 21,024 crowdsourced recordings representing 6 classes of vocal sounds: laughter, sighs, coughs, throat clearing, sneezes, and sniffs. These recordings were collected from 3,365 unique subjects, providing a diverse set of vocal sound samples. This dataset is primarily used for the study and classification of non-verbal vocal sounds.

Footnote 5: [https://github.com/YuanGongND/vocalsound](https://github.com/YuanGongND/vocalsound)

**DCASE Task 4:6**Mesaros et al. (2017) The DCASE Task 4 dataset is a sound event detection dataset with heterogeneous data and missing labels. It comprises two primary datasets: DESED and MAESTRO. DESED consists of 10-second-long audio clips that are either recorded in domestic environments or synthesized to simulate such environments. These recordings contain annotated sound events from 10 different classes. MAESTRO provides audio recordings with multiple temporally strong annotated events featuring soft labels across 17 classes. However, for the purposes of DCASE Task 4, only 11 classes from MAESTRO are considered.

Footnote 6: [https://dcase.community/challenge2017/](https://dcase.community/challenge2017/)

**Gunshot Triangulation:7**Turian et al. (2022) The Gunshot Triangulation dataset is designed for a novel multiclass classification task involving gunshot recordings. The dataset consists of 88 audio clips representing 22 gunshots from 7 different firearms, recorded in an open field using iPod Touch devices Cooper and Shaw (2020). The objective of the task is to classify the audio based on the specific iPod Touch that was recorded, effectively identifying the location of the microphone during each gunshot event.

Footnote 7: [https://hearbenchmark.com/hear-tasks.html](https://hearbenchmark.com/hear-tasks.html)

**SESA:8**Spadini (2019) The Sound Events for Surveillance Applications (SESA) dataset consists of audio recordings sourced from Freesound. The audio files have durations of up to 33 seconds and are categorized into 4 classes: Casual (not a threat), Gunshot, Explosion, and Siren. This dataset is intended for tasks related to sound event detection and classification in surveillance contexts.

Footnote 8: [https://zenodo.org/records/3519845](https://zenodo.org/records/3519845)

**Cochlscene:9**Jeong and Park (2022)The CochlScene dataset is a crowdsourced collection consisting of 76,000 audio samples from 831 participants, covering 13 different acoustic scenes. Each audio file is 10 seconds in length. This dataset is used for the classification of diverse acoustic environments, providing a wide range of real-world audio scenes.

**AudioSet10**Gemmeke et al. (2017) AudioSet is a large-scale multilabel audio event dataset comprising over 2 million 10-second video clips annotated by humans. It is labeled using a hierarchical ontology of 632 event classes, allowing the same sound to be tagged with multiple labels.

Footnote 9: [https://github.com/cochlearai/cohlsceine](https://github.com/cochlearai/cohlsceine)

**FSD50K11**Fonseca et al. (2021) Freesound Dataset 50k (FSD50K) is an open dataset of human-annotated sound events, featuring 51,197 audio clips from Freesound. The clips are unevenly distributed across 200 classes derived from the AudioSet ontology.

**Beijing Opera:12**Tian et al. (2014) The Beijing Opera Percussion Instrument dataset is a collection of audio examples featuring individual strokes from the four main percussion instruments used in Beijing Opera. These instrument classes include Bangu, Naobo, Daluo, and Xiaoluo. The dataset is designed for the study and classification of traditional Chinese percussion instruments within the context of Beijing Opera.

Footnote 10: [https://research.google.com/audioset/](https://research.google.com/audioset/)

**GTZAN:13**Tzanetakis et al. (2001) The GTZAN Genre dataset is widely used for music genre classification tasks. It contains 1,000 audio tracks, each with a duration of 30 seconds, categorized into 10 distinct genres: Blues, Classical, Country, Disco, Hip-hop, Jazz, Metal, Pop, Reggae, and Rock. Each genre is represented by 100 tracks.

**MUSDB:14**Rafii et al. (2017) The MUSDB18 dataset consists of 150 full-length music tracks, totalling approximately 10 hours of audio, covering various genres. It 

sic source separation tasks. The dataset provides four stem labels: drums, bass, vocals, and others (which include all other non-specific instruments and sounds). Each track is broken down into these stems, allowing for detailed analysis and separation of musical components.

**Medley:15**Bittner et al. (2014) MedleyDB, is a dataset of annotated, royalty-free multitrack recordings. It was curated primarily to support research on melody extraction. For each song melody, f\({}_{0}\) annotations are provided, as well as instrument activations for evaluating automatic instrument recognition. The original dataset consists of 122 multimrack songs, out of which 108 include melody annotations.

Footnote 15: [https://medlegydb.weebly.com/](https://medlegydb.weebly.com/)

**Mridangam Stroke:16**Anantapadmanabhan et al. (2014) The Mridangam Stroke dataset is a collection of 7,162 audio examples featuring individual strokes of the Mridangam, a pitched percussion instrument used in Carnatic music, a sub-genre of Indian classical music. The dataset captures strokes in various tonics and includes 10 different stroke types played on Mridangams.

Footnote 16: [https://compmusic.upf.edu/mridangam-stroke-dataset](https://compmusic.upf.edu/mridangam-stroke-dataset)

**Mridangam Tonic:**Anantapadmanabhan et al. (2014) This dataset is a subset of the Mridangam Stroke dataset. It includes 6 different tonic values associated with the 10 different stroke types played on the Mridangam.

**NSynth Instrument:17**Engel et al. (2017a) NSynth Instrument is a part of the NSynth audio dataset, which consists of 305,979 musical notes, each characterized by unique combinations of pitch, timbre, and envelope. The NSynth Instrument dataset focuses on the task of identifying the high-level instrument family to which each note belongs. These instrument families include categories like keyboard, string, brass, read, and mallet.

Footnote 17: [https://magenta.tensorflow.org/datasets/nsynth](https://magenta.tensorflow.org/datasets/nsynth)

**NSynth Source:**Engel et al. (2017a) NSynth Source is a subset of the NSynth dataset, where the task is to identify the method of sound production for each instrument's note. There are three categories of sound production: acoustic, electronic, and synthetic. Acoustic and electronic labels correspond to instruments recorded from physical sources, while the synthetic label applies to notes generated using digital synthesis.

## Appendix D Audio Augmentations

We use 8 types of audio augmentations in our experiments. We employ torchaudio-augmentations' 18 implementation to augment the audios.

Footnote 18: [https://github.com/Spijkervet/torchaudio-augmentations/](https://github.com/Spijkervet/torchaudio-augmentations/) 



**Delay.** Delay augmentation involves adding a delayed version of the audio signal back onto itself, creating an echo effect. In our experiment, we set the volume factor to 0.5, minimum delay to 200 ms, maximum delay to 500 ms and delay interval to 50 ms.

**Reverb.** Reverb augmentation involves adding reverberation effects to audio signals, simulating the natural reflections of sound in an acoustic environment like a room or hall. In our experiment, we set the minimum reverberance to 0, maximum reverberance to 100, damping factor to 75 and room size to 100.

## Appendix E Hyper-parameter tuning

Table 6 shows the hyper-parameter tuning for audio-guided and text-guided logit weights, \(\beta_{1}\) and \(\beta_{2}\) in PAT. Table 7 shows the list of best-performing hyper-parameters for PAT when applied to MSCLAP-23 across all the evaluation datasets

## Appendix F Additional Details

**Model Parameters:** Among the ALEs that we use, LAION-CLAP and LAION-CLAP Music have \(\approx\) 158M parameters. MSCLAP-22 has \(\approx\) 196M parameters and MSCLAP-23 has \(\approx\) 159M parameters. Wav2CLIP has \(\approx\) 140M parameters. CompA has \(\approx\) 300M parameters.

**Compute Infrastructure:** All our experiments are conducted on one NVIDIA A6000 GPUs. No training is required, and depending on the downstream task, a single inference run on a benchmark requires anywhere between 1 and 5 minutes.

**Implementation Software and Packages:** For our baselines, we use the original GitHub repository provided by the authors: LAION-CLAP 19, CompA-CLAP 20, MS-CLAP 21, Wav2CLIP 22.

Footnote 19: [https://github.com/LAION-AI/CLAP/tree/main](https://github.com/LAION-AI/CLAP/tree/main)

Footnote 20: [https://github.com/Sreyan88/CompA](https://github.com/Sreyan88/CompA)

Footnote 21: [https://github.com/microsoft/CLAP/tree/main](https://github.com/microsoft/CLAP/tree/main)

Footnote 22: [https://github.com/descripinc/lyrebird-wav2clip](https://github.com/descripinc/lyrebird-wav2clip)

**Potential Risks:** To create the prompt datastore, we use GPT4o, which might encode the biases inherent to an LLM. To avoid this, we manually filter the prompt templates generated by LLM.

## Appendix G Filtering Criteria For Prompt Selection

Table 9 shows scores for the top 11 prompts selected during zero-shot evaluation of PAT for each dataset. We refine our prompt datastore by filtering out incorrect and redundant prompts. First, we

\begin{table}
\begin{tabular}{c|c c c c c} \hline \hline Dataset & \multicolumn{2}{c}{Delay} & \multicolumn{2}{c}{Low Pass} & \multicolumn{2}{c}{Reweb} \\ \cline{2-5}  & MS CLAP & PXT & MS CLAP & PXT \\ \hline \multicolumn{5}{c}{Sconal} \\ \hline ESC-50 & 99.10 & **93.65** & 86.30 & **89.80** & 90.00 & **93.25** \\ USDPK & 76.52 & 75.95 & 72.79 & **78.46** & 76.54 & **81.88** \\ TVT & **42.72** & 42.22 & 43.22 & 30.74 & **43.44** & 42.50 & **43.64** \\ VS & 79.39 & **78.53** & **77.47** & 76.10 & **76.74** & 74.52 \\ DCASE & 81.11 & **44.4** & 41.19 & **41.80** & 83.32 & **44.47** \\ Gunstot Tr. & **25.80** & **25.00** & **25.00** & **25.00** & **30.68** & 22.77 \\ SSAK & 65.67 & **65.52** & 65.57 & **65.52** & 65.57 & 70.44 \\ AudioSet & 28.76 & **25.19** & 27.05 & **28.35** & 29.24 & **23.43** \\ PSNORC & at & **43.76** & 41.10 & **43.71** & 43.21 & 43.88 \\ Crollscen & 83.28 & **44.58** & 59.82 & **60.32** & 80.72 & **81.46** \\ \hline \multicolumn{5}{c}{Moistic} \\ \hline Bojing Op. & 65.54 & **70.21** & 64.33 & **65.44** & 71.61 & **72.45** \\ OTZAN & 54.41 & **57.49** & **54.00** & 55.36 & 53.34 & **57.33** \\ MUSGD & **71.29** & 60.00 & **62.40** & **62.00** & **60.20** & 58.80 \\ MGky & **94.90** & **59.34** & **59.34** & 92.57 & **70.27** & **95.96** \\ Ms. Sci. & 45.90 & **46.36** & 37.54 & **44.16** & **35.51** & 25.50 \\ Mo. Te. & 23.66 & **31.72** & 15.96 & **22.38** & 13.49 & **19.33** \\ Nyquist Inst & 59.94 & **61.65** & 35.91 & **57.00** & 56.76 & **59.27** \\ Nysynth Sec & 48.12 & **58.10** & 39.06 & **49.75** & 47.24 & **58.73** \\ \hline \end{tabular}
\end{table}
Table 6: The hyper-parameter tuning for PAT applied on MSCLAP-23.

\begin{table}
\begin{tabular}{l|c c} \hline \hline Dataset & \(\beta_{1}\) (Audio Guided) & \(\beta_{2}\) (Text Guided) \\ \hline ESC-50 & 0.01 & 0.1 \\ \hline USD8K & 0.02 & 1.2 \\ \hline TUT & 0.18 & 2.2 \\ \hline VS & 0.04 & 0.5 \\ \hline DCASE & 0.11 & 3.1 \\ \hline Gunshot Tri & 0.01 & 0.13 \\ \hline SESA & 0.01 & 0.11 \\ \hline AudioSet & 0.01 & 0.01 \\ \hline FSD50K & 0.02 & 0.01 \\ \hline Cochlscene & 0.3 & 2.5 \\ \hline Beijing Op. & 0.13 & 1.8 \\ \hline GTZAN & 0.4 & 3.2 \\ \hline MUSDB & 0.02 & 2.5 \\ \hline Medley & 0.04 & 1.3 \\ \hline Miri. St. & 0.23 & 1.4 \\ \hline Miri. To. & 0.1 & 0.1 \\ \hline NSynth Inst & 0.01 & 0.01 \\ \hline NSynth Src & 0.42 & 2.13 \\ \hline \end{tabular}
\end{table}
Table 7: Best hyper-parameter for PAT when used with MSCLAP-23 across 18 datasets

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline Dataset & \multicolumn{2}{c}{Delay} & \multicolumn{2}{c}{Low Pass} & \multicolumn{2}{c}{Reweb} \\ \cline{2-5}  & MS CLAP & PXT & MS CLAP & PXT \\ \hline \multicolumn{5}{c}{Sconal} \\ \hline ESC-50 & 99.10 & **93.65** & 86.30 & **89.80** & 90.00 & **93.25** \\ USDRK & 75.59 & 72.79 & **78.46** & 76.54 & **81.88** \\ TVT & **42.72** & 42.22 & 43.22 & 30.74 & **43.44** \\ VS & 79.39 & **78.53** & **77.47** & 76.10 & **76.74** & 74.52 \\ PCASE & 81.11 & **44.4** & 41.19 & **41.80** & 83.32 & **44.47** \\ Combined Tri. & **25.80** & **25.00** & **25.00** & **30.68** & 22.77 \\ SSAK & 65.67 & **65.52** & 65.57 & **65.52** & 65.70 & 70.44 \\ AudioSet & 28.76 & **25.19** & 27.05 & **28.35** & 29.24 & **23.43** \\ FSD50K & at & **43.76** & 41.10 & **43.71** & 43.21 & 43.88 \\ CELichensee & 83.28 & **44.58** & 59.82 & **60.32** & 80.72 & **81.46** \\ \hline \multicolumn{5}{c}{Monistic} \\ \hline Bojing Op. & 65.54 & **70.21** & 64.33 & **65.44** & 71.61 & **72.45** \\ OTZAN & 54.41 & **57.49** & **54.00** & 55.36 & 53.34 & **57.33** \\ MCSDP & **71.29** & 60.00 & **62.40** & **62.00** & **60.20** & 58.80 \\ MBCD & **70.49** & **59.34** & **59.34** & 92.57 & **70.62** & **95.96** \\ MoS & **70.90** & **59.38** & **59.34** & 92.57 & **70.62** & **95.96** \\ Ms. Sci. & 45.90 & **46.36** & 37.54 & **44.16** & **35.35** & 25.5

remove prompts that create inconsistent source-attribute associations, such as "The sound of a beach coming from a church" or "A feeble sound of thunder," as they are semantically illogical. This manual filtering ensures meaningful audio-text alignment for zero-shot evaluation. Next, we eliminate repeated prompts, often caused by GPT hallucinations.

 



\begin{tabular}{|l l c|} \hline Dataset & Prompt & Score \\ \hline Beijing Op. & A sound of a <sound>coming from a playground. & 0.00372 \\  & A sound of a <sound>coming from a parade. & 0.00365 \\  & A sound of a <sound>coming from a swimming pool. & 0.00364 \\  & A sound of a <sound>coming from a park. & 0.00363 \\  & A restrained sound of a <sound>. & 0.00362 \\ Beijing Op. & A soft sound of a <sound>. & 0.00362 \\  & The sound of <sound>can be heard near a playground. & 0.00361 \\  & A sound of a <sound>coming from a zoo exhibit. & 0.0036 \\  & A sound of a <sound>coming from a gym. & 0.0036 \\  & A subtle sound of a <sound>. & 0.0036 \\  & A gentle sound of a <sound>. & 0.00359 \\ \hline Cochlsene & An even sound of a <sound>. & 0.00357 \\  & The sound of <sound>can be heard near a rooftop garden. & 0.00357 \\  & A low-key sound of a <sound>. & 0.00355 \\  & A sporadic sound of a <sound>. & 0.00355 \\  & An irregular sound of a <sound>. & 0.00354 \\  & A subdued sound of a <sound>. & 0.00354 \\  & A delicate sound of a <sound>. & 0.00354 \\  & The sound of <sound>can be heard near a lighthouse. & 0.00353 \\  & A moderate sound of a <sound>. & 0.00353 \\  & A quiet sound of a <sound>. & 0.00353 \\  & A gentle sound of a <sound>. & 0.00353 \\ \hline DCASE & A reverberating sound of a <sound>. & 0.00352 \\  & An extensive sound of a <sound>. & 0.00352 \\  & A tremendous sound of a <sound>. & 0.00352 \\  & An overwhelming sound of a <sound>. & 0.00351 \\  & A sound of a <sound>coming from a church. & 0.0035 \\  & The sound of <sound>can be heard near a hedge maze. & 0.00348 \\  & The sound of <sound>can be heard near a cliff edge. & 0.00347 \\  & An enormous sound of a <sound>. & 0.00347 \\  & A massive sound of a <sound>. & 0.00347 \\  & The sound of <sound>can be heard near a fishing pier. & 0.00346 \\  & A vibrant sound of a <sound>. & 0.00346 \\ \hline ESC50 & An extensive sound of a <sound>. & 0.00363 \\  & An unwavering sound of a <sound>. & 0.0036 \\  & A sporadic sound of a <sound>. & 0.0036 \\  & A persistent sound of a <sound>. & 0.0036 \\  & An all-encompassing sound of a <sound>. & 0.00359 \\  & A sound of a <sound>coming from a garden. & 0.00359 \\  & A reverberating sound of a <sound>. & 0.00358 \\  & A sound of a <sound>coming from a barber shop. & 0.00358 \\  & An even sound of a <sound>. & 0.00358 \\  & A continuous sound of a <sound>. & 0.00356 \\  & An extreme sound of a <sound>. & 0.00356 \\ \hline GTZAN & A sound of a <sound>coming from a church. & 0.00379 \\  & The sound of <sound>can be heard near a church. & 0.00374 \\  & A sound of a <sound>coming from a park. & 0.00374 \\  & The sound of <sound>can be heard near a concert hall. & 0.00373 \\ \hline \end{tabular}



A sound of a <sound>coming from a theater.

 A sound of a <sound>coming from a zoo.

 The sound of <sound>can be heard near a fruit orchard.

 The sound of <sound>can be heard near a theater.

 A sound of a <sound>coming from a concert hall.

 An extensive sound of a <sound>.

 The sound of <sound>can be heard near a garden.

 The sound of <sound>can be heard near a forest trail.

 The sound of <sound>can be heard near a suburban neighborhood.

 The sound of <sound>can be heard near a wildlife reserve.

 A sound of a <sound>coming from a sports field.

 A sound of a <sound>coming from a park.

 Gunshot Tri.

 The sound of <sound>can be heard near a golf course.

 The sound of <sound>can be heard near a lake.

 A sound of a <sound>coming from a parking lot.

 A sound of a <sound>coming from a forest.

 The sound of <sound>can be heard near a greenfield.

 The sound of <sound>can be heard near a bridge.

 A gentle sound of a <sound>.

 A minor sound of a <sound>.

 A mild sound of a <sound>.

 A soft sound of a <sound>.

 A restrained sound of a <sound>.

 A feeble sound of a <sound>.

 A subdued sound of a <sound>.

 An even sound of a <sound>.

 A delicate sound of a <sound>.

 A major sound of a <sound>.

 A faint sound of a <sound>.

 A minimal sound of a <sound>.

 A firm sound of a <sound>.

 A resounding sound of a <sound>.

 A muted sound of a <sound>.

 A robust sound of a <sound>.

 A oven sound of a <sound>.

 A soft sound of a <sound>.

 A moderate sound of a <sound>.

 A feeble sound of a <sound>.

 A major sound of a <sound>.

 A gentle sound of a <sound>.

 A animal sound of a <sound>.

 A sound of a <sound>coming from a music box.

 A sound of a <sound>coming from a car wash.

 A sound of a <sound>coming from a dock.

 A sound of a <sound>coming from a river.

 Mridangam Tonic

 A sound of a <sound>coming from a microwave.

 The sound of <sound>can be heard near a vineyard.

 A sound of a <sound>coming from a clock.

 A sound of a <sound>coming from a washing machine.

 A sound of a <sound>coming from a car engine.

 

A sound of a <sound>coming from a toy.

 

A faint sound of a <sound>.

The sound of <sound>can be heard near a beach.

An even sound of a <sound>.

A calm sound of a <sound>.

A sound of a <sound>coming from a hallway.

A soft sound of a <sound>.

A sound of a <sound>coming from a hospital room.

\begin{table}
\begin{tabular}{l l r} A faint sound of a & \textless{sound}\textgreater{} & 0.00365 \\ The sound of & \textless{sound}\textgreater{} can be heard near a beach. & 0.00363 \\ An even sound of a & \textless{sound}\textgreater{}. & 0.0036 \\ A calm sound of a & \textless{sound}\textgreater{}. & 0.0036 \\ A sound of a & \textless{sound}\textgreater{} coming from a hallway. & 0.0036 \\ A soft sound of a & \textless{sound}\textgreater{}. & 0.00359 \\ A sound of a & \textless{sound}\textgreater{}coming from a hospital room. & 0.00359 \\ \hline USD8K & A sound of a & \textless{sound}\textgreater{}coming from a park. & 0.0036 \\  & A subtle sound of a & \textless{sound}\textgreater{}. & 0.0036 \\  & A soft sound of a & \textless{sound}\textgreater{}. & 0.0036 \\  & A mild sound of a & \textless{sound}\textgreater{}. & 0.0036 \\  & A slight sound of a & \textless{sound}\textgreater{}. & 0.00359 \\  & A feeble sound of a & \textless{sound}\textgreater{}. & 0.00358 \\  & A faint sound of a & \textless{sound}\textgreater{}. & 0.00357 \\  & A muted sound of a & \textless{sound}\textgreater{}. & 0.00357 \\  & The sound of & \textless{sound}\textgreater{}can be heard near a university. & 0.00357 \\  & A minimal sound of a & \textless{sound}\textgreater{}. & 0.00357 \\  & A persistent sound of a & \textless{sound}\textgreater{}. & 0.00357 \\ \hline Vocal Sound & A sudden sound of a & \textless{sound}\textgreater{}. & 0.00377 \\  & A sound of a & \textless{sound}\textgreater{}coming from a barber shop. & 0.00377 \\  & A quiet sound of a & \textless{sound}\textgreater{}. & 0.00377 \\  & An even sound of a & \textless{sound}\textgreater{}. & 0.00376 \\  & An abrupt sound of a & \textless{sound}\textgreater{}. & 0.00375 \\  & A gentle sound of a & \textless{sound}\textgreater{}. & 0.00372 \\  & A low-key sound of a & \textless{sound}\textgreater{}. & 0.00372 \\  & A faint sound of a & \textless{sound}\textgreater{}. & 0.00371 \\  & A sporadic sound of a & \textless{sound}\textgreater{}. & 0.00371 \\  & A major sound of a & \textless{sound}\textgreater{}. & 0.0037 \\  & A subtle sound of a & \textless{sound}\textgreater{}. & 0.0037 \\ \hline \end{tabular}
\end{table}
Table 9: Score of Top 11 prompts across various audio classification datasets