Title: Listenable Maps for Zero-Shot Audio Classifiers
Venue: NeurIPS 2024
Accepted: yes
Keywords: Zero shot audio classifiers

Posthoc explanations
Abstract: Interpreting the decisions of deep learning models, including audio classifiers, is crucial for ensuring the transparency and trustworthiness of this technology. In this paper, we introduce LMAC-ZS (Listenable Maps for Zero-Shot Audio Classifiers), which, to the best of our knowledge, is the first decoder-based post-hoc explanation method for explaining the decisions of zero-shot audio classifiers. The proposed method utilizes a novel loss function that aims to closely reproduce the original similarity patterns between text-and-audio pairs in the generated explanations. We provide an extensive evaluation using the Contrastive Language-Audio Pretraining (CLAP) model to showcase that our interpreter remains faithful to the decisions in a zero-shot classification context. Moreover, we qualitatively show that our method produces meaningful explanations that correlate well with different text prompts.

Listenable Maps for Zero-Shot Audio Classifiers

Abstract

Interpreting the decisions of deep learning models, including audio classifiers, is crucial for ensuring the transparency and trustworthiness of this technology. In this paper, we introduce LMAC-ZS (Listenable Maps for Zero-Shot Audio Classifiers), which, to the best of our knowledge, is the first decoder-based post-hoc explanation method for explaining the decisions of zero-shot audio classifiers. The proposed method utilizes a novel loss function that aims to closely reproduce the original similarity patterns between text-and-audio pairs in the generated explanations. We provide an extensive evaluation using the Contrastive Language-Audio Pretraining (CLAP) model to showcase that our interpreter remains faithful to the decisions in a zero-shot classification context. Moreover, we qualitatively show that our method produces meaningful explanations that correlate well with different text prompts.

1 Introduction

The widespread adoption of AI in critical decision-making processes makes interpreting the decisions of deep learning models crucial for ensuring transparency and trustworthiness. Recently, significant research has been devoted to explainable machine learning [1]. These efforts aim to either employ interpretable models or explain the decisions of black-box models using posthoc explanation methods. In the audio domain, however, only a few works exist on interpretable audio classifiers [2, 3, 4] as well as on posthoc explanation methods [5, 6, 7, 8]. The latter contributions are limited to standard closed-set classification and do not explore the challenging topic of interpreting zero-shot classifiers. Zero-shot classifiers, on the other hand, are gaining popularity for their exceptional adaptability, as they define audio classes based on a set of textual prompts [9]. The class labels are not necessarily predefined but can be generated dynamically during inference via natural language. The increased flexibility of zero-shot classifiers comes with a drawback: their predictions are challenging to interpret. This difficulty arises from their multi-modal nature, as learning an interpreter in the joint representation space between text and audio is required. A notable example of a zero-shot classifier is Contrastive Language Audio Pretraining (CLAP) [10], which jointly trains audio and text representations using contrastive learning, that we also work with in this paper.

This paper addresses the problem of posthoc explanations for zero-shot audio classifiers. To the best of our knowledge, this has never been attempted before in the literature. Following the masking idea proposed in [8], we propose LMAC-ZS (Listenable Maps for Audio Classifiers in the ZeroShot context), which consists of a decoder (the interpreter) that outputs a saliency map capable of highlighting the regions within the input audio that trigger the zero-shot classification. We introduce a novel loss function that incentives faithfully following the similarity between the original audio and the corresponding text prompt. Our method provides listenable explanations for linear and non-linear frequency-scale short-time Fourier transform (STFT) representations of audio waveforms. It can also operate on the raw audio domain directly. We applied our explanation method on top of a pretrained version of the popular CLAP [10] by considering different zero-shot classification datasets, including

4

INPUT

TEXT

t

⊤

1

a

1

t

⊤

1

a

2

t

⊤

1

a

3

t

⊤

2

a

1

t

⊤

2

a

2

t

⊤

2

a

3

t

⊤

3

a

1

t

⊤

3

a

2

t

⊤

3

a

3

a

1

a

2

a

3

t

1

t

2

t

3

This

is

a DOG

sound

This

is

a CAT

sound

This

is

a BIRD

sound

Text Encoder

Audio Encoder

t

⊤

1

a

test

t

⊤

2

a

test

t

⊤

3

a

test

a

test

t

1

t

2

t

3

the ESC50 [11], UrbanSound8K [12], as well as versions of ESC50 and UrbanSound8K where different types of contaminations are applied. We show extensive experimental results suggesting that the produced saliency maps correlate well with the corresponding text prompts and faithfully follow the original zero-shot classifier. In particular, our evaluation using various faithfulness metrics highlights that LMAC-ZS is able to provide explanations that are highly relevant to the decisions made by the CLAP model in the zero-shot context. Our method significantly outperforms traditional approaches such as GradCAM++ [13], highlighting their inefficiency in challenging tasks such as zero-shot audio classification.

In summary, our contributions are the following:

We propose a new method, LMAC-ZS, to explain zero-shot audio classifiers.

Weshow that LMAC-ZS maintains faithfulness to the CLAP predictions across diverse zero-shot scenarios.

We qualitatively show that LMAC-ZS produces meaningful explanations for different text prompts.

1.1 Related Work

Posthoc explanation methods aim to explain the decisions of pretrained neural networks. Several works exist on producing posthoc explanations with gradient-based approaches in the computer vision literature. These include the standard saliency method [14], GradCAM [15], GradCAM++ [13], SmoothGrad [16], Integrated Gradients (IG) [17], and several others. However, as suggested in [18], these methods often fail to follow the classifier very faithfully and tend to be insensitive even to random model weights. Another category of post-hoc explanation methods in computer vision generates explanations by applying masks to the input data. Key approaches in this category include [19, 20, 21, 22], which use optimization-based techniques to learn and generate these masks. There also exists a series of works that are most closely related to this paper, where a decoder is trained to produce explanations. Notable attempts in this vein include Dabkowski and Gal (2017) [23], Fan et al. (2017) [24], Zolna et al. (2020) [25], and Phang et al. (2020).

In the audio domain, several post-hoc explanation methods exist. These methods employ various techniques such as layer-wise relevance propagation [26], guided backpropagation [27], and LIME [28, 29, 6, 30]. More recent posthoc explanation methods that use a decoder to produce masks on spectrograms include Listen-to-Interpret [5], which uses a Non-Negative Matrix Factorization [31] based decoder to produce non-negative saliency maps. Other examples include Posthoc Interpretation via Quantization [7], which trains a VQ-VAE [32]-based decoder as an explanation module, and Listenable Maps for Audio Classifiers [8], which trains a decoder using a classification loss to promote faithfulness. These works are not directly applicable to zero-shot classification as they require a predefined set of labels to train the interpreter. In this paper, our goal is to produce explanations in a true zero-shot fashion. To achieve that, we train our decoder on the same data as the CLAP model (without using class labels that we will later test on). Subsequently, LMAC-ZS can produce explanations for arbitrary labels, encoded as natural language. This includes labels not previously seen during the training of the interpreter.

2 Preliminaries

In this Section, we first present the learning methodology for audio-text cross-modal representations in Section 2.1. Then, we introduce masking-based posthoc explanations in Section 2.2.

X

i

InputTf

INPUT TEXT

f

audio

(

.

)

h

i

f

text

(

.

)

t

j

M

θ

(

., .

)

M

InputTf

f

audio

(

.

)

L

ZS

(

θ

)

0

100

200

300

400

0

100

200

300

400

500

Listenable

Interp.

0

100

200

300

400

0

100

200

300

400

500

X

i

ISTFT

(

M

⊙

X

i

, X

phase

)

2.1 Contrastive Learning of Audio-Text Cross-Modal Representations

The goal of learning audio-text cross-modal representations is to create a joint latent space between text and audio. CLAP (Contrastive Language-Audio Pretraining) [10], achieves this via contrastive learning. That is, the similarity between the latent representations of a text and audio signal is maximized if they form a pair, otherwise this similarity is minimized. More specifically, consider X t and X a as batches of text and audio data, respectively. Within the CLAP model, the latent representation is derived by passing the text and audio through their respective encoders, denoted as g t ( . ) and g a ( . ) . This process produces the text and audio latent representations, denoted as L text = g text ( X text ) and L audio = g audio ( X audio ) , respectively. Here, L text is a matrix of dimensions R N × T , where N is the batch size and T represents the latent dimensionality of text. Similarly, L audio is a matrix of dimensions R N × A , where A denotes the latent dimensionality of audio. CLAP trains a joint latent space by passing L text and L audio through fully-connected layers such that,

where MLP ( . ) denotes the multi-layer perceptron transformation layers. The matrix t ∈ R N × d and a ∈ R N × d respectively denote the text and audio latent variables with the same latent dimensionality d . As a shorthand for the rest of the paper we will denote the combination of encoders and the MLP with f text ( . ) := MLP text ( g text ( . )) and f audio ( . ) := MLP audio ( g audio ( . )) for text and audio, respectively. The model aims to maximize the diagonal entries on the matrix C = ta ⊤ . The matrix C ∈ R N × N represents audio-text pairings. The diagonal elements C i,i correspond to positive samples, while other elements are negative samples. This translates into the following training loss function:

where Softmax t ( . ) and Softmax a ( . ) respectively denote Softmax functions along text and audio dimensions, τ is a temperature scaling parameter, and the C i,i denotes the diagonal elements of the C matrix. We show the training forward pass pipeline in the left panel of Figure 1.

We would like to note that with this framework, the zero-shot classification amounts to calculating the similarity of the representation of a given audio with a set of text prompts, each corresponding to a class labels. Namely, the classification decision is taken as:

where ̂ c is the zero-shot classification decision, a test is the embedding for the test audio, and t j is the text embedding corresponding to the label of class j (represented via prompt j ). We show the pipeline of zero-shot classification in the right panel of Figure 1.

2.2 Saliency Maps For Fixed Set Audio Classifiers

In this work, we adopt a posthoc explanation method that uses a learnable decoder, following the masking idea introduced in L-MAC [8]. Before we delve into how to generate a saliency map for a zero-shot classifier, we first explain how L-MAC produces a saliency map within the context of a standard classification setup. The loss function that is minimized during training in L-MAC to obtain faithful saliency maps is defined as follows:

The first term in this loss function aims to maximally align the classifier prediction ̂ y = arg max c f c ( X ) , with the classifier output obtained after masking the input, i.e. the logit f ( M θ ( h ) ⊙ X ) ∈ R N C , where N C is the number of classes. Note that CrossEnt(.) denotes the CrossEntropy loss function. The decoder network M θ ( h ) takes in the classifier representations h (which can consist of representations of several layers) and produces a mask (with values within the interval [0 , 1] and same size as the input) that is element-wise multiplied with the input X . A regularization term that consists of an L 1 loss is also used to prevent trivial solutions, such as a mask with all values set to 1. The mask-out term -CrossEnt ( ̂ y, f ((1 -M θ ( h )) ⊙ X )) minimizes the relevance of the mask-out portion to the predicted class ̂ y . In the next section, we introduce our framework for explaining zero-shot classifiers (that we have defined in Section 2.1), which again applies a mask to the input to replicate the original text-audio similarities.

3 Saliency Maps for Zero-Shot Audio Classifiers

Similarly to the method introduced in L-MAC [8] and summarized in Section 2.2, our goal is to generate explanations that faithfully follow the model. However, in the context of zero-shot classifiers, we do not have a model that outputs a fixed number of logits. Hence, we need a different loss function that promotes faithfulness between the explanations and the zero-shot audio classifier, which relies on similarities to make its decisions. We denote the similarity between the i -th text prompt and j -th audio recording with C i,j as,

Our methodology is based on obtaining a saliency map such that the text-audio cross-modal similarity matrix C is maximally preserved after masking the important parts of the spectrogram. In other words, we learn a decoder such that, after masking the audio, the similarity with text prompts within the batch is maximally preserved. To this end, we define the loss function as follows:

Here, the first term aims to minimize the discrepancy between the original similarities C i,j and the similarities after masking the input audio X audio ,j ∈ R T × F using the decoder M θ ( t i , h j ) , which outputs a mask of shape T × F . Importantly, the decoder is conditioned on both the text representation t i = f text ( X text ,i ) that corresponds to the i -th text prompt in the batch, and the representations h j , which includes the last 4 representations obtained from the audio encoder f audio ( X audio ,j ) . λ 1 , λ 2 are tradeoff parameters.

The second term in Equation 6 promotes sparsity in the generated mask to avoid trivial solutions. Finally, the last term D ( . ) aims to increase the diversity of masks generated for a given audio when conditioned on different text prompts. It is defined as:

̸

The goal of this term is to align the uni-modal similarity between text embeddings t i , t j with the uni-modal similarity between the corresponding audio embeddings f audio ( X audio ,i ⊙ M θ ( t i , h i ) ) , f audio ( X audio ,i ⊙ M θ ( t j , h i ) ) , obtained from the corresponding masked spectrograms. The intuition

is that the similarity between two text prompts should be reflected in the similarity of the audio embeddings from the corresponding masked spectrograms: the farther the text prompts, the farther apart should be the corresponding audio embeddings from masked spectrograms, and thus, the more different the masks should be. We show the effectiveness of this term on diversity with respect to different text prompts in Section B. The overall pipeline is shown in Figure 2.

Producing Listenable Explanations: Our method employs its masking in the linear Short-Time Fourier Transform (STFT) domain, and therefore generating listenable explanations through the inverse-STFT is possible. The listenable explanation is obtained through the following operation,

where both the explanation mask M and the input audio X are in the linear-scale STFT domain, and X phase is the phase of the original input audio. This operation is also shown in Figure 2.

4 Experiments

4.1 Metrics

To evaluate our method, we employ faithfulness metrics previously used in the audio interpretability literature for standard classification setups. We adapt such metrics to the zero-shot scenario by using the class prediction probabilities defined by audio-text similarities such that

where p ( c = j ) is the probability of predicting the class that corresponds to the j -th text prompt and N c is the total number of text prompts used in the zero-shot setting. Analogously to CLAP [10], to create prompts that correspond to the predefined classes in ESC50 [11] and UrbanSound8K [12], we augment the class labels with the prefix 'this is the sound of' , obtaining prompts such as 'this is the sound of baby crying' , 'this is the sound of cat' . When computing all the metrics for LMAC-ZS, we conditioned the decoder on the text prompt that corresponds to the model prediction ̂ c = arg max t ⊤ j a test .

Faithfulness on Spectra (FF): Introduced in [5], it assesses the importance of the provided explanation for the classifier. The metric is calculated by measuring how much does a class-specific prediction probability drops after removing the explanation signal from the original. It is defined as

where ̂ c is the class prediction given by the classifier. High faithfulness values mean that the maskedin portion of the input spectrogram X is highly influential for the classifier decision of the predicted class ̂ c . We report the average faithfulness over all examples by reporting the average quantity FF = ∑ n 1 N FF n . Larger is better.

Average Increase (AI): Introduced in [13], it measures the increase in confidence for the masked-in portion of the explanation, and it is calculated as follows:

where [ . ] , is the indicator function, which is one if the argument is true, and zero otherwise. For this metric, larger is better.

Average Drop (AD): Introduced in [13], it measures the decrease in model confidence when the input image is masked, and it is calculated as follows:

For this metric, smaller is better.

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Similarity

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Mask mean

LMAC-ZS

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Similarity

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Mask mean

GradCam++

1

2

3

4

5

6

ConvBlock ID

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

SSIM

Model Randomization Test - SSIM

LMAC-ZS ESC50

GradCam++ ESC50

LMAC-ZS US8K

GradCam++ US8K

Average Gain (AG): Introduced in [33], it measures the increase in confidence after masking the input image. It is calculated as follows (larger is better):

Input Fidelity (Fid-In): Introduced in [7], it measures whether the classifier outputs the same class prediction on the masked-in portion of the input image. It is defined as the following and the larger is better,

Sparseness (SPS): Introduced in [34], it measures whether only values with large predicted saliency contribute to the prediction of the neural network. Larger values indicate more sparse/concise saliency maps. We use the implementation from the Quantus library [35].

Complexity (COMP): Introduced in [36], it measures the entropy of the distribution of contributions from each feature to the attribution. Smaller values indicate less complex explanations. We again use the implementation from the Quantus library.

4.2 Experimental Setup

We use the official pretrained CLAP [10] weights 2 to perform zero-shot classification on ESC50 [11] and UrbanSound8K [12] datasets. We train LMAC-ZS on the datasets on which CLAP had been trained (namely, Clotho [37], FSD50K [38], AudioCaps [39], and MACS [40] which are publicly available). We also explored training LMAC-ZS only on Clotho to simulate the case where the computational budget is limited. The models were trained on a single NVIDIA RTX 3090 GPU. For the LMAC-ZS model that is trained on the Clotho dataset, we did 2 epochs on the complete dataset, for which an epoch approximately takes an hour. For the Full CLAP data we did 2 epochs as well, and an epoch takes around 4 hours. We quantitatively test whether LMAC-ZS follows the zero-shot classifier on In-Domain (ID) and Out-of-Domain (OOD) settings. For the In-Domain setting, we perform zero-shot classification on clean audio from ESC50 and UrbanSound8k and then produce explanations for the classifications using LMAC-ZS. We would like to emphasize that LMAC-ZS has only been trained on the training datasets for CLAP, and has not been fine-tuned on ESC50 or UrbanSound. For the Out-of-Domain setting, we contaminate the audio with various noise sources at 3dB Signal-to-Noise Ratio (other audio from the same dataset, white-noise, and human speech from the LJ-Speech [41] dataset).

We explore masking in the Mel-domain to explore the case where we produce explanations directly in the feature space on which CLAP operates. For Mel-domain we used 44.1kHz data on which the CLAP model is trained. We also explore masking in the linear frequency-scale log power-STFT

domain to be able to provide listenable explanations. For STFT domain filtering we worked with 16kHz data. We would like to note that this results in slight changes in zero-shot classification accuracies, which are reported in the Tables 1, 2, 3. We trained LMAC-ZS with a batch size of 2 using the Adam optimizer [42] with a learning rate of 1e -5. The decoder consists of a series of transposed convolutions to upsample from CNN14 [43] CLAP representations and incorporates text conditioning by using cross-attention similar to that used in Stable Diffusion [44]. The implementation is done using the SpeechBrain toolkit [45, 46] and it can be accessed through 3 .

4.3 Quantitative Comparison

We compare LMAC-ZS with popular gradient-based saliency map methods including GradCam [15], GradCam++ [13], SmoothGrad [16], and Integrated Gradients (IG) [17]. We apply these saliency map methods using only the CNN14 audio representations. The class logit with respect to which the class activation map for these methods is calculated is picked by using the zero-shot classification decision ̂ c = arg max j t ⊤ j a test .

In Table 1, we compare the faithfulness of the explanations obtained on In-Domain data, where we performed zero-shot classification on clean ESC50 and US8k recordings. We observe that on ESC50 with Mel-Domain masking, LMAC-ZS obtains better AI, AD, AG, FF, and Fid-In values. We observe a similar trend for AI, AD, and AG with STFT-domain masking also, while FF values are comparable. On the UrbanSound8K dataset, we also observe that in terms of AI, AD, and AG the best results are obtained with LMAC-ZS trained with the Full CLAP training datasets. In terms of mask sparseness (SPS) and Complexity (COMP) in most cases, the best results are obtained with the proposed model.

In Table 2, we compare the faithfulness of the explanations obtained on ESC50 samples contaminated with three different types of background noises. We observe that with Mel-Masking, LMAC-ZS

reaches better performance in terms of AI, AD, AG, and very comparable numbers in terms of Fid-In. We also observe that in terms of Sparsity and Complexity LMAC-ZS yields better masks in the Mel Domain. In the STFT domain except for LJ-Speech contamination, we observe that LMAC-ZS obtains better performance in terms of AI, AD, and AG. We would like to note that GradCAM++ obtains better FF numbers in general, but we note that GradCAM++ mask areas are larger as shown in the last column with MM. We also observe similar trends for the explanations obtained on US8K samples contaminated with various background noises shown in Table 3. Another point to note is that in general LMAC-ZS trained on the full CLAP training set yields better performance. However, we observe that training LMAC-ZS only on the Clotho dataset yields to comparable or better performance (e.g. ESC50, Mel, white noise contamination). This shows that, in situations where there is limited access to computational resources, training only on Clotho can produce faithful explanations. We furthermore compare the effect of changing the size of the training set size for the interpreter in Appendix C.

4.4 Qualitative Comparison and Sanity Checks

We provide some qualitative examples of generated explanations in Figure 4, and compare with GradCAM++ which seems to provide the most faithful explanations among the baselines according

0

100

200

300

400

500

600

0

100

200

300

400

500

Input 'Cat'

LMAC-ZS

0

100

200

300

400

500

600

0

100

200

300

400

500

Explain 'Cat'

0

100

200

300

400

500

600

0

100

200

300

400

500

Explain 'Glass breaking'

0

100

200

300

400

500

Input 'Cat'

GradCAM++

0

100

200

300

400

500

Explain 'Cat'

Sim=0.71

0

100

200

300

400

500

Explain 'Glass breaking'

Sim=-0.03

0

100

200

300

400

500

600

0

100

200

300

400

500

600

0

100

200

300

400

500

600

to the Tables 1, 2, and 3. We see that LMAC-ZS generates explanations that are much more sensitive to the similarity between the text prompt and the input audio. For instance in LMAC-ZS explanations we see that if there exists a large similarity between the text prompt and the input audio, the mask correctly highlights relevant portions of the input spectrogram. Also, we see that if the similarity between the input and the text prompt is small then the mask tends not to highlight any areas as expected. For instance in Figure 4, we see for the input recordings that corresponds to a 'Cat', both LMAC-ZS and GradCAM++ return reasonable explanations. However, when we prompt LMAC-ZS for an unrelated prompt (e.g. 'Glass Breaking' in this case), it correctly returns an empty explanation mask, as it is impossible to explain. On the contrary, when GradCAM++ returns a class activation map corresponding to the class "Glass Breaking," we observe that the explanation remains unchanged.

To measure the correlation between the mask mean and similarity, Figure 3 presents a scatter plot depicting the relationship between the similarity of the input text prompt and audio. For LMAC-ZS, we observe that explanations are appropriately returned as empty (indicating small Mask-Means) when the similarity score, estimated using CLAP embeddings, is low. Whereas for GradCAM++, the mask mean and similarity appear to be independent of each other.

Finally, we conduct a cascading model randomization sanity check [18] to assess the sensitivity of explanations returned by LMAC-ZS to the CLAP weights. As illustrated in Figure 3, after three layers of randomization, the similarity drastically decreases for LMAC-ZS, while it remains constant for GradCAM++. We visualize these explanations in Figure 5 and provide additional samples in Appendix A.2. More qualitative samples are available through our companion website 4 .

5 Limitations and Societal Impact

Limitations : Our current implementation focuses on fixed-length audio for simplicity. However, the core methodology of LMAC-ZS can be extended to handle variable-length inputs. Additionally, while this work employs standard faithfulness metrics that analyze the dominant class contribution, LMACZS allows for investigating contributions from the top k classes. Studying the top k contributions to faithfulness could provide further insights into the model's decision-making process. Lastly, our study is limited to the CLAP model, primarily selected for its widespread adoption within the field. It is worth mentioning that there is limited availability of alternatives. For instance, most alternative models such as LAION CLAP [47] are still variations of CLAP, offering minimal differences in their core architecture.

Societal Impact : We believe this research has the potential for societal benefits, particularly in healthcare applications. While this work does not directly target medical diagnosis, improved explainability of audio classifiers for speech pathologies could make them more trustworthy and accepted by medical professionals. We do not see direct negative societal impacts from this research.

6 Conclusions

This paper, to the best of our knowledge, represents the first attempt to develop a model specifically designed for interpreting the decisions of pre-trained zero-shot audio classifiers. In particular, we introduce LMAC-ZS, a novel post-hoc explanation method employing a specialized decoder that generates saliency maps highlighting the regions of the audio input that most contribute to the model predictions. Extensive evaluations highlighted that LMAC-ZS effectively generates explanations that closely align with the decisions made by the CLAP model in zero-shot settings. Our quantitative and qualitative comparisons show that LMAC-ZS outperforms or is comparable to the most popular baseline saliency methods on most quantitative faithfulness metrics. Additionally, LMAC-ZS offers the possibility of being prompted for an explanation. This ability is missing in traditional methods and allows users to gain further insights into the decision-making processes conducted by the model.

References

Christoph Molnar. Interpretable machine learning, 2022.

Pablo Zinemanas, Martín Rocamora, Marius Miron, Frederic Font, and Xavier Serra. An interpretable deep learning model for automatic sound classification. Electronics , 10, 2021.

Pablo Alonso-Jiménez, Leonardo Pepino, Roser Batlle-Roca, Pablo Zinemanas, Dmitry Bogdanov, Xavier Serra, and Martín Rocamora. Leveraging pre-trained autoencoders for interpretable prototype learning of music audio. In ICASSP Workshop on Explainable AI for Speech and Audio (XAI-SA) , 2024.

Luca Della Libera, Cem Subakan, and Mirco Ravanelli. Focal modulation networks for interpretable sound classification. In ICASSP Workshop on Explainable AI for Speech and Audio (XAI-SA) , 2024.

Jayneel Parekh, Sanjeel Parekh, Pavlo Mozharovskyi, Florence d'Alché-Buc, and Gaël Richard. Listen to interpret: Post-hoc interpretability for audio networks with NMF. In International Conference on Neural Information Processing Systems (NeurIPS) , volume 35, 2022.

Verena Haunschmid, Ethan Manilow, and Gerhard Widmer. audioLIME: Listenable explanations using source separation. In International Workshop on Machine Learning and Music , 2020.

Francesco Paissan, Cem Subakan, and Mirco Ravanelli. Posthoc interpretation via quantization. arXiv preprint arXiv:2303.12659 , 2023.

Francesco Paissan, Mirco Ravanelli, and Cem Subakan. Listenable Maps for Audio Classifiers. In International Conference on Machine Learning (ICML) , 2024.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2023.

Karol J. Piczak. ESC: Dataset for environmental sound classification. In Annual ACM Conference on Multimedia , 2015.

J. Salamon, C. Jacoby, and J. P. Bello. A dataset and taxonomy for urban sound research. In ACM International Conference on Multimedia , 2014.

Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In IEEE Winter Conference on Applications of Computer Vision (WACV) , 2018.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop Track , 2014.

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradientbased localization. International Journal of Computer Vision , 2019.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. SmoothGrad: removing noise by adding noise. In ICML Workshop on Visualization for Deep Learning , 2017.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML) , 2017.

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In International Conference on Neural Information Processing Systems (NeurIPS) , volume 31, 2018.

Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In IEEE International Conference on Computer Vision (ICCV) , 2017.

Ruth Fong and Andrea Vedaldi. Net2Vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized input sampling for explanation of black-box models. In British Machine Vision Conference (BMVC) , 2018.

Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representations (ICLR) , 2019.

Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In International Conference on Neural Information Processing Systems (NeurIPS) , 2017.

Lijie Fan, Shengjia Zhao, and Stefano Ermon. Adversarial localization network. In Learning with limited labeled data: weak supervision and beyond, NeurIPS Workshop , 2017.

Konrad Zolna, Krzysztof J. Geras, and Kyunghyun Cho. Classifier-agnostic saliency map extraction. In AAAI Conference on Artificial Intelligence , volume 33, 2019.

Sören Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Robert Müller, Sebastian Lapuschkin, and Wojciech Samek. AudioMNIST: Exploring explainable artificial intelligence for audio analysis on a simple benchmark. Journal of the Franklin Institute , 361, 2024.

Hannah Muckenhirn, Vinayak Abrol, Mathew Magimai-Doss, and Sébastien Marcel. Understanding and Visualizing Raw Waveform-Based CNNs. In Proc. Interspeech , 2019.

Saumitra Mishra, Bob L. Sturm, and Simon Dixon. Local interpretable model-agnostic explanations for music content analysis. In International Society for Music Information Retrieval Conference (ISMIR) , 2017.

Saumitra Mishra, Emmanouil Benetos, Bob L. Sturm, and Simon Dixon. Reliable local explanations for machine listening. In International Joint Conference on Neural Networks (IJCNN) , 2020.

Shreyan Chowdhury, Verena Praher, and Gerhard Widmer. Tracing back music emotion predictions to sound sources and intuitive perceptual qualities. In Sound and Music Computing Conference (SMC) , 2021.

Daniel D. Lee and H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature , 401, 1999.

Aaron Van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In International Conference on Neural Information Processing Systems (NeurIPS) , 2017.

Hanwei Zhang, Felipe Torres, Ronan Sicre, Yannis Avrithis, and Stephane Ayache. Opti-CAM: Optimizing saliency maps for interpretability. arXiv preprint arXiv:2301.07002 , 2023.

Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise explanations of neural networks using adversarial training. In International Conference on Machine Learning (ICML) , volume 119, 2020.

Anna Hedström, Leander Weber, Daniel Krakowczyk, Dilyara Bareeva, Franz Motzkus, Wojciech Samek, Sebastian Lapuschkin, and Marina Marina M.-C. Höhne. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research , 24(34), 2023.

Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and aggregating feature-based model explanations. In International Joint Conference on Artificial Intelligence (IJCAI) , 2020.

Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: an audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019.

Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30, 2022.

Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.

Irene Martín-Morató and Annamaria Mesaros. What is the ground truth? reliability of multiannotator data for audio tagging. In European Signal Processing Conference (EUSIPCO) , 2021.

Keith Ito and Linda Johnson. The lj speech dataset. https://keithito.com/ LJ-Speech-Dataset/ , 2017.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2020.

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Mirco Ravanelli, Titouan Parcollet, Adel Moumen, Sylvain de Langen, Cem Subakan, Peter Plantinga, Yingzhi Wang, Pooneh Mousavi, Luca Della Libera, Artem Ploujnikov, Francesco Paissan, Davide Borra, Salah Zaiem, Zeyu Zhao, Shucong Zhang, Georgios Karakasidis, SungLin Yeh, Pierre Champion, Aku Rouhe, Rudolf Braun, Florian Mai, Juan Zuluaga-Gomez, Seyed Mahed Mousavi, Andreas Nautsch, Xuechen Liu, Sangeet Sagar, Jarod Duret, Salima Mdhaffar, Gaelle Laperriere, Mickael Rouvier, Renato De Mori, and Yannick Esteve. Opensource conversational ai with SpeechBrain 1.0, 2024.

Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: A general-purpose speech toolkit. arXiv preprint arXiv:2106.04624 , 2021.

Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keywordto-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP , 2023.

Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 24-25, 2020.

Peijie Chen, Qi Li, Saad Biaz, Trung Bui, and Anh Nguyen. gscorecam: What objects is clip looking at? In Computer Vision - ACCV 2022: 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV , page 588-604, Berlin, Heidelberg, 2023. Springer-Verlag.

A Appendix / supplemental material

A.1 Results on UrbanSound8K Dataset with Contaminations

A.2 Qualitative Analysis of Model Randomization Test

Figure 5 presents a qualitative visualization of Model Randomization Test results for GradCAM++ and LMAC-ZS.

A.3 Qualitative Comparison with GradCAM++

Figures 6, 7 show an additional sample for the quality of the explanations on spectra.

LMAC-ZS

GradCAM++

500

400

300

200

100

0

500

400

300

200

100

0

0

0

Input 'Toilet flushing'

100

400

500

600

200

300

Input 'Toilet flushing'

100

200

300

400

500

600

Cascading randomization (Conv Block ID)

0

100

200

300

400

500

600

0

100

200

300

400

500

LMAC-ZS

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

GradCAM++

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

0

100

200

300

400

500

600

0

100

200

300

400

500

Input 'Dog'

LMAC-ZS

0

100

200

300

400

500

600

0

100

200

300

400

500

Explain 'Dog'

Sim=0.59

0

100

200

300

400

500

600

0

100

200

300

400

500

Explain 'Rain'

Sim=-0.10

0

100

200

300

400

500

600

0

100

200

300

400

500

Input 'Dog'

GradCAM++

0

100

200

300

400

500

600

0

100

200

300

400

500

Explain 'Dog'

Sim=0.59

0

100

200

300

400

500

600

0

100

200

300

400

500

Explain 'Rain'

Sim=-0.10

Explain 'Toilet flushing'

Explain 'Water drops'

Sim=0.69

0

100

200

300

400

500

600

Explain 'Toilet flushing'

Sim=0.69

0

100

500

400

300

200

100

0

500

400

300

200

100

600

0

200

300

400

500

Sim=0.14

100

200

300

400

500

600

Explain 'Water drops'

Sim=0.14

100

200

300

400

500

0

600

500

400

300

200

100

0

500

400

300

200

100

0

0

500

400

300

200

100

0

500

400

300

200

100

0

0

0

Explain 'Jackhammer'

Sim=-0.01

100

200

300

400

500

600

Explain 'Jackhammer'

Sim=-0.01

100

200

300

400

500

600

B Explanation sensitivity to Audio-Text Similarity

To showcase the effectiveness of the additional diversity term (Equation 7), we conducted qualitative and quantitative tests to evaluate the explanation sensitivity to text prompts.

B.1 Qualitative results

We compare the explanations obtained with LMAC-ZS with and without the diversity term in Eq. 7. We present the results in Figure 8 and Figure 9, respectively. The results are obtained for the model that does masking in the STFT domain and was trained on FSD50K. We present the plots using log-frequency scaling. In each plot, we give the original text-audio similarity (in the title of the first subplot), as well as the audio-text similarity after masking the audio (in the title of the third subplot). Note that the predicted class is also used as the prompt for the masking model. We observe that masks are more sensitive to text prompts because of the additional diversity term.

inp, original sim = 0.49

inp, original sim = 0.23

8192

8192

4096

4096

2048

2048

1024

1024

512

512

256

256

128

128

64

64

o

0

10

15

0

10

15

Time

Time

Predicted class = dog

Predicted class = train

Dominant audio = this is the sound of dog

Dominant audio = this is the sound of dog

8192

8192

4096

4096

2048

2048

1024

1024

512

512

256

256

128

64

128

64

o

0

10

15

0

10

15

Time

Time

m * x STFT masked sim = 0.56

m * x STFT masked sim = 0.30

8192

8192

4096

4096

2048

2048

1024

1024

512

512

256

256

128

128

64

64

0

0

0

10

15

0

10

15

Time

Time

inp, original sim = 0.49

inp, original sim = 0.23

8192

4096

8192

4096

2048

2048

1024

1024

512

512

256

128

128

256

64

64

0

0

5

Time

10

15

0

5

Time

10

15

Predicted class =dog

Predicted class = train

Dominant audio = this is the sound of dog

Dominant audio = this is the sound of dog

8192

8192

4096

4096

2048

2048

1024

1024

512

512

256

256

128

128

64

64

0

0

10

15

0

10

15

Time

Time

m * x STFT masked sim = 0.53

m * x STFT masked sim = 0.26

8192

8192

4096

4096

2048

1024

2048

1024

512

空

512

256

256

128

128

64

64

0

0

0

10

15

0

10

15

Time

Time

B.2 Quantitative results

In Figure 10, we present the 2D histogram of mask mean and similarity between text and audio after masking. This highlights the increased mask sensitivity of the model to different text prompts when the diversity term in Equation 7 of the paper is utilized. We see in the left panel of Fig. 10 that without the diversity term, the mask means do not have a discernible correlation with the text-masked audio similarity.

Without diversity term

With diversity term

0.25

16

0.16

50

14

0.20

0.14

12

Means

40

leans

S

0.12

10

0.15

30

M

0.10

Mask I

Mask

8

0.08

0.10

20

6

0.06

0.05

10

0.04-

0.02

-0.4-0.20.0

0.2

0.4

0.6

0.8

0.2

0.0

0.2

0.4

0.6

0.8

Similarities

Similarities

C Ablation on the training dataset size for LMAC-ZS

LMAC-ZS is a decoder-based interpreter. That is, we train the decoder based on the pre-trained classifier's representations. The amount and quality of training data can impact both the performance and the training time of the interpreter. We benchmarked our interpreter on different datasets with different sizes, i.e. the datasets that are included within the whole CLAP training set (denoted with All Data in Table 4) - The datasets that make up the whole CLAP training set are, Clotho [37], MACs [40], FSD50k [38] and AudioCaps [39]. We have also experimented with randomly subsampling the whole CLAP training set and denoted it as 'Subset' in 4.

In Table 4, we report the interpreter performance for the aforementioned training datasets with different sizes. We note that the explanation's faithfulness is comparable when training the decoder on the entire training data or a subset; this indicates that it is possible to train LMAC-ZS on a smaller dataset and still obtain faithful explanations.

However, we note that the MACs-only training results obtained the lowest performance on ESC50. We note that this is likely related to the differences in data distributions. To investigate this, we have computed Frechet Audio Distances (computed via CLAP embeddings) between ESC-50 and different subsets in Table 5. We observe that the highest distance is between MACs and ESC50 is the highest. This suggests that if the similarity between the target domain and the training set for the interpreter is relatively high, it is possible to train LMAC-ZS on smaller subsets.

In Table 4, we also present the performance of two additional baselines, ScoreCAM [48] and GScoreCAM [49], and we see that on ESC50, except the case where input audio is contaminated with another audio sample from the ESC50 dataset, LMAC-ZS is able outperform these baselines for the majority of the faithfulness metrics.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

[Yes]

Justification: The abstract and introduction clearly outline the development of LMAC-ZS as our primary contribution. This claim is supported by a detailed description of LMAC-ZS, its training process, and evaluation methodology in the subsequent sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the study are clearly described in Section 5 (Limitations and Societal Impact). Additionally, the paper addresses the computational resources required for the experiments in Section 4.2 (Experimental Setup).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides a detailed description of the experimental setup in Section 4.2. Additionally, to ensure reproducibility, the code will be publicly released using a popular toolkit like SpeechBrain. The code repository also includes documentation on how to run the experiments and replicate the results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be made publicly available. We provide documentation on how to run the experiments and replicate the results discussed in this paper. Our results are fully replicable as we only used publicly available datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 4 details the experimental setup, including the datasets employed and the evaluation metrics used. More granular implementation details are available in the code repository.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We think that the absence of error bars or explicit statistical significance measures does not detract from the robustness of our findings. The substantial performance gap observed between our proposed method and existing approaches gives us a high level of confidence in the reliability and significance of our results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 4.2 details the computational resources employed in our experiments. This includes information on the type of GPU, memory requirements, and training times.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully reviewed the NeurIPS Code of Ethics and believe our research is fully compliant with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section 5 (Limitations and Societal impact) discusses it.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper provides a detailed description of LMAC-ZS, including its architecture, training process, and evaluation methodology (Sections 3 and 4). Additionally, the code for LMAC-ZS will be released publicly on SpeechBrain alongside comprehensive documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes an extension of the LMAC method for explaining decisions made by audio classifiers.  The novelty in the present article is to extend from fixed-vocabulary settings to zero-shot / open text description settings.  To accomplish this, the authors propose a training objective that aims to preserve proximity of (learned) masked audio embeddings to embeddings of corresponding text descriptions.  The method is evaluated on the CLAP model over two standard sound event detection datasets, and compares quite favorably to prior work along several evaluation criteria.\n\nSTRENGTHS:\nOverall, I found this paper to be well written and easy to follow.  The topic of the paper is timely and relevant, as much audio analysis research does seem to be trending toward open vocabulary settings.  The empirical evaluations are appropriate and generally convincing.\n\nThe proposed method makes a lot of intuitive sense, and appears to be a natural extension of the fixed vocabulary setting (section 2.2).\n\nWEAKNESSES:\nThe main weakness I see in this work is a somewhat shallow investigation of the proposed method itself, as opposed to high-level comparisons to prior work.  The core technical contribution is described in equations 5-7, and consists of 3 terms: one to approximately preserve the embedding of audio (relative to text embedding) after masking, one to promote sparsity on the masks, and one to promote diversity of masks when conditioning on different text embeddings.  As in CLAP, contrast is obtained by comparing amongst other samples within the training batch.  This raises a handful of questions about the various components of the training objective which I will detail below; however, the critical issue here is that no ablation study is conducted to measure the importance of the various terms.  This leads the reader at a bit of a loss to understand how these components interact with each other, whether they're all necessary, and so on.\n\nQUESTIONS:\nMy questions primarily center around equation 6:\n\n- Notational quirk: $C_{i,j}$ is a scalar value (equation 5), right?  Why are norm bars ($\\|$) used in eq 6 when computing the loss on approximating $C_{i,j}$?\n- Reasoning through the first term of eq6, a few things jump out at me.  First, expanding out the definition in eq5, the summand should be equivalent to $\\left|t_i^\\mathsf{T} \\left(f_\\text{audio}(X_{\\text{audio},j}) - f_\\text{audio}(M_\\theta(t_i, h_j) \\odot X_{\\text{audio},j}) \\right)\\right|$.  That is, the gap between the original and masked audio embedding should be small (ie the mask doesn't do much) or approximately orthogonal to $t_i$, and this should hold for all $t_i$ in batch when the outer sum is computed.  \n    - What is the effect of the batch size / batch diversity on this loss?  Presumably, the more (and more diverse) text embeddings we compute against, the harder it will be to achieve this orthogonality while having nontrivial masks.  Similarly, a too-small batch could make it too easy to achieve this orthogonality in a high-dimensional embedding space while not producing useful masks.  Is there a sweet spot or range sizes where this seems to work (on CLAP)?\n- What happens if the third term is left out of the loss?  Similarly, how sensitive is this term to size and diversity of the batch?\n\nA couple of minor questions:\n\n- Line 155 describes the audio reconstruction by ISTFT.  It's not clear how phase is treated here, since $X$ appears to be magnitude spectra.  Presumably phase is retained from the STFT and propagated through, but if some other method is used here (eg griffin-lim or the like) it should be made explicit.\n- In general, the spectrogram visualizations (eg figure 4) are quite difficult to read in this paper.  It would be helpful to A) label the axes with appropriate units, B) use a log-scale frequency axis for display (even if the underlying method uses linear spectra, the results will be more visible), and C) use a log amplitude scaling on the magnitudes to enhance visual contrast.\n\nLIMITATIONS:\nThe limitations and impacts are well and appropriately described by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a post-hoc interpretation method for zero-shot audio classifiers, named LMAC-ZS (Listenable Maps for Audio Classifiers in the Zero-Shot context). It addresses the challenge of interpreting predictions from zero-shot audio classifiers that define audio classes based on textual prompts, where labels are not predefined but generated dynamically. LMAC-ZS outputs saliency maps that highlight important regions within input audio, correlating these with corresponding text prompts. The method involves a novel loss function that maintains the original audio-text similarity, enhancing interpretability. Experiments were conducted using the CLAP model on standard datasets like ESC50 and UrbanSound8K.\n\nSTRENGTHS:\n1. Motivation is Clearly articulated. It stresses the importance of interpretability in AI, particularly for models used in critical decision-making areas, focusing on the less-explored zero-shot audio classification.\n2. The problem of providing interpretable explanations for zero-shot audio classifiers is clearly formulated and identified as a novel challenge in the field.\n3. The paper compares LMAC-ZS with several baseline methods like GradCAM++, SmoothGrad, and Integrated Gradients, providing a thorough comparative analysis and justifying the selection based on their relevance and common use in related tasks.\n\nWEAKNESSES:\n1. It is observed that in some scenarios LMAC-ZS (CT) performs better than LMAC-ZS (FULL) in metrics such as AI, AD, and AG. This is counterintuitive, as one would expect the model trained on the full CLAP dataset to perform better. An explanation for this discrepancy is needed.\n2. The paper mentions exploring the training of LMAC-ZS only on the Clotho dataset to simulate a limited computational budget scenario. It raises the question of whether training on other individual datasets or comparing the performance across different single datasets versus the full dataset would yield different insights. This aspect needs further exploration and clarification.\n3. In Table 2, the use of ESC50 contamination for ZS classification on the ESC50 dataset seems confusing. The rationale behind using the same dataset for contamination needs to be explained in detail. Similarly, in Table 3, the reason for using US8K contamination for Mel-Masking and ESC50 contamination for STFT-Masking in ZS classification on US8K requires clarification.\n\nQUESTIONS:\n1. In Figure 7, the visual explanations for 'Toilet flushing' and 'Water drops' look very similar despite different similarity scores (Sim=0.69 and Sim=0.14). How does the model ensure distinguishable and meaningful explanations, and can more distinct examples be provided to demonstrate the method's effectiveness?\n2. I have a question about a set of textual prompts. What happens if there are sounds that are not in the text?\n\nLIMITATIONS:\nThe author has addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors focus on interpreting the decisions of zero-shot audio classifiers, particularly ones based on the Contrastive Language-Audio Pretraining (CLAP) model. To achieve this, the authors propose to learn a decoder that predicts a \"listenable\" audio saliency map (a mask on the input spectrogram) from an audio-text pair. The decoder is trained with a novel loss function that remains faithful to the characteristics of CLAP features. The authors demonstrate the effectiveness of their approach with extensive experiments across various datasets.\n\nSTRENGTHS:\n### Strengths\n\n1. Although there is a significant body of research focusing on interpretability of audio classifiers, interpretability of *zero-shot* audio classifiers (and zero-shot classifiers in general) is an under-explored area, which certainly merits more investigation.\n2. From a technical perspective, the proposed approach is sound and has no glaring weaknesses.\n3. The qualitative and quantitative results show that the method indeed works as claimed.\n    - Particularly the anonymized qualitative samples provided are highly appreciated.\n\nWEAKNESSES:\n### Weaknesses\n\n1. Even after reading two out of the three subsections in Section 2: Methodology, a reader learns nothing new from the paper.\n\n    To be more specific:\n    - Section 2.1 provides a high-level overview of \"Contrastive Language Audio Pretraining\" (CLAP) [10], while Section 2.2 gives a brief summary of the approach of the paper \"Listenable Maps for Audio Classification\" (L-MAC) [8]. Both these sections discuss the previous papers as is, without any new perspective.\n    - It is solely Section 2.3 that pertains to the method proposed by the paper. Even in Section 2.3, the sub-subsection @line-152, **\"Producing Listenable Interpretations\"** is not something new achieved by the proposed method, but rather a feature of the previous L-MAC paper [8].\n        - Notably, [8] also has an explicit sub-section with the *exact same title*; **Section 2.2: Producing Listenable Interpretations**. The authors make no effort in lines 152-156 to clarify that the \"listenable\" feature comes from [8]; thus this is not only redundant, but also very close to plagiarism.\n\n    - **Suggestion:** I strongly suggest that Sections 2.1, 2.2, and the sub-subsection (@lines 152-156) be discussed separately as a \"Background\" or \"Preliminaries\" section, instead of Methodology. Specifically, since the proposed approach builds significantly upon L-MAC [8], it is necessary to disentangle your contributions from that of the authors of [8].\n\n2. The technical novelty is limited by the L-MAC paper.\n    - From my understanding, the only novelty in the method is the loss function in Equation 6. And even that is mostly changing the cross entropy-based objectives in L-MAC (Equation (2) in [8]) to contrastive losses characteristic of CLAP. Overall, the same min-max objective is retained, and the same regularization term is added for mask sparsity. The same general framework from L-MAC is followed: a decoder is trained on the same dataset as the encoder to predict a mask on the input audio spectrogram.\n\n3. (minor weakness) Comparison against baselines.\n\n    - The authors are comparing against the baselines of GradCAM, GradCAM++, SmoothGrad, Integrated Gradients. While all these methods were also used for comparison with the original L-MAC in [8], that was in a regular classification setting; these methods are, by nature, poorly suited to zero-shot settings.\n\n    - For instance, for both \"cat\" and \"glass-breaking\", the CLAP model needs to \"look\" at the important regions of the audio to give an encoded audio feature---which can be observed in Figure 4 (for GradGAM) in the paper.\n\n    - **Suggestion**: A better baseline would perhaps be from the paper \"gScoreCAM: What objects is CLIP looking at?\" [a]. The paper empirically establishes that for CLIP, in zero-shot settings, ScoreCAM performs better than the other forms of CAM.\n\n    - *Note:* I understand that in general, there is a lack of baselines to compare to (for instance, listen-to-interpret and L-MAC are not applicable). I do not expect the authors to conduct any additional experiments.\n\n---\n\nIn general, the paper has a significant amount of similarity/redundancy with the L-MAC paper [8], from the naming of sections and content (noted in Weakness 1), figures (Fig 1 in [8] and fig 2 in current paper), and even the exact set of metrics posed in the identical order in Section 3.1, which makes it difficult to discern the novelty and contributions.\n\n---\n[a] Chen, Peijie, et al. \"gscorecam: What objects is clip looking at?.\" Proceedings of the Asian Conference on Computer Vision. 2022.\n\nQUESTIONS:\n### Questions\n1. **line 147** *\"The intuition is that the similarity between two text prompts should be reflected in the similarity of the audio embeddings from the corresponding masked spectrograms\"*.\n\n    Equation (7) is the only part of the loss function that is distinctly novel, as it adds a third term to the original loss in [8]. Have you tried training the overall decoder without the third term? It would be helpful to the paper if you can show that adding the third term yields a noticeable improvement over just using the first two terms; otherwise, the hypothesis remains unsupported by evidence.\n\n2. It is common to perform zero-shot classification with foundation models trained on a large dataset. Suppose we have one such hypothetical model, trained on a dataset with millions of samples.\n    - Is it necessary to train the decoder on the same large dataset? If not, can an estimation be made as to what percentage of the training data the decoder needs to see to achieve reliable performance? (For instance, it may be the case that after seeing 20% of the data, the model achieves 80% of its full performance).\n    - If it is necessary to train the decoder on the pre-training dataset, using LMAC-ZS becomes expensive in many cases. Can LMAC-ZS be jointly learned during the pre-training process of the base model (e.g. CLAP)?\n        - Note that this may not be so straightforward, as the first term in the loss objective will try to match the decoder predictions to faulty entries of $C$ at the start of the training.\n\n### General Suggestions\n- **line-114**; **line-122**: The authors note that they omitted a part of Equation (4) for brevity. I strongly suggest against doing this; *clarity* is more important than brevity when posing a loss function or optimization objective. \n\n    Equation (4) is supposed to be a min-max objective; the goal is to maximize the classification confidence of the masked-in (salient) part of the audio, while minimizing the confidence of the masked-out part. Thus the whole equation should be written together, and parts of it should not be omitted. If brevity is desired, it may be achieved by abbreviating $\\text{CrossEntropy}$ to $CE$ or $\\mathcal{L}_{CE}$.\n\n- **line 120:** The abbreviation L-MAC is used, but it has not been defined previously. The authors should define the abbreviation on line 70.\n- **line 62-63:** The citation style is inconsistent. For example, in lines 59-60, it has been written \"Key approaches in this category include [19, 20, 21, 22],\". The same citation style can be followed in line 64: \"Notable attempts in this vein include [23, 24, 25].\"\n\n---\n[b] Shimada, Kazuki, et al. \"Zero-and Few-Shot Sound Event Localization and Detection.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n\nLIMITATIONS:\n### Limitations\n- One limitation that I feel is not addressed is that the approach needs to be specifically trained on CLAP's data; it is not plug-and-play like Grad-CAM or similar approaches.\n    - In **line 216** It is mentioned that the decoder uses CNN14 layers, presumably because the audio part of CLAP is based on CNN14.\n        - Now if we have another zero-shot audio classifier, LAION-CLAP, that (suppose) has the same dataset but a different transformer-like architecture, then the decoder may not be transferable. Another decoder architecture needs to be designed to suit the alternative zero-shot classifier. So for different zero-shot foundation models, it may become necessary to have different architectures, which can be a hurdle.\n        - As noted earlier, for same architecture but different datasets, it is still needed to train the decoder again. Unless of course, the decoder from one pre-training dataset transfers to another dataset, as a general purpose audio method.\n\nApart from these scalability concerns, I believe the remaining limitations are adequately addressed.\n\n---\n\nUpdate: The rebuttal addresses most of my concerns, so I will raise my score to 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}