Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMF
Authors: Jayneel Parekh
Sanjeel Parekh
Pavlo Mozharovskyi
Florence d'Alché-Buc
Gaël Richard
Venue: NeurIPS 2022 Accept (2022)
Keywords: audio interpretability
post-hoc explainability
non-negative matrix factorization
audio recognition

Abstract: This paper tackles post-hoc interpretability for audio processing networks. Our goal is to interpret decisions of a trained network in terms of high-level audio objects that are also listenable for the end-user. To this end, we propose a novel interpreter design that incorporates non-negative matrix factorization (NMF). In particular, a regularized interpreter module is trained to take hidden layer representations of the targeted network as input and produce time activations of pre-learnt NMF components as intermediate outputs. Our methodology allows us to generate intuitive audio-based interpretations that explicitly enhance parts of the input signal most relevant for a network's decision. We demonstrate our method's applicability on popular benchmarks, including a real-world multi-label classification task.

Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMF
Jayneel Parekh'
Sanjeel Parekh Pavlo Mozharovskyil
Florence d' 'llch--Buc' Gaël Richard!
LTCL, Télécom Paris, Institut Polytechnique de Paris Audio Analytic
[jayneel parekh, pavlo. mozharovskyi, florence....o dalche, gael ri rteeharddtelllecmmp
sanjeel. www..araadiiiaaaaaacaaaoa..a.a canalytic...iaiccccaaaommimmm com
Abstract
This paper tackles post-hoc interpretability for audio processing networks. Our goal is to interpret decisions of trained network in terms of high-level audio objects that are also listenable for the end-user. To this end, we propose a novel interpreter design that incorporates non-negative matrix factorization (NMF). In particular, a regularized interpreter module is trained to take hidden layer representations of the targeted network input and produce time activations of pre-leart NMF components as intermediate outputs. Our methodology allows us to generate intuitive audio-based interpretations that explicitly enhance parts of the input signal most relevant for a network's decision. We demonstrate our method's applicability popular benchmarks, including woaldworld multi-label classification task.
Introduction
Deep learning models, while state-of-the-art for several tasks in domains such computer vision, natural language processing and audio, are typically not interpretable. Their increasing use, especially in
domains, necessitates interpreting their decisions. good interpretation is often characterized by its understandability for the end users (see for instance [16]). More importantly, attributes that aid understandability may largely be dependent on the data modality. In this paper, our aim is to generate post-hoc human-une uuman-understandable interpretations for deep networks that process the audio modality. Here, post-hoc interpretability refers to the problem of interpreting decisions of fixed trained network.
Traditional approaches generate interpretations through input attribution, either directly the raw input features or on given simplified representation [30
38
36
27). To generate more
understandable interpretations, small number of approaches consider other means, such logical
rules 37 sentences [19 and high-level concepts [1]].
Most existing post-hoc interpretability methods primarily designed for application to images and tabular data. This limits their applicability to other data modalities such as audio. Although many audio processing networks operate spectrogram-like representations, which can be seen 2D) time-freuuency images, a visualization or attribution this space is not as meaningful to a common user as it is for images (26).
This leads us to build an interpretation system that takes into account audio-specific understandability features. We motivate these features through an example: suppose audio event detection network deployed in a house recognizes an "alarm" sound event. An ideal interpreter for this classification decision would have the ability to "show" that it indeed alarm sound that triggered this decision. To do so, must be able to localize the alarm amid other events in the house (for e.g. dog
Work conducted while the author was Télécom Paris
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

barks, baby cries, background noise etc.) and make it listenable for the end end-user. It is important to highlight here the role of listenable interpretations for better understanding of an audio network's decisions note that it would be much less meaningful for a human to see the alarm sound as highlighted parts of a spectrogram. Thus making the following aspects important for our system design: (i) generating interpretations in terms of high-level audio objects that constitute scene, (ii) segmenting parts of the input signal most relevant for a decision and providing it as listenable audio. It's worth emphasizing that audio interpretability is not the same classical tasks of separation or denoising. These tasks involve recovering complete object of interest in the output audio. On the other hand, classifier network might focus more salient regions. When interpreting its decision and making it listenable we expect to uncover such regions and not necessarily the complete object of interest.
To this end, we propose a novel post-hoc interpreter for audio that employs a popular signal decom- position technique called Non-negative Matrix Factorization (NMF; Lee and Seung (25 NMF seeks decompose an audio signal into constituent spectral patterns and their temporal activations. Unlike principal component analysis, NMF is known to provide part-based decompositions [12]. Owing to these properties, we first use NMF to pre-learn a spectral pattern dictionary on our train- ing data. This dictionary is then incorporated as fixed decoder within our interpretation module. Specifically, we train our system to determine an intermediate encoding that performs two roles: (i) able to reconstruct the input through the fixed NMF dictionary decoder, thus corresponding to time activations for dictionary components, (ii) at the same time, a function of this encoding is able to mimic the classifier's 's output. Training with these constraints allows to generate, for any classifier decision, importance values over spectral patterns in our dictionary. Listenable interpretations are readily produced by inverting most important NMF spectral patterns back to the time domain.
In summary, we make the following contributions:
We propose a novel NMF-based interpreter module for post-hoc interpretability that gen- erates interpretations in terms of meaningful high-level audio objects, listenable for the end-user.
We present an original formulation that constrains the interpreter encoding through two loss functions, one for input reconstruction through NMF dictionary and the other for fidelity to the network's decision. From a learning perspective, we show a new way to link NMF with deep neural networks, especially for generating interpretations.
We extensively evaluate on two popular audio event analysis benchmarks, tackling both multi-clas and multi-label classification tasks. The dataset for the latter is very challenging due to its collection in noisy real-world setting. Our method's design allows us to simulate feature removal and perform faithfulness evaluation.
2
Related Works
In this section, we position our work in relation to: (i) interpretability methods for audio, (ii) methods for concept- interpretability and, (ii) use of NMF within the audio community, particular, attempts to link it with deep networks.
44 have shown usability of atten- tion/visualization techniques for interpreting audio processing networks or generated instance-wise feature importance [46] for time-series data [1 However, our focus is methods that attempt to address audio interpretability beyond image-based visualizations or raw input attribution. Muckenhirn al. [3I illustrate usefulness of GuidedBackprop [40] for analyzing CNNs operating on raw ID waveforms by analyzing relevance signal in frequency domain. This analysis does not extend to spectrogram input or address listenability of interpretations. A few works have applied the popular LIME algorithm with a simplified input representation more suited for audio. In particular, SLIME proposes to segment the input along time or frequency. The input is perturbed by switching "on/off" the individual segments. AudioLIME [88 [0] proposes to separate input using predefined sources to create the simplified representation. AudioLIME arguably generates more meaningful interpretations than SLIME as it relies on audio objects readily listenable for end-user. However, can only be applied for limited applications as requires existence known and meaningful predefined sources that compose the input audio. APNet 47] takes another promising direction by
Interpretability methods for audio Some approaches [
28
29

---

Log magniude spedtrogam
Input C
X
CNMF
Time Activaions
Components
Hz(x)
f1
hj
f
Classifier
bg
(z)
4
big
W2
WX
1
Interpreter Module
LFID
Output
I
f(x)
Figure 1: System overview: The interpretation module (right block) accesses hidden layer outputs of the network being interpreted (left block). These are used predict an intermediate encoding. Through regularization terms, we encourage this encoding to both mimic the classifier's output and also serve as the time activations of pre-learnt NMF dictionary.
extending interpretable prototypical networks for audio input. However, they propose an interpretable system by-design. They don't 't tackle the problem post-hoc interpretation.
Concept-based interpretability Our method relies on high-level objects for interpretation. In this sense, it is most closely related to post-hoc concept-based methods [21 [5]. An interesting approach is that of post-hoc version of FLINT [2] with whom we share the idea of utilizing the hidden layers and loss functions to encourage interpretability. However we crucially differ from FLINT and other related approaches in concept representation and their applicability for audio interpretations. FLINT represents concepts by a dictionary of attribute functions over input space. The Iearnt concepts are not obviously comprehensible to user, requiring separate visualization pipeline to get insights. Approaches based on TCAV (21]. such as ACE [15). ConceptSHAP [45), define concept using set of images and learn representation for it in terms of hidden layers of the network, termed as concept activation vector (CAV). J. These designs for concepts are not related to our NMF-inspired dictionary representation. Importantly, none of the above mentioned approaches generate listenable interpretations which is key for understandability of audio processing networks.
NMF for audio has been widely used for numerous tasks ranging from separation to transcription 39 42 6 Its traditional usage as supervised dictionary or feature learning method involves learning class-wise dictionaries over training data [12]. Time activations, which are the so-called features, are generated for any data point by projecting it onto the learnt dictionaries. Extracted features subsequently be used for downstream tasks such classification. Bisot al. [7] couple NMF-based features with neural networks to boost performance of acoustic scene classification.
NMF has also been successfully employed with audio audio-visual deep learning models for separation 13 and classification [33. Another line of research explored unfolding iterations of different NMF optimization algorithms as deep neural network [23 43). These systems, commonly known as Deep NMF, have primarily been used for audio source separation tasks.
While these works share with us the idea of combining neural networks and NMF, there is no overlap between our goals and methodologies. Unlike aforementioned studies, we wish investigate classifier's decision in post-hoc manner using NMF as regularizer. Furthermore, to our best knowledge, attempting regress temporal activations of a fixed NMF dictionary by accessing intermediate layers of an audio classification network is novel even within the NMF literature.
3 System Design
We begin this section with brief note data notation and NMF. Subsequently, in Sec. 3.1 discuss interpreter module's design and learning. This followed by a description of our interpretation generation methodology in Sec. 3.2 An overview of the proposed system is presented Fig. 1
Data notation. We denote training dataset by S = (æ, y) where is the time domain audio signal and y, label vector. The label vector could be one-hot or binary encoding depending
3

---

upon multi-class or multi-labe dataset, respectively. Since multiple audio representations are used in this paper. note on their notation is in order. Very often, audio signals are processed in the frequency domain through a short-time Fourier transform (STFT) on called spectrogram. Log-mel spectrograms are a popular input to audio classification networks [4) which also the one we use in this paper. To keep notation simple, we refer to input of the network with æ. For NMF however, we favor a representation spectrogram X that is computed by applying the magnitude spectrogram. This is preferred over using magnitude spectrograms as it corresponds more closely to human perception of sound intensity [17
æ that can be easily inverted back to the time-domain and use a log-magnitude
element-wise transformation 2o + log(1 + zo) on
NMF basics. NMF popular technique for unsupervised decomposition of audio signals [3]. Given any positive frequeryy representation X R_xT consisting of F frequeney bins and T time frames, NMF decomposes it into a product of two non-negative matrices, such that,
X WH
Here, W = -[w1. W....., ... E R Fx interpreted as the spectral pattern or dictionary matrix containing K components and H = [h, hy...., h E R a matrix containing the correspond- ing time activations. Typically, a f-ddivereence measure between X and WH is minimized and multiplicative updates are used for estimating W and H [25]. Note that it is possible to reconstruct signal corresponding each or group of spectral components. This is typically done using procedure called soft-masking. For a single component k; this is written as,
walh WH
X
OX
Both ./. and @ are clement-wise operations. If X is an invertible representation of the magnitude spectrogram, time domain signal for X₁ is easily recovered using the inverse STFT operation. We extensively utilize this procedure for generating listenable interpretations. NMF can also be used for dictionary learning, by estimating W on a training dataset matrix X X As discussed later, we use variant of NMF called Sparse-NMF 24 to pre-learn dictionary for subsequent usage in the interpretation module.
3.1 Interpreter Design
As depicted in Fig. the interpreter module I contains two components: an interpreter network and NMF dictionary decoder. The so-called interpreter network computes the following function + O V o (z) where V/ is the function responsible for generating an intermediate encoding from hidden layer representations of the classification network, and @ attempts mimic the classifier's output given the intermediate representation. The NMF decoder based on pre-trained dictionary plays two roles: (i) during training, it constrains the intermediate representation to correspond to time activations of pre-learnt spectral pattern dictionary and (ii) when interpreting a classifier's 's prediction, it is used to build listenable interpretation. To the best of our knowledge, this is an original usage of NMF that allows us to interpret a network's 's decisions in terms of a fixed dictionary.
Design of V. The function V processes outputs of of hidden layers of the classifier, given by fz(x). It's output, (fz())) k produces intermediate encoding of the interpreter. For simplicity, we will denote this intermediate encoding as Hz (x) = V fz fz(), a function over input x.
In practice, V is modelled as neural network that takes as input convolutional feature maps from different layers of f. To concatenate and perform joint processing on them, each feature map first appropriately transformed to ensure same width and height dimensions. Two important aspects were kept in mind while designing subsequent layers of V. Firstly, audio feature maps for spectrogram-like inputs naturally contain the notion of time and frequency along the width and height dimensions. Secondly, through appropriate regularization we wish to produce an intermediate encoding that also serves as time activations of the pre-learnt NMF dictionary, matrix of dimensions K x T. To achieve this, we continuously downsample on the frequency axis and upsample the time axis to T frames. Similarly, the number of input feature maps is re-sampled to reach a size of K, equal the number components in dictionary W. Additionally, we ensure non non-negativity of Hz(x) through use of ReL.U activation. All learnable parameters of V are denoted by Vv.
Design of θ Hz(x), the intermediate encoding output by V is then fed to θ which aims to mimic output of the classifier. This directly helps in learning a representation which can interpret f(x). Its

---

design consists of two parts. The first part pools activations (z) across time. While this pooling
can implemented in multiple ways, we opt for attention-based pooling [20], i.e., z (z(z)a.
where a E R are the attention weights and z E R is the pooled vector. The pooled representation vector passed through linear layer. This is followed by an appropriate activation function to convert its output to probabilities, that is, softmax for multi-class classification and sigmoïd for multi-label classification. All learnable parameters of e are denoted by Ve.
Fidelity loss. Generalized cross-entropy between interpreter's ' output ((Hz(z)) and classifier's output f(x) is minimized to encourage interpreter mimic the classifier. For multi-class classification this loss function is written as,
(ZD Ve. Ve) -f(2)) log(O(Hz(z)))
On the other hand, for multi-label classification this loss reads,
(aD Vo, Ve) = Σ [(x) o log(O(Hz(z)) + (1 f(x) の log(1 ((Hz(z)))
Here denotes element-wise multiplication.
NMF dictionary decoder and regularization. We additionally constrain the intermediate encoding, such that, when fed to decoder it is able to reconstruct the input audio. As already discussed, we choose this decoder to be a pre-learnt NMF dictionary. W. Formally, through LNMP we require (z) approximate log- oog-manntuude spectrogram X of input audio r as X = WHz(z)):
LNME(I, Vv) |X - WHz
(3)
This allows us to consider (x) as a time activation matrix for W.
Training loss. In addition to Lrı and LNMF. we impose & regularization on Hz(x) to encourage sparsity and well-behavedness, is often done in classical NMF (24). The complete training loss function over our training dataset S can thus be given as:
C(V,, Vo)
Σ CrI(L, Vv, Vo) aLxMF(x, Vv) + Hz(x))l zES
(4)
where ax B 2 0 are loss hyperparameters, All the parameters of the system are constituted in the functions V, and dictionary W. Since is pre-learnt and fixed, the training loss optimized only w.r.t Vg, Ve. As a reminder, when training the interpreter for post-hoc analysis, the classifier network is kept fixed.
Algorithm 1 Learning algorithm
Input: Classifier f, Training data S, parameters {Vv, Vel. hyperparameters {a, pJ)
number of batches B, number of training epochs Nepoch
2: W < PRE--LEARN NMF DICTIONARY (S,p)
(// Sparse-NMF algorithm)
Random initialization of parameters Vo
4: V < TRAIN (f,S, W, Vo, a, (3, B, Nopoch
(// Train with Eq.
5: Output: V = {Ve, Vo}
Learning algorithm. The complete learning pipeline is presented in Algorithm I
The learnable
parameters of the interpreter module given by V (Vv, Ve The pre-specified dictionary (Step
2 in Algorithm [T is learnt using Sparse-NMi NMF (24] wherein, the following optimization problem is solved through multiplicative updates to learlearn W:
min [XXin WH) + H|| s.t. 2 0, H. 0, | = 1, Vk.
(5)
Here (-I.)) is divergence cost function. In practice euclidean distance is used. The reader is referred to appendix A.1 for more details regarding Sparse-NMP optimization problem, such as construction of X our datasets.
3.2 Interpretation generation
Finally, to generate audio that interprets the classifier's decision a sample x and a predicted class follow two-step procedure: The first step consists of selecting the components which are
5

---

considered "important" for the prediction. This is determined by estimating their relevance using the pooled time activations in and the weights for linear layer, and then thresholding it. Precisely, given sample the pooled activations are computed as z (z)a. Denoting the weights for
(2, 0
class è in the linear layer as 0% the relevance of component is estimated as Tk,c, maxi 2 c.
This is essentially the normalized contribution of component & in the output logit for class c. Given a threshold the selected set of components are computed as Le.= {k Tkk,, > 7).
The second step consists of estimating a time domain signal for each relevant component & E Le. and also for set Le.z as a whole. In this paper, we refer to the latter as the generated interpretation audio, Zint- For certain classes, may also be meaningful listen each individual component, Tk- As discussed earlier under NMF basics, estimating time domain signals from spectral patterns and their activations typically involves soft masking and inverse STFT procedure. The inversion performed using input audio phase P.. We detail this step with appropriate equations in Algorithm
Algorithm 2 Audio interpretation generation
I: Input: log- magnitude spectrogram X, input phase P., components W -WWI:... Wk ) time
activations Hx(x)) = [f))))..... h (T))T, set of selected components Tio,z (ki,...; kn).
2: for all 1: E Le,z do
web 3: X f ΣR (w) O
(// Soft masking)
4: Tk = X Pa)
(// Inverse STFT)
5: end for
6: Xine Eee Xs
7: Fint -INV(Xim: P2)
8: Output: {Dki......
Ikn Fin
4 Experiments
We experiment with two popular audio event analysis benchmarks, namely ESC-50 [35 and SONYC-
UST [9]. While the former is multiclass environmental sound classification dataset, the latter appeared for DCASE 2019 and 2020 multi-label urban sound tagging task. We quantitatively and qualitatively evaluate different aspects of our interpretations, including a subjective evaluation carried out on SONYC-UST. The implementation of our system is available on GitHul!" This section organized as follows: quantitative metrics and baselines are discussed in Sec. 4.1 followed by implementation details in Sec. 4.2 Experiments on ESC-50 and SONYC-UST are detailed in Sec.
and Sec. 4.4 respectively. We discuss some limitations in Sec. 4.5
4.1 Quantitative metries and baselines
Metrics. We quantitatively evaluate our interpretations in two ways. First, by evaluating how well agrees with the classifier's 's output. For multi-class classification, this is done by computing fraction of samples where the class predicted by f is among the top-k classes predicted by the interpreter. We refer to this the top-k fidelity. To compute fidelity on multi-label classification tasks, we compute Area Under Precision-Resdt Recall Curve (AUPRC) based metrics between the classifier output f f(x) and its approximation by interpreter ((Hz(z) We compute macro-AUPRE micro-AUPRC. Additionally, we report the maximum micro Fl Fc-oe over different thresholds for the interpreter's output.
We also conduct a faatthfulnesss evaluation for our interpretations. In general for any interpretability method, faithfulness tries to assess if the features identified to be of high relevance are truly important in classifier's s prediction [I. Since a "ground-truthh importance measure for features is rarely available, attribution based methods evaluate faithfulness by performing feature removal (generally by setting feature value to 0) and observing the change in classifier's ' output [.]. However, it is hard to conduct such evaluation for non-attribution or concept based interpretation methods on data modalities like image/audio, as simulating feature removal from input is not evident in these cases.

---

Interestingly, our interpretation module design allows us to simulate removal of set of components from the input. Given any sample z with predicted class c, we remove the set of relevant components
L0, {k Pk,c,z > } by creating a new time domain signal Z/2 = INV(X2, Pz), where x== X -IEELO, X We define faithfulness of the interpretation to classifier f for sample x with:
FF, - f(x) - f(22).
(6)
where f())e: (12)) denote the output probabilities for class c. It should be noted that this strategy to simulate removal may introduce artifacts in the input that can affect the classifier's 's output unpre- dictably. Also, interpretations on samples with poor fidelity can lead to negative FFz. Both of these observations point to the potential instability and outlying values for this metric. Thus, we report the final faithfulness of the system as median of FF, over test set, denoted by mediian- A positive mediann would signify that interpretations generally tend to be faithful to the classifier.
Evaluated systems. We denote our proposed Listen Interpret (I.21) system, with attention based pooling in e by L2I + ATT- The most suitable baselines to benchmark its fidelity are post-hoc methods that approximate the classifier over input space with single surrogate model. We select two state-of-the
systems, FLINT (32 and VIBI [4. A variant of our own proposed method, L2I + MAX is also evaluated. Herein, attention is replaced with ID max-pooling operation. Implementation details of the baselines are discussed in appendix A.7.
Faithfulness benchmarking: As already discussed, it is not possible to measure faithfulness for concept-based post-hoc interpretabilityy approaches. While measurement for input attribution based approaches is possible, the interpretations themselves and the feature removal strategies different, making comparisons with our system significantly less meaningful. We thus compare our faithfulness against a Random Baseline, wherein randomly chosen components are removed. To compare fairly, we remove the same number of components that are present in Le. on average. This would validate that, if the interpreter selects truly important components for the classifier's decision, then randomly removing the less important ones should not cause a drop in the predicted class probability.
We also emphasize at this point that works related to audio interpretability (see Sec. 2 are not suitable for comparison on these metrics. Particularly, APNet [47] is not designed for post-hoc interpretations. AudioLIME [8] is not applicable on our tasks as it requires known predefined audio sources. Moreover, SLIME [29] and AudioLIME still rely on LIME (36) for interpretations. It is a feature-atrribttion method that approximates classifier for each sample separately. As discussed before, these characteristics are not suitable for comparison on our metrics.
4.2 Implementation details
Classification network. We interpret a VGG-style convolutional neural network proposed by Kumar et al. (22). This network was chosen due to its popularity and applicability for various audio scene and event classification tasks. It can process variable length audio and has been pretrained on AudioSet 14]. a large-scale weakly labeled dataset for sound events. Further details about the network and its fine-tuning can be found in appendix A.2.
Hyperparameters and training. The hidden layers input to the interpreter module are selected from the convolutional block outputs. As is often the case with CNNs, the latter layers are expected to capture higher-order features. We thus select the last three convolutional block outputs as input to the network V. For ESC-50, we pre-learn the dictionary W with K 100 components and 80 components. Reasons for choice of K are discussed in
for SONYC-UST, we learn with appendix A.3. Ablation studies for other hyperparameters are in appendix A.4.
4.3 Experiments ESC-50
Dataset. ESC-50 35 is a popular benchmark for environmental sound classification task. It is a multi-class dataset that contains 2000 audio recordings of 50 different environmental sounds. The classes are broadly arranged in five categories namely, animals, natural soundscapes/water sounds, human/non-speed sounds, interior/domestic sounds, exterior/urban noises. Each clip is five-seconds long and has been extracted from publicly available recordings on the freesound. org project. The dataset is prearranged into 5 folds.
Classifier performance. The classifier achieves an accuracy of 82.5 + 1.9% over the 5 folds, higher than the average human accuracy of 81.3% on ESC-50.
7

---

Fidelity (in %
System
Threshold
FF
System
top-1
top-3
top-5
r 0.9
0.002
7 - 0.7
0.004
1.21 + QArr
65.7 + 2.8
81.8 2.2
88.2 + 1.7
1.21 + OAT
0.5
0.012
L21 OMAx
73.3 d: 2.3
87.8
92.7 1.2
r 0.3
0.040
FLINT
32
73,5
89.1 0.4
93.4 0,9
r 0.1
0.113
VIBI
2.3
45.4 2.2
53.0 1.8
Random Baseline
0.1
< 10
Table I: Quantitative results on ESC-50 environmental sound classification test data. (Left) top-k
fidelity (in %). FLINT and VIBI help benchmark fidelity but are not themselves suitable for audio interpretations. (Right) Faithfulness results (drop in probability) for different thresholds, We report modiian for proposed L2I+ + e xrr and the Random Baseline.
Quantitative results. Mean and standard deviation of top-k fidelity is calculated over the 5 folds. We show these results in Table (Leeft for k: = 1,3, 5. Among the four systems, VIBI performs the worst in terms of fidelity. This is very likely because it treats the classifier as black-box, while the other three systems access its hidden representations. This strongly indicates that accessing hidden layers can be beneficial for fidelity of interpreters. FLINT achieves the highest fidelity, very closely followed by L2I + OAAX and then L2I O This experiment serves as sanity check for our system, that while achieving fidelity performance comparable to state-of-the-ar we hold the advantage of providing listenable interpretations in terms of pre- reaane spectral patterns.
In Table T (Right), we report median faithfulness FF meddinn averaged over the 5 folds for our primary system 1.21 + AXT, at different thresholds т. Smaller T corresponds to higher |Lcal,, which denotes the number of components being used for generating interpretations. Thus, for Random Baseline, report FF FF modian at the lowest threshold r = 0.1, to ensure removal of maximal number of components. To recall the definition of Random Baseline, please refer Sec. 4.1 Fe for L2I OATT is positive for all thresholds. It is also significantly higher than the Random Baseline, indicating faithfulness of interpretations.
Audio corruption experiment: an interpretability illustration. We qualitatively illustrate that the interpretations are capable of emphasizing the object of interest and are insightful for end- to understand the classifier's prediction. To do so, we generate interpretations after corrupting the testing data for fold-1 in different ways (i) either with white noise at OdB SNR (signal-to-noise ratio), (ii) or mixing it with sample of different class. It should be noted that in both these cases the system is exactly the same before and not trained with corrupted samples. Some examples, covering both types of corruptions are shared on our companion website A detailed qualitative analysis of this experiment can be found in appendix A.5 along with discussion about interpretations from other methods in appendix A.6.
4.4 Experiments on SONYC-UST
We now discuss experiments for the urban sound tagging task from the well known Detection and Classification of Acoustic Scenes and Events (DCASE) challenge 2019 & 2020 edition.
Dataset. The DCASE task used a very challenging real-world dataset called SONYC-UST (
contains audio collected from multiple sensors placed in the New York City to monitor noise pollution. consists of eight coarse-level and 20 fine-level labels. We opt for the coarse-level labeling task that involves multi-label classification into: 'engine', 'machinery-impcct; 'non-machinn conn-machiner--ipaaae 'powered-sawe -aww', 'alert-signals', 'music', 'human-voice', 'dog". This task is highly challenging for several reasons: (i) since it is real-world audio, the samples contain very high level of background noise, (ii) the audio sources corresponding to the classes are often weak in intensity, as they are not necessarily close to the sensors, (iii) some classes may also be highly localized in time and more challenging to detect, (iv) lastly, noisy audio also makes it difficult to annotate, leading to labeling noise. This is especially true for training data that was labeled by volunteers.
Classifier performance. Our fine-tuned classifier achieves mUCRO-AUPRC (official metric for DCASE 2020 challenge) of 0.601. This is higher than the DCASE baseline performance of 0.510 and comparable to the best performing system macro-AUPRC of 0.649 (2. Note that is obtained without use of data augmentation or additional strategies to improve performance.
httpp::/ssanneepparekhh gwthhb..oo/lssttnnhherrreee

---

Quantitative results. In Table 2 we report the macro-AUPRC, micro-AUPRC and max-Fl for the interpreter output w.r.t classifier. For fairness, we ignore the class non-machinery impact' from all class-wise evaluations involved in fidelity (i.e. macro-AUPRC) or faithfulness. This is because the classifier predicts only one sample in test set with positive label for this class, causing AUPRC scores to vary widely for different interpreters. VIBI has the worst performance on all three metrics for this dataset as well. In contrast to ESC-50, here the best performing system is L2I + θATT followed by L2I + OMAX. and FLINT performing worse than both. The fidelity results on ESC-50 and SONYC-UST jointly demonstrate that our interpreter can generate high-fidelity post-hoc interpretations. Moreover, its design is flexible w.r.t different pooling functions.
The results for class-wiss faithfulness are illustrated in Fig. 2a We show FF mediaan (absolute drop in probability) for our system and the Random Baseline. The results indicate that, for most classes, interpretations can be considered faithful, with a significantly positive median compared to random baseline results, which are very close to O.
Qualitative observations. Qualitatively, we observe good interpretations for classes 'alert-signal', "dog" and 'music", For them, the background noise is significantly suppressed and the interpretations mainly focus on the object of interest. Interpretations for class 'human' are also able to suppress noise to a certain extent and focus on parts of human voices. However, for this class, we found presence of some signal from other audio sources too. For the remaining classes, namely 'Engine'. 'Powered-sawe and "Machineyy-mmp -imaact' the quality of the interpretation is more sample dependent. This is due to their acoustic similarity with the background noise. We provide example interpretations for SONYC-UST on our companion website. We present an additional visualization to demonstrate coherence of our interpretations appendix A.5.
Subjective evaluation. We perform a user study (15 participants) to evaluate quality and understand- ability of interpretations for L2I against SLIME on SONYC-UST test data. it worth emphasizing that understandability is one important aspect of an interpretation but is not necessarily related to its faithfulness, which should be evaluated separately, for example as proposed in Sec. 4.1 As discussed earlier, SLIME is not suitable for comparison on our quantitative metrics. Nevertheless, it the only relevant baseline for qualitative study of listenable interpretations. Details about SLIME implementation are in appendix A.7. A participant was provided with 10 input samples, a predicted class by the classifier for each sample and the corresponding interpretation audios from SLIME and 1.21. They were asked to rate the interpretations on a scale of 0-100 for the following question: "How well does the interpretation correspond to the part of input audio associated with the given class?". The 10 samples were randomly selected from a set of 36 (5-6 random test examples per class). For each sample, we ensured that the predicted class was both, present in the ground-truth and audible in input. Class-wise preference results and average ratings are shown in Fig. 2b L2I is preferred for 'music", 'dog' & 'alert-signal', SLIME is preferred for 'machiner--impae for others. A t-test with null hypothesis that the favourable system has lower mean score yielded p-value < 0.005 for 'music", < 0.05 for 'dog' and maahinerr-imppen and < 0.1 for 'alert-signal'.
no clear preference
4.5 Limitations
Finally, we list below some limitations of this study: (a) Tuning the hyperparameters requires some experience with deep architectures and audio. (b) We use phase of original input spectrogram for time-domain inversion. One could employ phase estimation algorithm to possibly improve over this strategy. (c) The current experiments are on two datasets and one network architecture. The design of the interpreter is dependent on task and architecture of base network. Our current design of Vi was proposed keeping in mind interpretations for CNN operating spectrogram-lice representations. Nevertheless, it should be appropriately experimented with and modified when applying on any new data or network architecture.
5 Conclusion
To sum up. we have presented system for post-hoc interpretation of networks that process audio. We posit that generating interpretations in terms of high-level audio objects and making them listenable are important attributes to aid understanding. Novel usage of NMF within our interpreter helps us satisfy both aforementioned requirements. Our original loss function formulation enables linking a classifier's decision to importance values over lee-ne NMF spectral dictionary through an intermediate encoding. We perform extensive evaluation over popular audio event analysis datasets. We present a ittt-k kind faithfulness evaluation for our non con-attribution based method. Finally,

---

Faithfulness SONYC UST 0:1)
5,PC antrig 1001
04g
Paect C2' Ban
Precis Handen naseine
5 (histal velee)
30
No0ee
14402
Deg
Alert
Husic
Engine
Powered
Human Nacknnery
(b)
Figure 2: (a) Faithfulness results (drop probability) for SONYC-UST arranged class-wise for threshold, = 0.1 (b) Subjective evaluation results. Average scores for 1.21 and SLIME and fraction of votes favour of each system are reported.
Fidelity
System
maPro-AUPRC
micro-AUPRC
max-FI
L.1 e AAT
0,909
0.011
0.917 + 0.008 0.847 + 0.010
L2I Guax
0.866
0.014
0.913 + 0.012
0.840 = 0.012
FLINT
0.816
0.013
0.907 = 0.011
0.825 = 0.012
VIBI
0.608 0.027
0.575 = 0.019 0.549 = 0.020
Table Fidelity results on SONYC-UST multi-label urban sound tagging task. We report AUPRC- based metrics and max Fl score for the interpreter w.r.t classifier's output (over three runs).
user study confirms usefulness of our listenable interpretations. Modular design of our system calls for further experimenting with decoder and other block architectures. We hope our work facilitates future research into designing modality-speccin interpreters that aid understanding.
Acknowledgments and Disclosure of Funding
This work has been funded by the research chair Data Science & Artificial Intelligence for Digitalized Industry and Services (DSAIDIS) of Télécom Paris. This work also benefited from the support of French National Research Agency (ANR) under reference ANR-20-CE23-0028 (LIMPID project). The authors would like to thank anonymous reviewers for their valuable comments and suggestions.
References
[1] David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in Neural Information Processing Systems (NeurIPS), pages 7775-7784, 2018.
[2] Augustin Arnault and Nicolas Riche. CRNNs for urban sound tagging with spatiotemporal context. Technical report, DCASE2020 Challenge, October 2020.
[3] Roland Badeau and Tuomas Virtanen. Nonnegative matrix factorization. Audio Source Separa- tion and Speech Enhancement, pages 131-160, 2018.
[4] Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. Explaining a black- box by using deep variational information bottleneck approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1396-11404, 2021.
[5] Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. arXiv preprint arXiv: 1807 1807.03418. 03448, 2018.
[6] Nancy Bertin, Roland Badeau, and Gaël Richard. Blind signal decompositions for automatic transcription of polyphonic music: NMF and K-SVD the benchmark. In IEEE International
10

---

Conference on Acoustics, Speech and Signal Processing (ICASSP), volume 1, pages 1-65. IEEE, 2007.
[7] Victor Bisot, Romain Serizel, Slim Essid, and Gaël Richard. Feature learning with matrix factorization applicd to acoustic scene classification. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(6):2116-1229 2017.
[8] Mark Cartwright, Ana Elisa Mendez Mendez, Jason Cramer, Vincent Lostanlen, Graham Dove, Ho-Hsiang Wu, Justin Salamon, Oded Nov, and Juan Bello. SONYC urban sound tagging (SONYC-UST): A multilabel dataset from urban acoustic sensor network. In Proceedings of the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), pages 35-39, October 2019.
[9] Mark Cartwright, Jason Cramer, Ana Elisa Mendez Mendez, Yu Wang, Ho-Hsiang Wu, Vin- cent Lostanlen, Magdalena Fuentes, Graham Dove, Charlie Mydlarz, Justin Salamon, al. SONYC-UST-V2: An urban sound tagging dataset with spatiotemporal context. arXiv preprint arXiv:2009.05788, 2020.
[10] Shreyan Chowdhury, Verena Praher, and Gerhard Widmer. Tracing back music emotion predictions to sound sources and intuitive perceptual qualities. In Sound and Music Computing Conference, pages 246-252, 2021.
[11] Christian Dittmar and Daniel Gärtner. Real-time transcription and separation of drum recordings based on NMF decomposition. In DAFx, pages 187-194, 2014.
[12] Cédric Févotte, Emmanuel Vincent, and Alexey Ozerov. Single-channel audio source separation with nmf: divergences, constraints and algorithms. Audio Source Separation, pages 1 1-24, 2018.
[13] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 53-53, 2018.
[14] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780. IEEE, 2017.
[15] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. In Advances in Neural Information Processing Systems (NeurIPS), pages 9277-9286, 2019.
[16] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Ka- gal. Explaining explanations: An overview of interpretability of machine learning. In IEEE International Conference Data Science and Advanced Analytics (DSAA), pages 80-89, 2018.
[17] JL Goldstein. Auditory nonlinearity. The Journal of the Acoustical Society of America, 41(3):
676-699, 1967.
[18] Verena Haunschmid, Ethan Manilow, and Gerhard Widmer. audiolime: Listenable explanations using source separation. arXiv preprint arXiv:2008.0000 00588, 2020.
[19] Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European Conference on Computer Vision, pages 3-19. Springer, 2016.
[20] Maximilian Ilse, Jakub Tomezak, and Max Welling, Attention-based deep multiple instance learning. In International Conference on Machine Learning, pages 2127-2136. PMLR, 2018.
[21] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pages 2668-2677 PMLR, 2018.

---

[22] Anurag Kumar, Maksim Khadkevich, and Christian Fügen. Knowledge transfer from weakly labeled audio using convolutional neural network for sound events and scenes. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 326- 330. IEEE, 2018.
[23] Jonathan Le Roux, John R Hershey, and Felix Weninger. Deep NMF for speech separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 66-70. IEEE, 2015.
[24] Jonathan Le Roux, Felix J Weninger. and John R Hershey. Sparse NMF-half-baker well donc? Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA, Tech. Rep., no. TR2015-023, 11:13-15, 2015.
[25] Daniel Lee and H. Sebastian Seung. Algorithms for Non-negative Matrix Factorization. In Advances in Neural Information Processing Systems, volume 13. MIT Press, 2001.
[26] AM Liberman, Franklin S Cooper, Donald P Shankweiler, and Michael Studdert- Kennedy. Why are speech spectrograms hard to read? American Annals of the Deaf, pages 127-133, 1968.
[27] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.
[28] Saumitra Mishra, Bob L Sturm, and Simon Dixon. Local interpretable model-agnostic explana- tions for music content analysis. In ISMIR, pages 537-543, 2017.
[29] Saumitra Mishra, Emmanouil Benetos, Bob LT Sturm, and Simon Dixon. Reliable local explanations for machine listening. In International Joint Conference on Neural Networks (IJCNN, pages 1-8. IEEE, 2020.
[30] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
[31] Hannah Muckenhirn, Vinayak Abrol, Mathew Magimai-Doss, and Sébastien Marcel. Un- derstanding and visualizing raw waveform-based CNNs. In Interspeech, pages 2349, 2019.
[32] Jayneel Parekh, Pavlo Mozharovskyi, and Florence d' d'Alché Buc. A Framework to Learn with Interpretation. In Advances in Neural Information Processing Systems, volume 34, pages 24273-24285, 2021.
[33] Sanjeel Parekh, Alexey Ozerov, Slim Essid, Ngoc QK Duong, Patrick Pérez, and Gaël Richard. Identify, locate and separate: Audio-visual object extraction in large video collections using weak supervision. In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 268 272. IEEE, 2019.
[34] Geoffroy Peeters and Gaël Richard. Deep learning for audio and music. In Multi-faceted Deep Learning: Models and Data, pages 231-266. Springer, 2021.
[35] Karol Piczak. ESC: Dataset for environmental sound classification. In ACM International Conference on Multimedia, pages 1015-1018, 2015.
[36] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1 144. ACM, 2016.
[37] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High- figh-precision model- agnostic explanations. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 32, 2018.
[38] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626, 2017.
12

---

[39] Paris Smaragdis. Non- non-negative matrix factor deconvolution; extraction of multiple sound sources from monophonic imputs. In International Conference on Independent Component Analysis and Signal Separation, pages 494 499. Springer, 2004.
[40] Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 2014.
[41] Sana Tonekaboni, Shalmali Joshi, Kieran Campbell, David K Duvenaud, and Anna Goldenberg. What went wrong and when? Instance- feature importance for time-series black-box models. Advances in Neural Information Processing Systems, 33:799-809, 2020.
[42] Kevin W Wilson, Bhiksha Raj, Paris Smaragdis, and Ajay Divakaran. Speech denoising using nonnegative matrix factorization with priors. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6029-4033 IEEE, 2008
[43] Scott Wisdom, Thomas Powers, James Pitton, and Les Atlas. Deep recurrent NMF for speech separation by unfolding iterative thresholding. In IEEE Workshop on Applications of Signal Processing Audio and Acoustics (WASPAA), pages 254-258. IEEE, 2017.
[44] Minz Won, Sanghyuk Chun, and Xavier Serra. Toward interpretable music tagging with self-attention. arXiv preprint arXiv: 049779996.497, 2019.
[45] Chih-Kuan Yeh, Been Kim, Sercan O Arik, Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfis- ter. On concept-based explanations in deep neural networks. arXiv preprint arXiv: 1910.07969, 2019.
[46] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. In International Conference on Learning Representations, 2018.
[47] Pablo Zinemanas, Martín Rocamora, Marius Miron, Frederic Font, and Xavier Serra. An interpretable deep learning model for automatic sound classification. Electronics, 10(7):850, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We discuss potential societal impacts in appendix A.9
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] All assets are free and public for research
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] Please appendix A.8
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] was optional for participants
14


Reviews:
[{"review_id": "viVPfr2GBxA", "reviewer": "Reviewer_GYnA", "summary": "This paper aims to provide a methodology to interpret the decisions of neural networks that does classification in the audio domain. The designed methodology, 'listen-to-interpret' (L2I) is called to be 'post-hoc', which designates the fact that the proposed methodology is suitable to interpret the outputs of neural networks that have already been trained. The proposed methodology makes use of Non-Negative Matrix Factorization (NMF) to reconstruct the parts of the input that influence the decision the most. The intermediate representation layers of the original network is given to a learnable layer (psi) to estimate the temporal activation coefficients of a pre-trained NMF dictionary. The learnable layer that estimates the NMF temporal activation coefficients, as well a pooling layer+linear layer (theta) are trained to minimize the deviation between the NMF reconstruction and the input spectrogram, and to minimize the deviation between the classification output of the interpreter module and the original classification output given by the network. \n\nThe authors claim that the proposed methodology is preferable over the other concept based interpretability methods as the proposed method is able to generate audio. Also, compared to the other existing methods in the litterature, the authors claim that listen-to-interpret is a post-hoc interpretation method, as opposed to being an interpretable system by-design. \n\nThe authors quantitatively compare their method L2I against FLINT and VIBI in terms of the fidelity metric. The results seem to indicate that L2I outperforms VIBI, gives similar performance compared to FLINT. On the DCASE dataset, the authors also do a user study and show that 'listen-to-interpret' in general leads to better opinions. ", "questions": "- More details should be given about how you used FLINT in your problem. I think it would be nice to show the interpretations obtained with FLINT. These are not audio signals, if I understand correctly, but it would be nice to see if the interpratations obtained with FLINT are useful or not. I think this is important as the faithfulness scores are similar with FLINT and listen to interpret. This can go in the appendix. \n\n- If I am understanding correctly, you are simply using a learnable vector of length T to pool the NMF activation vectors over time. I have two concerns with regards to this:\n\ta) You are fixing the length of this vector. This means that in test time your system is limited to recordings of length T. Am I right? \n\tb) Did you consider for instance using something simpler such as calculating the average over the time axis? I think it would help to comment on why you opted use this pooling method over something simpler such as averaging. In fact in 1, it seems like your max-pooling based method seems to work better in terms of fidelity, compared to the attention-vector based method.\n\n- In the in-line equation on $z=\\sum_t H_I(x) a$ on page 5, the argument does not have a $t$ index. It would be worth to add an index over t, so that the notation is mathematically correct.\n\n- In Figure 2, I would suggest using the same ordering with respect to classes in figures 1, 2. \n\n", "limitations": "I think this work presents a nice step towards postdoc-interpretability of audio-classification networks. However, one thing I am not super clear about is the motivation for using NMF. Is it because of the limited reconstruction ability of the linear model that enables the reconstruction of the input sound that is focused more on the salient parts of the input? One could imagine simply modifying the interpretation part of FLINT so that it construct audio e.g. with transposed convolutional layers? \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n\tThe paper proposes an interesting methodology that would be able to reconstruct the salient parts of the audio that supposedly affects the decision. In terms of the faithfulness metric, the reconstructed audio by the interpreter seems to outperform the VIBI approach and works about the same as the FLINT method. \n\tTo the best of my knowledge, the fact that L2I is able to reconstruct audio for posthoc-interpretability is a plus, as the earlier methods are not capable of that. \n\n- Weaknesses: \n\tThe method seems to be quite similar to FLINT. Basically the main difference (if I am not missing anything) is the fact that you use NMF decoding to reconstruct the audio signal. This is not a big deal, as the usage of NMF seems to provide some benefit for interpretability. I think it would be good to know if the FLINT interpratations (even though it's not the audio modality) are useful at all. You can potentially show the obtained features with FLINT.  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jWkMg-zSG-v", "reviewer": "Reviewer_1KUh", "summary": "The paper presents a novel approach for interpreting trained neural network models using audio as input. The approach is based on the NMF technique, and is using two neural networks: one to estimate the time activation of the NMF from the hidden representations of the trained model and one to approximate the output of the trained model using the time activations. The approach is evaluated on the SED task on two corpora. Two novel quantitative metrics are proposed, fidelity and faithfulness, and the proposed approach is compared to other audio interpretation baselines. The results show that the proposed approach is on par with the state-of-the-art while being able to provide listenable interpretations.", "questions": "- About the writing:\n    - I think the pre-learn NMF dictionary should be described more clearly in the main paper instead of being hidden in the Supplementary material, at least briefly. \n    - I don't really like the use of symbols as names for the two models in the Interpreter (\\Psi and \\Theta), it would helpful if they have a proper name, something like the \"Generator\" for Psi and the \"Approximator\" for Theta.\n- The proposed approach is a bit similar to the idea of \"relevance map\", i.e. finding what part of the input is the most useful for the classificaition, for instance in [1], the author should discuss that kind of approach.\n\n[1] Muckenhirn, Hannah, Vinayak Abrol, Mathew Magimai-Doss, and Sébastien Marcel. \"Understanding and Visualizing Raw Waveform-Based CNNs.\" In Interspeech, pp. 2345-2349. 2019.", "limitations": "The limitations are only addressed in the Supplementary material and not in the main paper. It's also quite short, the authors should discuss the performance of the interpreter on out-of-domain data for instance.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is generally well written and easy to follow.\n- The approach is clearly novel, especially for audio interpretability.\n- The experiments are well designed and thorough, clearly showing the capabilities of the approach. The provided audio examples are interesting and illustrate the approach well. The subjective evaluation is a very welcome addition.\n- The proposed approach is a good step towards more interpretability in neural networks, specially on the audio modality. \n\nWeaknesses:\n- The significance of the approach is limited, because:\n    - It is not clear if it would be applicable to other types of audio data and tasks, such as speech recognition, or to other modalities like images.\n    - I would generally hope that interpretability papers also bring new insights on what the neural networks have learned, but it's not the case in this paper, it's just showing that the model is doing what is expected, i.e. discard obvious irrelevant information. It would be more interesting to apply the method on other tasks where it's not obvious which part of the input is relevant to the task, for instance Speech Emotion Recognition.\n- The writing could be improved (see below).\n- Missing references in the related work.\n\nOverall, I recommend this paper for acceptance as the approach is novel and could be of interest of the NeurIPS community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cys8dzika5", "reviewer": "Reviewer_iYoX", "summary": "This paper introduces a novel method for post-hoc interpretability of a trained deep audio classifier in order to exhibit what components of a spectrogram is majorly used in the decision-making. To do so, the authors propose to rely on a pre-trained dictionary of source components learned through a NMF decomposition trained over the original dataset. Their model then tries to use the activation of the network as a conditioning information to learn activation matrices of the corresponding dictionary. In order to ensure that this \"interpreter\" converges to an adequate solution, the learning objectives are both to reconstruct the original spectrogram, and also to deduce a similar representation to that of the final layer of the pre-trained network.", "questions": "- I feel that the bottleneck of your proposal lies in both the properties of the components (W) matrix. The quality of the results seem to be bounded by the cardinality of this set ?\n- Could you provide more details on the classifier performance (sec. 4.3) and also shortly discuss the applicability of your method to other types of networks ?\n- It is hard to gauge the result based on the \"FF\" (since it appears that it is a home-crafted metric), so the evaluation appears weakened. Could you provide a range of the FF as it is hard to know if \"1.29\" is a good result (since your baseline gives 0.0)", "limitations": "- You do not discuss at all the impact of the phase in your inversion. From what I understood, you simply use the phase information of the original spectrogram ? However, you could still have some incoherence depending on what you select, and if your masking information is a combination of several external components of W. Wouldn't it make sense to perform at least a few iterations of a phase reconstruction algorithm (Griffin-lim, or even better in your case as you have a pretty decent estimate to begin with, using PGHI to correct phase imperfections) ?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I feel that the topic of interpretability is of prime importance and the authors also nicely motivate their overall underlying motivations for this subject. The paper itself is nicely written and clearly exposes the different constituent aspects of the proposed approach. I think that the method is quite original and there are some interesting contributions to the interpretability field in general.\n\nRegarding weaknesses, I think that my major concern is on the overall philosophy behind the approach. \n- First, I feel that this approach could be quite easily compared to the \"attribution\" literature, which I did not see in the references. For instance, the following paper\nSchulz, K., Sixt, L., Tombari, F., & Landgraf, T. (2020). Restricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.00396.\nhttps://arxiv.org/pdf/2001.00396.pdf\n Wouldn't it be simply possible to use such attribution methods as a spectral masking information in order to perform inverse STFT (akin to source separation) on the original spectrogram ? The strength of your approach is in uncovering the time activations matrices, but I feel that it is under-exploited in your results analysis.\n- Second, I am wondering what would be the true use of the corresponding extracted sounds outside of the realm of simple classification tasks (eg. \"car alarm\" \"dog barking\"). I am unsure that this method could be insightful if it was applied to more complex tasks such as music genre classification ?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "02w5FpZ1i_F", "reviewer": "Reviewer_AaCp", "summary": "This paper proposes a system for interpreting deep audio classifiers. It works by creating audio excerpts of the parts of the input most relevant to the classifier's decision. To this end, a dictionary of spectral components is learned using NMF. Additionally, representations from a trained audio classifier are extracted and passed to an interpreter network that estimates activations for these NMF components. The interpreter is trained using a loss that encourages reconstruction of the input from the estimated activations and (fixed) spectral components as well as a loss for recovering the classifier's prediction from the activations. The system is evaluated on two audio classification datasets with regard to fidelity (how well does the interpreter recover the classifier's predictions?) and faithfulness (how well does the interpreter determine parts of the input that are relevant to the classifier's decision?). The proposed system admits a natural formulation for measuring faithfulness.", "questions": "I have a number of questions and concerns that should be clarified before accepting the paper.\n\nMajor:\n- The proposed method contains learnable parameters. This raises the question: How much do the interpretations obtained depend on the dataset used for training the interpreter? Do the interpretations give insights into the classifier's behavior or are they merely artifacts of interpreter training? These questions should be openly discussed in the text. An interesting experiment to perform would be a cross-dataset experiment. How does the system perform when combining a ESC-50 classifier with a SONYC-UST-trained interpreter and vice-versa? Do we still obtain useful interpretations? How are fidelity and faithfulness impacted?\n- In line 239, it is mentioned that for computing faithfulness in the multi-class case, changes in logit values instead of changes in probability are used. This makes no sense to me. This way, one can obtain a positive faithfulness value even thought the probability predicted for $c$ has increased for $x_2$ compared to $x$! Additionally, the use of logits here means that the $FF_{median}$ values reported in Table 1 (right) have no interpretable units. What does $1.29$ mean here? Is this a lot or is it little? Probabilities should be used for the multi-class case as well.\n- The definition of the \"Random Baseline\" in line 256 is problematic. Here, components that are not in $L_{c,x}$ are removed which potentially makes this baseline overly pessimistic. It would make more sense (and correspond much more with the term \"Random Baseline\") if a random subset of all components was removed (not only those that are not in $L_{c,x}$). In addition, please report: What is the total number of components? What is the average size of $L_{c,x}$ for different $\\tau$?\n- The subjective listening test as described in this paper is highly questionable and should be removed. The test asked participants to rate the similarity between interpretations and sounds associated with the ground truth class. But that is not necessarily what the interpretations should sound like! The interpretations should correspond to whatever parts of the input habr been used for the classifier's prediction. These parts may or may not correspond to sounds associated with the ground truth class. The listening test presented here is unable to capture anything about the classifier's decision process. It is likely that the subjective scores correspond more to overall separation or sound quality. In addition, no kind of significance testing has been performed.\n\nMinor:\n- It is not clear from the main text of the paper how phase information is obtained to generate the listenable outputs from magnitude spectrograms. From Algorithm 2, which mentions \"Input Phase\", I presume that the phase of the original, unmodified input spectrogram is used. This should be clarified in the text and it should be emphasized that it will lead to artifacts.\n- How do you ensure non-negativity of $H_\\mathcal{I}(x)$ while minimizing $\\mathcal{L}_{NMF}$? If $H_\\mathcal{I}(x)$ is not constrained to be non-negative, the resulting decoder does not perform \"true\" NMF but rather semi-NMF. The resulting decomposition would not be truly part-based. This should be clarified in the text.\n- It is unclear why the additional $l_1$ regularization on $H_\\mathcal{I}(x)$ is used. What is the impact of this choice? Line 185 mentions \"well-behavedness, especially for large dictionary sizes\", but the reference given is for Sparse NMF, which solves a different optimization problem. In Sparse NMF, both $W$ and $H$ are unknown, whereas here, $W$ is fixed, right?\n- Algorithm 1 is an overly mathy way of writing a very straightforward thing and should be dropped. It is clear from the text that the dictionary is learned prior to training the interpreter.\n- As a baseline, it would be more interesting to compare the proposed architecture with a $\\Psi$ that takes the input $x$ directly instead of intermediate representations. This way, the impact of utilizing intermediate representations would have been more clearly demonstrated. The paper does contain comparisons with some published baselines that directly work on the input $x$, but these baselines also differ from the proposed system in other aspects.\n- Why is $\\tau=0.1$ in Figure 2? What is the impact of $\\tau$ here? For the other dataset, different values of $\\tau$ were used.\n- Section A.1.1 in the supplemental: It is mentioned that a special procedure was performed to reduce the number of noise-like components in the dictionary for SONYC-UST. What is the impact of this? Does it lead to better or worse fidelity or faithfulness? Do the generated interpretations sound more natural?\n- Section A.1.3 in the supplemental: This is very confusing. Firstly, the number of components that is ultimately chosen for the proposed system should be mentioned in the main text and this section should then be referenced. Secondly, what is $\\mathcal{L}_{of}$? Thirdly, are we looking at the test loss of the interpreter here? Why not simply look at the loss during dictionary learning and choose K depending on that?\n- Line 58 in the supplemental: Which early stopping criterion has been used? In other words: how have the numbers 35 and 21 been obtained?\n- Tables 1 and 2 in the supplemental: On which datasets have these values been computed?\n- The accompanying website is quite nice, but rather than just showing the audio examples and spectrograms, it would be very important to also give the corresponding probabilities predicted by the original classifier. After all, we are not interested in seeing source separation performance here but in explaining the classifier predictions.\n- Throughout the paper and supplemental material, the text in figures is too small (with the exception of most of the text in Figure 1). The text in Figure 2 of the main paper is much (!) too small.\n- More care should be taken in preparing the bibliography. For example, different styles are used to refer to the DCASE workshop in [2] and [8], a URL is given for [25] but for no other bibliography item, references for some venues are formatted inconsistently (e.g. WASPAA [40], [41] or NeurIPS [1], [39]), ...\n\nAdditional points:\n- $x$ is first defined in line 117 and then redefined in line 123. This is confusing.\n- Line 146 refers to some $f_\\mathcal{I}(x)$ which is only defined in line 154. This is confusing.\n- Equations (1) and (2) should be dropped. Cross-entropy losses are textbook material.\n- Contractions should be avoided in the text (e.g. \"It is\" instead of \"It's\" in line 40).", "limitations": "The discussion on limitations of the proposed method could be extended. It should be emphasized that the interpretations obtained by this method depend on the datasets used for training the interpreter components and networks $\\Theta$ and $\\Psi$. Additionally, this section should mention that the method has only been applied to one classifier architecture and two datasets and that it may behave differently in other scenarios. It is also worth noting that the proposed method is quite computationally expensive (taking 150 minutes to train for one of the datasets, see Section A.1.4).\n\nWith regard to societal impact, the biggest problems would probably arise not from misuse of the proposed method but from false confidence in the given interpretations. At this point, all interpretability methods for deep learning systems should be approached with care. This is especially true for systems that intrdoduce additional learnable parameters (as is the case for the present submission). This should be clarified in the text.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a well written paper with a neat and original idea at its core. The proposed incorporation of NMF into an interpretability pipeline is interesting and the ability to obtain listenable interpretations is very useful. I particularly liked that the proposed system admits a very natural formulation for measuring faithfulness of interpretations. The proposed system, or some of the ideas behind it, are likely to be used by other works in the future.\n\nI do have several questions about technical details and concerns about some of the results presented. The experimental results are not totally convincing in their current form and one evaluation (the listening test) is highly problematic and should be removed altogether.\n\nNevertheless, if the points below are addressed, I think this could be a valuable contribution. For now, I recommend a borderline accept. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept