Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a novel approach for interpreting trained neural network models using audio as input. The approach is based on the NMF technique, and is using two neural networks: one to estimate the time activation of the NMF from the hidden representations of the trained model and one to approximate the output of the trained model using the time activations. The approach is evaluated on the SED task on two corpora. Two novel quantitative metrics are proposed, fidelity and faithfulness, and the proposed approach is compared to other audio interpretation baselines. The results show that the proposed approach is on par with the state-of-the-art while being able to provide listenable interpretations.
strengths_and_weaknesses: Strengths:
- The paper is generally well written and easy to follow.
- The approach is clearly novel, especially for audio interpretability.
- The experiments are well designed and thorough, clearly showing the capabilities of the approach. The provided audio examples are interesting and illustrate the approach well. The subjective evaluation is a very welcome addition.
- The proposed approach is a good step towards more interpretability in neural networks, specially on the audio modality. 

Weaknesses:
- The significance of the approach is limited, because:
    - It is not clear if it would be applicable to other types of audio data and tasks, such as speech recognition, or to other modalities like images.
    - I would generally hope that interpretability papers also bring new insights on what the neural networks have learned, but it's not the case in this paper, it's just showing that the model is doing what is expected, i.e. discard obvious irrelevant information. It would be more interesting to apply the method on other tasks where it's not obvious which part of the input is relevant to the task, for instance Speech Emotion Recognition.
- The writing could be improved (see below).
- Missing references in the related work.

Overall, I recommend this paper for acceptance as the approach is novel and could be of interest of the NeurIPS community.
questions: - About the writing:
    - I think the pre-learn NMF dictionary should be described more clearly in the main paper instead of being hidden in the Supplementary material, at least briefly. 
    - I don't really like the use of symbols as names for the two models in the Interpreter (\Psi and \Theta), it would helpful if they have a proper name, something like the "Generator" for Psi and the "Approximator" for Theta.
- The proposed approach is a bit similar to the idea of "relevance map", i.e. finding what part of the input is the most useful for the classificaition, for instance in [1], the author should discuss that kind of approach.

[1] Muckenhirn, Hannah, Vinayak Abrol, Mathew Magimai-Doss, and Sébastien Marcel. "Understanding and Visualizing Raw Waveform-Based CNNs." In Interspeech, pp. 2345-2349. 2019.
limitations: The limitations are only addressed in the Supplementary material and not in the main paper. It's also quite short, the authors should discuss the performance of the interpreter on out-of-domain data for instance.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a novel method for post-hoc interpretability of a trained deep audio classifier in order to exhibit what components of a spectrogram is majorly used in the decision-making. To do so, the authors propose to rely on a pre-trained dictionary of source components learned through a NMF decomposition trained over the original dataset. Their model then tries to use the activation of the network as a conditioning information to learn activation matrices of the corresponding dictionary. In order to ensure that this "interpreter" converges to an adequate solution, the learning objectives are both to reconstruct the original spectrogram, and also to deduce a similar representation to that of the final layer of the pre-trained network.
strengths_and_weaknesses: Overall, I feel that the topic of interpretability is of prime importance and the authors also nicely motivate their overall underlying motivations for this subject. The paper itself is nicely written and clearly exposes the different constituent aspects of the proposed approach. I think that the method is quite original and there are some interesting contributions to the interpretability field in general.

Regarding weaknesses, I think that my major concern is on the overall philosophy behind the approach. 
- First, I feel that this approach could be quite easily compared to the "attribution" literature, which I did not see in the references. For instance, the following paper
Schulz, K., Sixt, L., Tombari, F., & Landgraf, T. (2020). Restricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.00396.
https://arxiv.org/pdf/2001.00396.pdf
 Wouldn't it be simply possible to use such attribution methods as a spectral masking information in order to perform inverse STFT (akin to source separation) on the original spectrogram ? The strength of your approach is in uncovering the time activations matrices, but I feel that it is under-exploited in your results analysis.
- Second, I am wondering what would be the true use of the corresponding extracted sounds outside of the realm of simple classification tasks (eg. "car alarm" "dog barking"). I am unsure that this method could be insightful if it was applied to more complex tasks such as music genre classification ?

questions: - I feel that the bottleneck of your proposal lies in both the properties of the components (W) matrix. The quality of the results seem to be bounded by the cardinality of this set ?
- Could you provide more details on the classifier performance (sec. 4.3) and also shortly discuss the applicability of your method to other types of networks ?
- It is hard to gauge the result based on the "FF" (since it appears that it is a home-crafted metric), so the evaluation appears weakened. Could you provide a range of the FF as it is hard to know if "1.29" is a good result (since your baseline gives 0.0)
limitations: - You do not discuss at all the impact of the phase in your inversion. From what I understood, you simply use the phase information of the original spectrogram ? However, you could still have some incoherence depending on what you select, and if your masking information is a combination of several external components of W. Wouldn't it make sense to perform at least a few iterations of a phase reconstruction algorithm (Griffin-lim, or even better in your case as you have a pretty decent estimate to begin with, using PGHI to correct phase imperfections) ?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a system for interpreting deep audio classifiers. It works by creating audio excerpts of the parts of the input most relevant to the classifier's decision. To this end, a dictionary of spectral components is learned using NMF. Additionally, representations from a trained audio classifier are extracted and passed to an interpreter network that estimates activations for these NMF components. The interpreter is trained using a loss that encourages reconstruction of the input from the estimated activations and (fixed) spectral components as well as a loss for recovering the classifier's prediction from the activations. The system is evaluated on two audio classification datasets with regard to fidelity (how well does the interpreter recover the classifier's predictions?) and faithfulness (how well does the interpreter determine parts of the input that are relevant to the classifier's decision?). The proposed system admits a natural formulation for measuring faithfulness.
strengths_and_weaknesses: This is a well written paper with a neat and original idea at its core. The proposed incorporation of NMF into an interpretability pipeline is interesting and the ability to obtain listenable interpretations is very useful. I particularly liked that the proposed system admits a very natural formulation for measuring faithfulness of interpretations. The proposed system, or some of the ideas behind it, are likely to be used by other works in the future.

I do have several questions about technical details and concerns about some of the results presented. The experimental results are not totally convincing in their current form and one evaluation (the listening test) is highly problematic and should be removed altogether.

Nevertheless, if the points below are addressed, I think this could be a valuable contribution. For now, I recommend a borderline accept. 
questions: I have a number of questions and concerns that should be clarified before accepting the paper.

Major:
- The proposed method contains learnable parameters. This raises the question: How much do the interpretations obtained depend on the dataset used for training the interpreter? Do the interpretations give insights into the classifier's behavior or are they merely artifacts of interpreter training? These questions should be openly discussed in the text. An interesting experiment to perform would be a cross-dataset experiment. How does the system perform when combining a ESC-50 classifier with a SONYC-UST-trained interpreter and vice-versa? Do we still obtain useful interpretations? How are fidelity and faithfulness impacted?
- In line 239, it is mentioned that for computing faithfulness in the multi-class case, changes in logit values instead of changes in probability are used. This makes no sense to me. This way, one can obtain a positive faithfulness value even thought the probability predicted for $c$ has increased for $x_2$ compared to $x$! Additionally, the use of logits here means that the $FF_{median}$ values reported in Table 1 (right) have no interpretable units. What does $1.29$ mean here? Is this a lot or is it little? Probabilities should be used for the multi-class case as well.
- The definition of the "Random Baseline" in line 256 is problematic. Here, components that are not in $L_{c,x}$ are removed which potentially makes this baseline overly pessimistic. It would make more sense (and correspond much more with the term "Random Baseline") if a random subset of all components was removed (not only those that are not in $L_{c,x}$). In addition, please report: What is the total number of components? What is the average size of $L_{c,x}$ for different $\tau$?
- The subjective listening test as described in this paper is highly questionable and should be removed. The test asked participants to rate the similarity between interpretations and sounds associated with the ground truth class. But that is not necessarily what the interpretations should sound like! The interpretations should correspond to whatever parts of the input habr been used for the classifier's prediction. These parts may or may not correspond to sounds associated with the ground truth class. The listening test presented here is unable to capture anything about the classifier's decision process. It is likely that the subjective scores correspond more to overall separation or sound quality. In addition, no kind of significance testing has been performed.

Minor:
- It is not clear from the main text of the paper how phase information is obtained to generate the listenable outputs from magnitude spectrograms. From Algorithm 2, which mentions "Input Phase", I presume that the phase of the original, unmodified input spectrogram is used. This should be clarified in the text and it should be emphasized that it will lead to artifacts.
- How do you ensure non-negativity of $H_\mathcal{I}(x)$ while minimizing $\mathcal{L}_{NMF}$? If $H_\mathcal{I}(x)$ is not constrained to be non-negative, the resulting decoder does not perform "true" NMF but rather semi-NMF. The resulting decomposition would not be truly part-based. This should be clarified in the text.
- It is unclear why the additional $l_1$ regularization on $H_\mathcal{I}(x)$ is used. What is the impact of this choice? Line 185 mentions "well-behavedness, especially for large dictionary sizes", but the reference given is for Sparse NMF, which solves a different optimization problem. In Sparse NMF, both $W$ and $H$ are unknown, whereas here, $W$ is fixed, right?
- Algorithm 1 is an overly mathy way of writing a very straightforward thing and should be dropped. It is clear from the text that the dictionary is learned prior to training the interpreter.
- As a baseline, it would be more interesting to compare the proposed architecture with a $\Psi$ that takes the input $x$ directly instead of intermediate representations. This way, the impact of utilizing intermediate representations would have been more clearly demonstrated. The paper does contain comparisons with some published baselines that directly work on the input $x$, but these baselines also differ from the proposed system in other aspects.
- Why is $\tau=0.1$ in Figure 2? What is the impact of $\tau$ here? For the other dataset, different values of $\tau$ were used.
- Section A.1.1 in the supplemental: It is mentioned that a special procedure was performed to reduce the number of noise-like components in the dictionary for SONYC-UST. What is the impact of this? Does it lead to better or worse fidelity or faithfulness? Do the generated interpretations sound more natural?
- Section A.1.3 in the supplemental: This is very confusing. Firstly, the number of components that is ultimately chosen for the proposed system should be mentioned in the main text and this section should then be referenced. Secondly, what is $\mathcal{L}_{of}$? Thirdly, are we looking at the test loss of the interpreter here? Why not simply look at the loss during dictionary learning and choose K depending on that?
- Line 58 in the supplemental: Which early stopping criterion has been used? In other words: how have the numbers 35 and 21 been obtained?
- Tables 1 and 2 in the supplemental: On which datasets have these values been computed?
- The accompanying website is quite nice, but rather than just showing the audio examples and spectrograms, it would be very important to also give the corresponding probabilities predicted by the original classifier. After all, we are not interested in seeing source separation performance here but in explaining the classifier predictions.
- Throughout the paper and supplemental material, the text in figures is too small (with the exception of most of the text in Figure 1). The text in Figure 2 of the main paper is much (!) too small.
- More care should be taken in preparing the bibliography. For example, different styles are used to refer to the DCASE workshop in [2] and [8], a URL is given for [25] but for no other bibliography item, references for some venues are formatted inconsistently (e.g. WASPAA [40], [41] or NeurIPS [1], [39]), ...

Additional points:
- $x$ is first defined in line 117 and then redefined in line 123. This is confusing.
- Line 146 refers to some $f_\mathcal{I}(x)$ which is only defined in line 154. This is confusing.
- Equations (1) and (2) should be dropped. Cross-entropy losses are textbook material.
- Contractions should be avoided in the text (e.g. "It is" instead of "It's" in line 40).
limitations: The discussion on limitations of the proposed method could be extended. It should be emphasized that the interpretations obtained by this method depend on the datasets used for training the interpreter components and networks $\Theta$ and $\Psi$. Additionally, this section should mention that the method has only been applied to one classifier architecture and two datasets and that it may behave differently in other scenarios. It is also worth noting that the proposed method is quite computationally expensive (taking 150 minutes to train for one of the datasets, see Section A.1.4).

With regard to societal impact, the biggest problems would probably arise not from misuse of the proposed method but from false confidence in the given interpretations. At this point, all interpretability methods for deep learning systems should be approached with care. This is especially true for systems that intrdoduce additional learnable parameters (as is the case for the present submission). This should be clarified in the text.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to provide a methodology to interpret the decisions of neural networks that does classification in the audio domain. The designed methodology, 'listen-to-interpret' (L2I) is called to be 'post-hoc', which designates the fact that the proposed methodology is suitable to interpret the outputs of neural networks that have already been trained. The proposed methodology makes use of Non-Negative Matrix Factorization (NMF) to reconstruct the parts of the input that influence the decision the most. The intermediate representation layers of the original network is given to a learnable layer (psi) to estimate the temporal activation coefficients of a pre-trained NMF dictionary. The learnable layer that estimates the NMF temporal activation coefficients, as well a pooling layer+linear layer (theta) are trained to minimize the deviation between the NMF reconstruction and the input spectrogram, and to minimize the deviation between the classification output of the interpreter module and the original classification output given by the network. 

The authors claim that the proposed methodology is preferable over the other concept based interpretability methods as the proposed method is able to generate audio. Also, compared to the other existing methods in the litterature, the authors claim that listen-to-interpret is a post-hoc interpretation method, as opposed to being an interpretable system by-design. 

The authors quantitatively compare their method L2I against FLINT and VIBI in terms of the fidelity metric. The results seem to indicate that L2I outperforms VIBI, gives similar performance compared to FLINT. On the DCASE dataset, the authors also do a user study and show that 'listen-to-interpret' in general leads to better opinions. 
strengths_and_weaknesses: - Strengths:
	The paper proposes an interesting methodology that would be able to reconstruct the salient parts of the audio that supposedly affects the decision. In terms of the faithfulness metric, the reconstructed audio by the interpreter seems to outperform the VIBI approach and works about the same as the FLINT method. 
	To the best of my knowledge, the fact that L2I is able to reconstruct audio for posthoc-interpretability is a plus, as the earlier methods are not capable of that. 

- Weaknesses: 
	The method seems to be quite similar to FLINT. Basically the main difference (if I am not missing anything) is the fact that you use NMF decoding to reconstruct the audio signal. This is not a big deal, as the usage of NMF seems to provide some benefit for interpretability. I think it would be good to know if the FLINT interpratations (even though it's not the audio modality) are useful at all. You can potentially show the obtained features with FLINT.  


questions: - More details should be given about how you used FLINT in your problem. I think it would be nice to show the interpretations obtained with FLINT. These are not audio signals, if I understand correctly, but it would be nice to see if the interpratations obtained with FLINT are useful or not. I think this is important as the faithfulness scores are similar with FLINT and listen to interpret. This can go in the appendix. 

- If I am understanding correctly, you are simply using a learnable vector of length T to pool the NMF activation vectors over time. I have two concerns with regards to this:
	a) You are fixing the length of this vector. This means that in test time your system is limited to recordings of length T. Am I right? 
	b) Did you consider for instance using something simpler such as calculating the average over the time axis? I think it would help to comment on why you opted use this pooling method over something simpler such as averaging. In fact in 1, it seems like your max-pooling based method seems to work better in terms of fidelity, compared to the attention-vector based method.

- In the in-line equation on $z=\sum_t H_I(x) a$ on page 5, the argument does not have a $t$ index. It would be worth to add an index over t, so that the notation is mathematically correct.

- In Figure 2, I would suggest using the same ordering with respect to classes in figures 1, 2. 


limitations: I think this work presents a nice step towards postdoc-interpretability of audio-classification networks. However, one thing I am not super clear about is the motivation for using NMF. Is it because of the limited reconstruction ability of the linear model that enables the reconstruction of the input sound that is focused more on the salient parts of the input? One could imagine simply modifying the interpretation part of FLINT so that it construct audio e.g. with transposed convolutional layers?
review_title: null

Meta-review: metareview: The paper presents a novel approach for interpreting a neural network's decision on audio input. The motivation of the research is clear, and all the reviewers have agreed on the importance of the addressed task and the originality of the proposed solution. Also, the authors have adequately responded to the reviewers' concerns during the rebuttal, including some additional experiments such as a comparison with attribution interpretation or a modified baseline. Therefore, I gladly recommend this paper be accepted in NeurIPS 2022. 

## Strengths
- The paper addresses one of the underexplored yet important tasks of interpretability of neural networks for the audio domain. As visualization is helpful for interpreting neural networks in the vision domain, sonification is a reasonable and intuitive way to analyze how the neural network works on audio. This paper presents a promising solution for this purpose.
- The architecture of the proposed model is original and shows high novelty. The experiment for the evaluation was well-designed, showing meaningful results.

## Weaknesses
- Many reviewers suggested that it would be interesting to see how the proposed model works on more complex data, such as music audio. The authors have explained why they focused on sound event detection as an initial goal. The reviewer's suggestion would remain a challenging but valuable goal for future research.

confidence: Certain
Decision: Accept