Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper Summary

The authors propose a method to explore how neurons interact within a neural network and derive rules of interactions that can help interpret the inner workings of the neural network and open up the black box. The algorithm, EXPLAINN, identifies rules between successive layers where each rule represents a set of neurons that are activate simultaneously and conditionally based on the previous layer. Minimum Description Length principle is used to derive an objective that minimizes the number of bits used to encode the rules. The rule sets are identified using a greedy heuristic and improved until convergence of the objective. The algorithm is then evaluated to demonstrate the interpretation of images with MNIST, GoogLeNet and VGG-S. 


Positives
* The problem at hand is clearly important and can help with interpretation of the neural networks.  This is particularly important in fields such as biology (genomics) where better interpretation is often times desired at a slight cost in performance 
* The formulation using Minimum Description Length Principle is definitely an interesting idea - particularly formulating the objective as set of independent and robust rules 
* The framework is flexible and allows for discovery of rules relevant for subsets / combinations of classes and not just individual examples or global rules across examples 
* The paper in general is well written and I particularly liked how the authors guide the readers through the different aspects of the algorithm evaluation. I particularly liked Appendix A since the concrete example made the objective function clear. The notations however can be better laid out and clarified. 


Concerns
* My primary concern is the use of greedy heuristic to identify the rule set. This necessitates that each interaction individually carry some degree of information for a robust composite rule. While the rules identified through the heuristic could be informative, it is not clear whether they are the best set of rules since there are no bounds or guarantees of how the heuristic relates to the global optimum. 
* On a related note, I felt the evaluation presented by authors while extensive is rather qualitative in nature. While the prototypes of identified rules across different datasets look relevant and interesting, it is not clear whether they are the best set of rules. So, I believe this manuscript needs benchmarking in datasets constructed with known rules (possibly through simulation) to alleviate these two major concerns 
* The quantitative outputs of the ReLU activations are binarized for rule learning - the authors have not addressed how this impacts the accuracy of the rules since that quantitative information is used by the subsequent layers 
* The impact of the threshold parameters theta and mu (Algorithms 1 & 2) are also not addressed - this seems critical to me since these thresholds define the initial set of rule candidates.


review_title: Review for Paper 3168

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an approach to explainable supervised learning by extracting sets of rules for two individual layers within a neural network. The authors build their work on recent published work for patttern-based rule mining [0] to efficently find so-called robust rules. The authors evaluate the approach for image processing tasks with convolutional neural networks on MNIST, ImageNet and Oxford Flower by comparing generated rules against activation maps and prototypes.

The proposed approach is interesting, but I am left with several concerns and open questions with respect to contributions, available related works and the conducted evaluation. 

While the paper discusses quite nicely summarized several relevant related works in the introduction section, I am still wondering how the approach is related to OpenAI's Circuits [1], where already individual neurons and connections between neurons are studied with respect to interpretability/explainability of neural nets? This is not to say that prior work already learned sets of rules among layers, but - if relevant - it should be evaluated to what extent the proposed method is superior for explaining neural nets to end-users. A smaller comment to the related work is that there might be missing references for model destillation, such as [2].

The taken approach to rule generation follows recent work on association rule mining, which is sensible. To this end, the paper misses to clearly address the difference to GRAB, i.e. the referenced algorithm used for learning the rules using model description length (MDL). Could you therefore elaborate why GRAB cannot be applied to the rule mining task out-of-the-box or clearer state if this what you have actually done? I feel like the presentation of the paper could benefit from answering this question, one could  establish a background section and/or focus on novel aspects of the proposed algorithm. In addition, there might be more space for the evaluation, which should be a major part of the paper's contribution, especially explainable ML.

The idea of exploring neural connections among two layers is generally intriguing. The paper could benefit from better motivating this design choice, i.e. why use "only" two layers? Would it make sense to use more than two convolutional layers for deeper networks? Would this be computationally tractable? Are two layers better to interpret for humans?

While the evaluation is insightful, I am not convinced that single-neuron-prototypes and activation maps are representative for all ongoing works on explainable and interpretable ML. More specifically with respect to activation maps, is it possible to rule out all activation map approaches for CNNs at once? What about perturbation-based approaches (e.g. [3])? To this end, I am wondering why you only compare to single neuron prototypes and not more complex prototypes of the individual class (for Sec. 3.1 and the MNIST experiment)? Lastly, I am also wondering if a user study would be helpful to confirm that the proposed explanations provide added value for end-users.  

A minor comment is that it is not helpful for the reader that numerous references figures are in the appendix (e.g. the comparison of generated MNIST rules in Sec. 3.1 to the prototype in the appendix). A minor question would be why you define "prototype" as on a single-neuron-basis (in the introduction)? Is there a reference for the approach?

References:
[0] Fischer, J. and Vreeken, J., 2019, September. Sets of robust rules, and how to find them. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 38-54). Springer, Cham.
[1] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Zoom In: An Introduction to Circuits. Distill, 5(3), pp.e00024-001.
[2] Bastani, O., Kim, C. and Bastani, H., 2017. Interpreting blackbox models via model extraction. arXiv preprint arXiv:1705.08504.
[3] Fong, R.C. and Vedaldi, A., 2017. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3429-3437).

### Update after author response ###
Thanks again for the clarifications - After reading the author responses, the other reviewers' comments and the new version of the manuscript, I increase my score for the paper, as the authors now better state the relationship to Circuits and GRAB, and provide a significantly improved evaluation. The enhanced experimental section is now adequate for the paper's claims and offers additional insights into the usability of the generated rules. 
review_title: The proposed method for black-box model exploration/inspection is sensible and provides a good evaluation, but I am not yet convinced of the contributions, distinction to related works and the empirical evaluation.

final_score: rating: 3: Clear rejection
review_content: This paper proposes to extract interpretable rules from a learned neural network. The authors claim that they are the first to propose rules connecting 1) multiple neurons together, and 2) do this at a dataset level. Their approach relies on using minimum description length and well known principles from the data mining community (e.g., downward closure lemma of apriori algorithm). The authors claim that experiments conducted on image data shows that their approach leads to more faithful, interpretable rules than other approaches such as prototyping or model distillation.

Compiling rules from neural networks has been proposed before (e.g., see "Deep Logic Networks: Inserting and Extracting Knowledge from Deep Belief Networks" by Tran and d'Avila Garcez, 2018). I think the authors need to compare against such previous approaches to quantitatively show how their work extracts better rules. Otherwise, its difficult to appreciate the value of ExplaiNN. Also, the paper doesn't say anything about how faithful the rules are to the learned neural network. I mean, it seems possible that for some input the rules could produce a different output from the neural networks. There exist other works that also try to interpret neural networks consisting of affine layers with Relu activation that guarantee consistency (see "Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution" by Chu et al in KDD'18). The authors should at the very least compare and contrast with such works to highlight the pros and cons. Another undesirable property of ExplainNN seems to be that it relies on a dataset to derive its rules. Is it possible that when run with a different dataset and the same learned network, ExplaiNN would produce a different set of rules? Then how much faith do we place on ExplaiNN's output?

Writing wise, the paper is presented well enough. There's a few paragraphs in the Experiments section where the authors point repeatedly to the Appendices. In the best case this makes reading a chore. I would advise the authors to refrain from using the main body of the paper as a listing of contents and simply pointing to the appendices. The pictures in the experiments section were difficult to make out. I couldn't figure out from the image whether the husky's pointed snout had been identified as a defining feature. I would hope the authors find more compelling ways to make their point.
review_title: Proposes to extract rules from a neural net; Not the first time this has been proposed; Lacks comparison with prior art

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work makes a convincing case that we need to trace information flow (or at least, to find clusters of neurons working together) within a deep neural network in order to get the clearest picture of how the network is working.  The approach is quite simple but also quite novel in that it uses concepts from coding theory which are not widely known in the deep learning community.  The paper is well written and presents examples from deep CNNs (~20 layers) trained on ImageNet.

Quality:
The method is well thought-out and explained, and two different evaluations are used (MNIST and ImageNet).  It would be more impressive if the authors had included a problem from a different data modality since in principle the method is quite general.  The examples look very impressive, but my main concern is with whether the examples could have been cherry-picked, in the sense that most of the thousands of rules produced may not be useful.  Relatedly, I would like to know the reproducibility of the result.  If you train twice with different seeds, how similar are the results?  Or if you fit ExplainNN with different seeds on the same network?  And what is the danger of false positive findings?

Clarity:
The paper is very clear with regards to the problem setting, previous work, and the methods.  I am somewhat familiar with coding theory, having read much of MacKay's Information Theory, Inference, and Learning Algorithms, but I am by no means an expert in coding theory.  Still, I am confident that I understand the principles of the method.  However, I did not exactly follow how the authors carried out the tracing (pg 7 second paragraph.). Do you just apply ExplainN as usual and then filter for rules that (strictly/non-strictly) include Y?  This seems important to explain since the tracing, in my view, is the main contribution, given that there already exist tools for understanding the similarities of classes such as representational similarity (https://roberttlange.github.io/posts/2019/06/blog-post-3/).  

Originality:
I have not previously seen the idea of mining association rules for deep neural networks, although it is a simple enough idea and I would not be surprised if the idea has appeared before.  However, the application of MDL to solving the problem for binarized activations is likely to be novel.

Significance:
The work is promising, but to me it is not conclusive that it will make a lasting impact.  There are a number of important practical questions to be addressed that could make or break the method as a tool for the field, such as the reproducibility and usability of the method (whether most rules produced are meaningful or significant manual filtering is required).  On the other hand, even if the method does not meet practical needs, it could still be of great utility for NN methods researchers interested in investigating, say, the redundancy of specific architectures. the true degree of similarity between two trained models, convergent dynamics between alternative architectures, and the value of overparameterization.  The paper could be even more significant if the authors could comment on the generalizability to other architectures such as RNNs, GNNs or transformers.  The method itself is interesting enough and the examples sufficiently compelling (even if cherry-picked) that I would recommend the paper to almost anyone interested in neural network interpretability.

Note on rating:  In the face of limited details, I am willing to give the authors the benefit of the doubt that they did not cherry-pick overly aggressively and that the examples are representative of typical outputs. If it turns out that most rules do not look like the examples, then my rating would decrease.  On the other hand, if the authors can address my concerns about cherry-picking in the response, it is possible that I would raise my rating.

Pros:
 * important application
 * method is quite general
 * method is simple and intuitive

Cons:
 * evaluation of method performance limited to selected examples
 * reproducibility not addressed
 * control of false positives not addressed
 * method for tracking across layers not well-explained
review_title: Important complement to existing NN interpretation tools, solid implementation, some questions about usability and replicability

Meta-review: This paper proposes a method to explore neuron interactions within a neural network by deriving rules for the activations of units at different layers. The rules can presumably help interpret the inner workings of the neural network. 
The reviewers have very different opinions on the paper and the views did not converge.  However, there is a common concern on the lack of quantitative evaluation on the faithfulness of the rules to the models. I therefore do not recommend accept. 

R1[5]: On a related note, I felt the evaluation presented by authors while extensive is rather qualitative in nature.
R2[3]: Given that I could provide you with a couple of references that you admit is relevant, and this was just off the top of my head, would you care to comment on a quantitative comparison with the referenced approaches?
R3[8]: The examples look very impressive, but my main concern is with whether the examples could have been cherry-picked, in the sense that most of the thousands of rules produced may not be useful.



Decision: Reject