What's in the Box? Exploring the Inner Life of Neural Networks with Robust Rules
Authors: Jonas Fischer
Anna Oláh
Jilles Vreeken
Venue:  (2021)
Keywords: Neural Networks
CNN
explaining
interpretable
Rules
black box

Abstract: We propose a novel method for exploring how neurons within a neural network interact. In particular, we consider activation values of a network for given data, and propose to mine noise-robust rules of the form $X \rightarrow Y$ , where $X$ and $Y$ are sets of neurons in different layers. To ensure we obtain a small and non-redundant set of high quality rules, we formalize the problem in terms of the Minimum Description Length principle, by which we identify the best set of rules as the one that best compresses the activation data. To discover good rule sets, we propose the unsupervised ExplaiNN algorithm. Extensive evaluation shows that our rules give clear insight in how networks perceive the world: they identify shared, resp. class-specific traits, compositionality within the network, as well as locality in convolutional layers. Our rules are easily interpretable, but also super-charge prototyping as they identify which groups of neurons to consider in unison.

Under review as a conference paper at ICLR 2021
WHAT'S IN THE Box? EXPLORING THE INNER LIFE OF NEURAL NETWORKS WITH ROBUST RULES
Anonymous authors Paper under double-blind review
ABSTRACT
We p novel method for exploring how neurons within a neural network in- teract. In particular, we consider activation values of network for given data, and propose to mine noise-robust rules of the form X - Y. where X and Y are sets of neurons in different layers. To ensure we obtain a small and non-redundant set of high quality rules, we formalize the problem in terms of the Minimum Descrip- tion Length principle, by which we identify the best set of rules as the one that best compresses the activation data. To discover good rule sets, we propose the unsupervised EXPLAINN algorithm. Extensive evaluation shows that rules give clear insight in how networks perceive the world: they identify shared, resp. class-specific traits, compositionality within the network, well as locality in convolutional layers. Our rules are easily interpretable, but also super-charge pro- totyping as they identify which groups of neurons to consider in unison.
INTRODUCTION
Neural networks provide state of the art performance in many settings. How they perform their tasks, how they perceive the world, and especially, how the neurons within the network operate in concert, however, remains largely elusive. While there exists a plethora of methods for explaining neural networks, most focus on the mapping between input and output (e.g. model distillation) or only characterize individual neurons within the network (e.g. prototyping). In this paper, we introduce a new method that explains how the neurons a neural network interact. particular, we consider the activations of neurons in the network over given dataset, and propose to characterize these in terms of rules X -> Y, where X and Y are sets of neurons in different layers of the network. A rule hence represents that neurons Y are typically active when neurons X are. For robustness we explicitly allow for noise, and ensure that we discover succinct and non-redundant rules we formalize the problem in terms of the Minimum Description Length principle. To discover good rule sets, we propose the unsupervised EXPLAINN method. The rules that we discover give clear insight in how the network performs its task. As will see, they identify what the network deems similar and different between classes, show how information flows within the network, and show which convolutional filters it expects be active where. Our rules are easily interpretable, give insight in the differences between datasets, show the effects of fine-tuning, as well as super-charge prototyping they tell which neurons to consider in unison.
Explaining neural nets is of widespread interest, and especially important with the emergence of applications in healthcare and autonomous driving. In the interest of space we here shortly introduce the work most relevant to ours, while for more information we refer to surveys (Adadi & Berrada, 2018: Ras al., 2018: Xie al., 2020; Gilpin al.. 2018). There exist several proposals for investigating how networks arrive at a decision for given sample, with saliency mapping techniques for CNNs among the most prominent (Bach et al., 2015; Zhou et al., 2016; Sundararajan et al., 2017: Shrikumar et al., 2017). Although these provide insight on what parts of the image are used, they are inherently limited to single samples, and do not reveal structure across multiple samples or classes. For explaining the inner working of CNN, research mostly focuses on feature visualization techniques (Olah et al., 2017) that produce visual representations of the information captured by neurons (Mordvintsev et al., 2015; Gatys et al., 2015). Although these visualizations provide insight on how CNNs perceive the world (M. Øygard. 2016; Olah et al., 2018) it has been shown that concepts often encoded over multiple neurons, and that inspecting individual neurons does not

---

Under review as a conference paper at ICLR 2021
provide meaningful information about their role (Szegedy al., 2013; Bau al., 2017). How to find such groups of neurons and how the information is routed between layers remains unsolved.
An orthogonal approach is that of model distillation, where train easy to interpret white box models that mimic the decisions of a neural network (Ribeiro et al., 2016; Frosst & Hinton, 2017; Bastani et al., 2017; Tan al., 2018). Rules of the form (if-then) are easily interpretable, and hence a popular technique for model distillation (Taha & Ghosh, 1999; Lakkaraju et al., 2017). Although some consider individual neurons to determine what rules to return (Robnik- Sibbiia & Kononenko, 2008: Özbakır al., 2010; Barakat & Diederich, 2005) all these techniques only yield rules that directly map input to output, and hence do not provide insight into how information flows through the network. Tran & dAvila Garcez (2018) mine association rules from Deep Belief Networks. Their approach, however, suffers from the pattern explosion in typical in frequency based rule mining, and not applicable to state of the art networks. Chu et al. (2018) aim explain piecewise linear NNs utilizing polytope theory to derive decision features of the network. Providing strong guarantees, this approach however limited to PLNNs of extremely small size (< 20 neurons in hidden layers).
We instead propose to mine sets of rules to discover groups of neurons that act together across differ- ent layers in feed forward networks, and so reveal how information is composed and routed through the network to arrive at the output. To discover rules over neuron activations, we need an unsuper- vised approach. While a plethora of rule mining methods exists, either based on frequency (Agrawal & Srikant, 1994; Bayardo. 1998; Moerchen et al., 2011) statistical testing (Hämäläinen, 2012; Webb, 2010), these typically return millions of rules even for small datasets, thus undermining the goal of interpretability. We therefore take pattern set mining approach similar to GRAB (Fischer & Vreeken, 2019). where we are after that set of rules that maximizes a global criterion, rather than treating each rule independently. Providing succinet and accurate sets of rules, GRAB is however limited to conjunctive expressions. This traditional pattern language is too restrictive for our setting, as we are also after rules that explain shared patterns between clases, and are robust the inher- ently noisy activation data, which both require a more expressive pattern language of conjunctions, approximate conjunctions, and disjunctions. We hence present EXPLAINN, non- non-parametric and unsupervised method that discovers sets of rules that can model these types of rules.
2
THEORY
We first informally discuss how to discover association rules between neurons. We then formally in- troduce the concept of robust rules, and how to find them for arbitrary binary datasets, last, we show how to combine these ideas to reveal how neurons are orchestrated within feedforward networks.
2.1 PATTERNS OF NEURON CO-ACTIVATION
Similar to neurons in the brain, when they active, artificial neurons also send information along their outgoing edges. To understand flow of information through the network, it is hence essential to understand the activation patterns of neurons between layers. Our key idea is to recent advances in pattern mining to discover a succinct and non-redundant set of rules that together describe the activation patterns found for a given dataset. For two layers 4,, Ij. these rules X Y, X 4,, Y C lj express that the set of neurons Y are usually co-activated when neurons X are co-activated, That is, such rule provides local information about co-activations within, as well as the dependence of neurons between layers. Starting from the output layer, we discover rules between consecutive layers I,, I--1. Discovering overlapping rules between layers X + Y and Y + Z. C II,. 1,-1, 2 C Ij-2. allows us to trace how information flows through the entire network.
Before we can mine rules between two sets of neurons - e.g. layers - T; and I; of network, we have to obtain its binarized activations for given data set 'D = {dg (sk. ox)}- In particular, for each sample Sy and neuron set 4,, we take the tensor of activations di and binarize it to of. For networks with ReLU activations, which binarize naturally at threshold 0, we might lose some information about activation strength that is eventually used by subsequent layers, but it allows us to derive crisp symbolic, and directly interpretable statements how neurons interact. Furthermore, binarization reflects the natural on/off state of biological neurons, also captured by smooth step functions such as sigmoid or tanh used in artifical neural networks. We gather the binarized activations into dataset
2

---

Under review as a conference paper at ICLR 2021
Neural Network
Database D
neurons x in 4
neurons y in
Output
0
0
0
0
0
1
0
transactions
1
0
0
1
0
0
4
0
0
0
0
1
1
1
1
4
0
1
1
1
0
1
0
0
0
0
0
0
0
0
1
100100
0
0
0
1
1
1
0
1
0
0
Input
Rule Set M
M =
Xx1 - y1>2.
X4 - Ysy6H,
sample
X12g y:Y6]
Figure 1: Overview: For given network (left), binarized activations are gathered for the layers 1,, J for each sample, and summarized in the binary database D (right). Rules are discovered over D, where a good rule set M is given with the bottom right, with rules X Y. I, Y YEI,.
D where each row ty corresponds to the concatenation of of and of of I and 1n for sk. i.e., ty E D a binary vector of length 1y] + 4 See Fig. for toy example.
Next, given D, our goal is to find that set of rules that together succinctly describe the observed acti- vations. The Minimum Description Length (MDL) principle lends itself as an objective to find such sets. MDL is a statistically well-founded and computable approximation of Kolmogorov Complex- ity (Li & Vitányi, 1993). First introduced by Rissanen (1978), the essential idea that the model M E .M that best describes data D is the model that losslessly encodes D using the fewest bits M* = arg MEM T. (D, M). Here, our model class M the superset of possible rules over D, and by MDL identify the best model M the that compresses the data best. Tradition- ally, rule mining restricted to conjunctions over items, which is not sufficient for our application; neuron activations tend be noisy, labels are inherently mutually exclusive, and hence consider an extended language that allows for partial disjunctions of items (neurons, labels) and introduce a codelength function L(D, M) instantiate MDL for our model class of rule
2.2 MDL FOR ROBUST RULES
Our goal is to find set of rules M that, in terms of description length L(D, M), best describes a binary database D = (t C I} that consists transactions t that are subsets of items I. Each rule is of the form X - X,Y I, and indicates that Y is strongly associated with, i.c. occurs mostly in transactions where X is present. We say a rule X + Y applies to a transaction I iff X C and say rule holds for additionally Y C We indicate these transactions Tx = (i t E D. X C t.). respectively Tyxx = (i t E Tx. Y c t.). Based on these definitions of rule transaction sets, we can now formally introduce our codelength function L(D,M).
as
Baseline model Our base model Mind (0 I VI E 7) consists of singleton rules only, i.e. it models that all items I are generated independently. To send the transac- tions of D using Minds we simply send for each item in which out all transactions in the database it appears. We can do so optimally using log binomial code, which is given by log (1/7)) = log (177))- To unambiguously decode, the recipient needs know each [Ty], which we can optimally encode via the parametric complexities of the binomials, which are defined as (m=#) (=)))) and can be computed in linear time (Kontkanen & Myllymäki, 2007). We thus have L(D, Mind) = Σier (log (17)) + Lpc(n)). Mind serves as our baseline model, and its singleton rules are required part of any more complex model as they ensure we can always send any data over I.
n! Lp (n) log Σ> [n-RjiRT
Non-trivial models A non-trivial model M contains rules of the form X + Y. X.Y C I that are not part of Mind- The idea is that we first transmit the data for where these non- con-trivial rules hold, and then send the remaining data using Mind. To determine where such rule applies, the receiver
3

---

Under review as a conference paper at ICLR 2021
needs to know where X holds, and hence the data over X needs to be transmitted first. To ensure that we can decode the data, we only consider models M for which the directed graph G = (I. E) is acyclic, in which there exists edge between two items é1 ig iff they occur in the head and tail of a rule, that is E = {(11,22) 12) XE - Y E M. in E X A 12 E YJ. We thus get a codelength
I(D| MU Mina)
Σ
log
((FFX))
Tx
)
Σ
log
\TT)
X-YEM
0-IEMMnd
= 1 Tylx)) is modified transaction set containing all transactions with item / not covered by any non-trivial rule.
where Ti = {1 E D (V
In addition to the parametric complexities of the binomial codes, the model cost of a non-trivial model also includes the cost of transmitting the non-trivial rules. To transmit rule X - Y. we first send the cardinalities of X resp. Y using the universal code for integers LN (Rissanen, 1983).
For 2 1, this is defined as Lx (z) summing only over the positive components (Rissanen, 1983). To satisfy the Kraft equality we set co 2.865064. Knowing the cardinalities,
log" + log co with log (=) (z)
log = + log log = + ....
then send the items of X resp. Y one by [Tal
one using an optimal prefix code given by L(X) EzEx log Eiez 17y1' For a particular rule x + E M, the model costs for a rule, respectively the full model thus amount to L(X Y) = Lx( + (x(() + L(X)
L(Y) pa (Txl),
L(M U Mind) = | x Tppc (n) + Ls([ +
E 1(X Y). X-YEM
We provide an example in App. A.1. With these definitions, we have an MDL score that identifies the best rule set M for data D as
M
arg min (L(M U Mind) + L(D| |MU U.na))))
MEM
where M is the space of all possible rule sets over the items in D.
Robust Rules In real world applications, which are inherently noisy, we need a score that is robust and can approximate where rules hold and apply. In addition, as output neurons are only active mutually exclusively, rules need to be able to model disjunctions. The key problem with noisy data is that a single missing item in a transaction can cause a whole rule not to hold or apply. To discover rules that generalize well, we need to explicitly account for noise. The idea is to let rules apply, and hold, approximately. Specifying how many items I, and k. out of all items in the rule head, relax the original rule definition to account
respectively tail, need to be part of transaction, for random noise (see App. Figs. 7, 8). Setting / = and & = 1 coincidentally corresponds to a disjunction of items in the head and tail of the rule X - Y, thus allowing to model output neurons correctly, and 1 = |X| and k = Y correspond to the original stringent rule definition. Varying between the two extremes accounts for varying levels of noise. The optimal and kt are those that minimize the MDL score. To ensure a lossless encoding, we need to make sure that the receiver can reconstruct the original data. Thus, for the relaxed head definition, for each rule we send the number of items I for the rule to apply using (t) bits. Knowing each 1, the receiver can reconstruct where each rule applies. Sending where a rule holds now leaves the receiver with an approximation of the data. To be able to reconstruct the actual data, Fischer & Vreeken (2019) introduced error matrices that when XORed with the approximation yield the original data. These two matrices X Xx-,: and Xx_y correct for the errors made in the part where the rule applics and holds, respectively applies but does not hold (for example see App. Fig. 7). These error matrices part of the model M and have to be transmitted with an adapted L(D,M), M), which we review in App. A.2.
Complexity of the search To discover rules over the activations of two layers 7,, I,. we have to ex- plore all rules formed by subsets of neurons in 1, for the head, combined with any subset of neurons of I; for the tail. There exist 2 244 such rules, and hence 2 (4ggg distinct models would need to be explored. Fischer & Vreeken (2019) showed that the rule set search space does not lend itself to efficient search as it is neither monotone nor submodular, the counterexamples also holding for our model definition. In fact, for robust rules, we additionally have to consider where rules should apply respectively hold optimizing k and 1 which results in approximately 2 (224++))) models (detailed derivation in App. A.4). Exhaustive search is therewith infeasible, which is why present EXPLAINN, heuristic algorithm to efficiently discover good sets of rules.
4

---

Under review as a conference paper at ICLR 2021
2.3 DISCOVERING GOOD RULE SETS WITH EXPLAINN
EXPLAINN is based on the idea of iteratively refining the current model by merging and refining already selected rules. The key insight of the algorithm is that for a rule X + Y to summarize the data well, also rules X + Y' with only part of the tail, Y' C Y, should summarize well, as all tail items should be similarly co-occuring with head X. Starting from the baseline model Mind we iteratively and greedily search for better models until we can no longer improve the MDL score. As search steps we consider either introducing new rule to M. by taking good set of items X C. 4, for the head and a single item A E 1 for the tail and refine the model to M' = M (X - A), seeing if it decreases the overall MDL costs (Bq. 2.2). Or, we merge two existing rules 111 = X + Y1 E M and r2 X + Y2 E M, to form a new rule r' = X - Y, U Y2 and refine the model to (M \ (2)r)2) U (r'). For a rule r', the refinement operator is adding the rule r = x + Y to M, removing the merged rules that led to r', if any, and updating the singleton transaction lists Ta for all items A E Y, removing all transactions where r' holds.
M' M
+ (r])
To make this search scheme applicable to the scale of a typical neural net, we have to discuss how to efficiently search for candidate rules, with heads that can express anything from conjunctions to disjunctions. Immediately after, we present the full algorithm EXPLAINN for mining high quality rule sets for two arbitrary sets of neurons (e.g. layers) of network.
Searching for candidates A key component of EXPLAINN is the candidate generation process, which implements the two possible steps of generating new and merging existing rules. Given two layers [,. Lj. to efficiently discover rules that are both robust to noise, and may include disjunctively active neurons in the head, we can not enumerate all possible rule heads for each individual tail neuron, as this would result in |T x 2 many rules. Instead, we keep a list Hy for each item y I,, storing all head neurons x 1; for which y is frequently active when x is active, that is IT, > 0, where 0 is confidence threshold. We consider a rule X + Y to be good, if Jzy To
when neurons X are active, the neurons Y are also likely to be active, which is directly represented by the confidence 0. With parameter j we account for early decisions on rule merging that later hinder us to see more general trend. The lists are sorted decreasing on We search in each Hy for the rule with highest gain over all unions of first t = 1... |Hyl neurons in the list. We add that rule X -> y with highest gain the candidate list (see App. Alg. 1). To compute the gain, we consider all possible values A = 1 1... [X] to determine for which transactions T = {t E D [ 2 k} the rule should robustly apply, where k corresponds to disjunction and
|X] conjunction.
For an individual neuron such a rule would be optimal, but, our goal is to discover groups of neurons that act in concert. To this end we hence iteratively merge rules with similar heads- similar, rather than same, as this gives robustness both against noise in the data, as well as earlier merging decisions of the algorithm. For two rules X1 + Y1, X2 + Y2 with symmetric difference e X2 (X, X2) U (X, X1), we consider possible candidate rules X1 U X2 - Y1 Y22 and X,nX, - Y1 U Y2, iff [X] e X2l S j. for some threshold J E N. For example, μ = corresponds to the case that one head has one label more than the other, all other labels are the same
Both parameters 9 and μ are simple, yet effective runtime optimizations, the best results with respect to MDL will always be obtained with the largest search space, i.e. with 0 and set to 0, respectively [X1] + [X2). Besides impacting run-time, many of those rules may be uninteresting from a user- perspective, μ and 0 allow to directly instruct EXPLAINN to ignore such rules.
EXPLAINN Assembling the above pieces, we arrive at EXPLAINN, which given two sets of neu- rons Tt, I, and database of activations of these neurons, yields a heuristic approximation to the MDL optimal model M* By first introducing all relevant single neuron rules, it then proceeds by iteratively merging existing rules using the approach described above, until it can achieve no more gain. For efficiency, we separate the generation of the new rules from the merging of existing rules. In practice, this does not harm performance, as we allow merging of similar heads and can thus revert too greedy decisions introduced earlier. Furthermore, by observing that independent rules X, + Y, X2 > Y2, Y1 UY2 = do not influence each others impact on codelength, we can add all independent rules with the highest respective gain once. We give pseudocode as App. Alg. 3.
5

---

Under review as a conference paper at ICLR 2021
precision recall
precision reccll
0.25
0.00
10
50
100
200
300
500
0.1
0.5
5
10
# rules
% noise
1,00-
Accuricy
0.75-
0.50
0.00
Classes
Figure 2: Evaluation of rule quality. Top: Performance of EXPLAINN as precision and recall on data with varying number of planted rules with mutual exclusive head items (left) and co-occurring head items with varying noise (right). 10% noise corresponds to more noise than signal in the data. We provide the average (bar) and distribution (boxplot) across 10 repetitions. Bottom: Accuracy per class of VGG-S before (yellow) and after (blue) intervention weights connecting neurons to class given by a rule, and 90% quantile of accuracies obtained for randomized intervention (red).
The generation of new rules results in time O(n x |7 x 17,19), by iterating over each neuron in J,. and considering each subset of the most overlapping neurons in 7, and considering each threshold k = 1.... | for when the rule should apply, and the factor n from intersecting transaction lists 7 compute the gain. We can have most [4 generated rules before considering rule merges, and in every iteration of merging we combine two rules, reducing the rule set size by 1. In each such step, we consider | merges, for each of which we compute the gain considering noisy head and tail. We thus have worst case runtime of O(n x |T11² [T.). As MDL. ensures we consider succinct models that capture only the relevant structure in the data, EXPLAINN in practice much faster and easily scales to several thousands of neurons.
Complexity of EXPLAINN
3 EXPERIMENTS
In this section we empirically evaluate EXPLAINN on synthetic data with known ground truth and real world data to explore how CNNs perceive the world. We focus on CNNs as they count towards the most widespread use of feedforward networks and naturally lend themselves for visualization, which helps us to interpret the discovered rules. We consider CNNs of different complexity, includ- ing GettLeNet and VGG-S. We compare to traditional prototyping and activation map approaches on MNIST (LeCun & Cortes, 2010), and examine which information is used how arrive at clas- sification for ImageNet (Russakovsky et al., 2014). Finally, we investigate the effect of fine-tuning in transfer learning on the Oxford Flower data (Nilsback & Zisserman, 2008). We will make the implementation of EXPLAINN publicly available upon publication. For the below experiments, running on commodity hardware EXPLAINN took minutes for MNIST and Flower, and 6 hours for ImageNet, yielding rule sets of hundreds up 3000 rules, depending on data and choice of layers.
3.1 RECOVERING GROUND TRUTH
To evaluate how well EXPLAINN is able to recover the ground truth from data, we first gen- erated synthetic binary data sets of size n = 10000 samples initially filled with zeros with m = {10, 50, 100, 200, 300, 500} planted rules with up to items in head and tail, respectively. For each rule, the frequency is drawn from U.02, .08), the confidence is drawn uniformly from U(5,, 1). We introduce noise by flipping 0.1% of the entries chosen uniformly at random. Addi- tionally, 5 noise features with the same frequency as the rule features are introduced. Fischer & Vreeken (2019) showed that such an MDL model works for conjunctive rules, hence we will focus on investigating performance for mutually exclusive rule heads and noise. In the first set of exper- iments, we set head items to in a mutual exclusive fashion, in line of finding rules over the NN output labels. EXPLAINN achieves high recall and precision (see Figure 2) in terms of retrieving

---

Under review as a conference paper at ICLR 2021
0
6
Figure 3: MNIST Average activation of neurons for the classes in MNIST for filter 2 in the first convolutional layer. Overlayed are the EXPLAINN rules, where pixel groups of the same colour (e.g. purple pixels top left for classes 2. belong single rule spanning multiple classes.
exact ground truth rules, and does not retrieve any redundant rules. Next, we investigate the impact of noise the performance, generating data of n 10000 samples and m 100 rules in simi- lar setting as above, with head items now initially set co-occuring (conjunctive), but adding varying levels of noise with a (0.1% 1%, 0.5%, 1%, 3%, 5%, 10%} of bitflips in the matrix, where 10% noise means more noise than actual signal. We report the results in Figure 2. As expected by its design, EXPLAINN is fairly robust noise, even when facing almost the same amount of noise and signal.
3.2 TAKING A GLANCE INTO THE REAL WORLD
How is information filtered As first simple example consider MNIST. We train simple CNN of 2 convolutional and one fully connected layer using Keras, achieving 99% classification accuracy on test data (see App. B.1 for details). We are interested in what the individual filters learn about the digits, and how EXPLAINN reveals shared features across several classes. We compare to average activation maps and single neuron prototypes. Examining different filters, we observe that average class activation maps do not reveal the purpose of filter, whereas rules that EXPLAINN discovers identify which pixels together trigger filter. As an example, consider Fig. 3, in which for filter 2 in layer I we visualize the discovered rules on top of the average activation maps (which just represents the original numbers). Whereas the rules identify shared structure (indicated by pixels of the same colour, in the same location, for different classes) such as top-left curve for 0, 4, and (depicted in orange), the prototype (see App. Fig. 9) looks like a maze and does not reveal any insight. Similarly, for filter 36 in layer 2, directly from the discovered rules that this filter detects horizontal edges in a class specific manner, whereas prototyping and activation maps again fail to reveal this information (see App. Fig. 10). Interestingly, we also find filters that learn negative, with activated areas corresponding to background of the digit (see App. Fig. 11).
How does information flow Next, we investigate more complex setting and examine the activa- tions for the ImageNer data set of pretrained VGG-S and GoogLeNet architectures (Chatfield et al., 2014; Szegedy et al., 2015). We here foeus VGG-S, for this network there exist optimized prototyping methods that yield highly interpretable images for sets of neurons (M. Øygard, 2016). We provide the results for GoogLeNet in App. B.2.1. Starting with rules from the labels to the last layer, we observe that EXPLAINN yields complex rules spanning multiple labels and multiple neu- rons in the last fully connected layer (FC7). which together encode the information shared between labels. Examples include the faces of certain dog breeds, as well as the coloured beaks of birds (see App. B.2.2). Notably, if we visualize these neurons individually, meaningful from the images: the information is really encoded in the set of neurons that act together (see App. Fig. 14). We also observe cases where rules describe how the network discriminates be- tween very similar classes. We give example in Fig. 4 for the neurons EXPLAINN discovers to be associated with huskies, with malamutes, and for both classes together. These dog breeds are visually very similar, sharing black- white fur pattern, as well as head and ear shapes. These traits are reflected by the neurons corresponding to the rule for both breeds. Looking closer, we can see that distinct traits, the more defined snout and ears of the husky, respectively the fluffy fur of the
hard to extract anything
7

---

Under review as a conference paper at ICLR 2021
Siberian hisky AND NOT Malamute
Siberian husky AND Malamute
Malamute AND NOT Siberian husky
(a) Top:
Malamute, Bottom: Siberian Husky
(b) Rule prototypes for FC7
Neurons discriminating Huskies and Malamutes. a) Huskies and Malamutes are very similar looking dog breeds. b) Prototypes for rules X + Y discovered for classes X, Siberian husky (red frame), class Ma Malamute (yellow frame), resp. both (orange frame) and neurons Y in FC7. The neurons associated with both classes represent typical features shared between the two classes, those associated only with Siberian an huskies show their slightly sharper, more defined head, while those associated only with Mal amutes capture their more fluffy fur.
Figure 4:
malamute, are picked up by the neurons discovered for the individual dog breeds. Beside discover- ing what shared and distinct traits the network has learned for classes, we also find out when it learns differences across samples of the same class. For the dog breed Great Danes, we discover three rules that upon visualization each correspond to visually very different sub-breeds (see App. Fig. 15).
Next investigate the information flow within the network. Starting with the EXPLAINN rules X + Y from output layer to last fully connected layer FC7, we apply EXPLAINN to discover rules Y + Z between FC7 and FC6, where restrict the heads Y to be groups of neurons found as tails in the previous run. We recursively apply this process until we covered a convolutional layer. This gives traces of neuronal activity by chaining rules X Y - Z discovered in the iterative runs. We visualize one such trace in Fig. 5, which gives insight in how the network perceives totem poles. We find that the relevant neurons of FC7 and FC6 yield prototypes that show typical totem pole heads with animalistic faces. The neuron sets found for the different filters of the last convolutional layer CONV5, together detect specific parts the class, including the vertical pole, and the rooftop- like structure, decorated with animalistic shapes with eyes, that is typically found at the top of totem. We provide another example of an information trace in App. Fig. 16, which shows how the network learns to detect features located typically at certain parts of the image due to the very prevalent side-view photos taken from dogs, a clear sign of overfitting by the neural network.
Rules carry class information To determine the quality of a rule set found across classes, we next investigate how the classifier performance changes when we intervene on the neurons in the last fully connected layer that were found associated to a class. For each class c, we set incoming weights from neurons y to for which we have discovered a rule X Y,C X, // Y, compar- ing classification rate before and after intervention. To further evaluate the practical impact of these rules, we perform 100 runs intervening on an equally sized random subset all weights leading to class c, again measuring classification rate after intervention. We report the results in Fig. 2 (bot- tom). For all classes, the performance dropped tremendously, most cases even to O. This shows observe
that the discovered rules capture necessary information for classification. Furthermore, that in most cases very related classes are instead predicted, for example Fire Salamander to Spotted Sa l amanndrr, Ba rbell to D Dumbbell 1, Pal acce to Monastery, which provides interesting opportunities for adversarial attacks.
The effect of fine tuning Finally, we show that EXPLAINN can elucidate the effect of fine-tuning in transfer learning. For this we consider Oxford Flower data Nilsback & Zisserman (2008), which consists of 8k images of flowers of 102 classes. For investigation, we consider both the vanilla VGG- network trained on ImageNet from above, and a fine-tuned version from the Caffee model zoo!. We run EXPLAINN to obtain rules between the output and the final layer of both networks, and give
8

---

Under review as a conference paper at ICLR 2021
Labels
FC7
FC6
Toteem pole
CONVS
Figure 5: Information flow for Totem poles. Example from rule cascades found for ImageNer starting at label Totem pole. For each rule X - Y, the group of neurons of the tail Y are used to generate a prototype. To discover rule cascades, we start mining rules between output and FC7 layer. Rule tails, which are neurons of FC7, are then used as heads for rules between FC7 and FC6. Similarly, rule tails are used as heads for rules between FC6 and CONV5.
an example in App. Fig. 17. The visualizations show, expected, strong emphasize colour and shape of the corresponding flower class. Interestingly, the visualizations of the same neurons for original VGG-S show almost identical shapes and pattern, but with less intense colour. Furthermore, we observe some prototypes with animal-like features such as eyes or beaks.
DISCUSSION AND CONCLUSION
The experiments show that EXPLAINN is uniquely able to discover distinct groups of neurons that together capture traits shared and distinct between classes, within-class heterogeneity, and how fil- ters are used to detect shared features, segment background, are revealed by activation maps, which miss the local information that patterns provide, nor by great
detect edges locally. None of these
saliency maps, which investigate network attention for individual image. Prototyping tool for visualizing neuron information content, but, by itself is limited by the massive number of possible combinations of neurons, requiring thousands of hours painstakingly handpick and nect the information of just individual neurons (Olah et al., 2020). Combining EXPLAINN with prototyping permits exploring networks beyond single neurons, by automatically discovering which neurons act in concert, which information they encode, and how information flows through the NN.
In particular, we discover distinet groups of neurons in fully connected layers that capture shared respectively distinct traits across classes, which helps in understanding how the network learns gen- erality but still can discriminate between classes. Due to the local information that our rules provide, can also detect differences in the perception across samples of single class, where for example different groups of neurons describe visually different sub-breeds of a class of dogs. By connecting rules that we find across several layers, we can trace how information is gathered and combined to arrive at a classification, from filters that detect typical class specific features in the image, through fully connected layers where multiple neurons together encode the combined information, up the final classification output. Applying EXPLAINN to investigate the impact of fine-tuning in transfer learning, we found that for the groups of neurons in the given fine-tuned CNN, surprisingly, the contained information is almost identical to the original CNN, but capturing the traits of the new classes almost perfectly. For the given transfer task the fine-tuning thus mostly resulted in routing information differently. rather than having to learn to detect new features.
Overall, EXPLAINN performs well and convineingly answers 'what's the box?'. finds surpris- ing results that help to understand how CNNs perceive the world. While many important tasks are solved by such networks, recurrent neural networks play an important role e.g. language pro- cessing. Although rules can likely also help to understand what these models learn, these networks encode an entirely different type of information that is inherently hard understand and visualize, and hence an exciting challenge for future work. Here, our main interest was characterizing infor- mation flow through neural networks, and hence, we focused on subsequent layers. EXPLAINN, however, operates
arbitrary sets of neurons, and hence also naturally allows investigating e.g. residual networks, where the previous two layers contribute information. Currently scaling to thou- sands of neurons, it will make for engaging future work to scale entire networks once.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access, 6:52138-5211 -213366--600, 2018.
Rakesh Agrawal and Ramakrishnan Srikant. Fast algorithms for mining association rules. In Pro- ceedings of the 20th International Conference Very Large Data Bases (VLDB), Santiago de Chile, Chile, pp. 487-499, 1994.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Blaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130100, 2015.
Nahla Barakat and Joachim Diederich. Eclectic rule-extraction from support vector machines. In- ternational Journal of Computational Intelligence, 2(1):59-22, 62, 2005.
Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpreting blackbox models via model extrac- tion. CoRR, abs/ 705.08504, 2017.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. P Proceedings the IEEE conference on computer vision and pattern recognition, pp. 6541-6548 2017.
Roberto Bayardo. Efficiently mining long patterns from databases. In Proceedings of the ACM International Conference Management of Data (SIGMOD), Seattle, WA, pp. 85-93, 1998.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. CoRR, abs/ 1405.333,,, 2014. URL http: / //aaxiv..rgg/aaaaaaeee/ 1405. 3531.
Lingyang Chu, Xia Hu, Juhua Hu, Lanjun Wang, and Jian Pei. Exact and consistent interpretation for piecewise linear neural networks: A closed form solution. CoRR, abs/1802.06259, 2018. URL http:: / rxii.. org/ abs / 8028802.00 00659..
Jonas Fischer and Jilles Vreeken. Sets of robust rules, and how to find them. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML. PKDD), Würzzurg, Germany. Springer, 2019.
Nicholas Frosst and Geoffrey Hinton. Distilling neural network into soft decision tree. arXiv preprint arXiv:1711.0977 2017.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508. 057.0::600800676 2015.
Leilani Н Gilpin, David Bau, Ben Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018.
Wilhelminna Hämäläinen. Kingfisher: an efficient algorithm for searching for both positive and negative dependency rules with statistical significance measures. Knowledge and information systems, 32(2):383 414, 2012.
Petri Kontkanen and Petri Myllymäki. linear-time algorithm for computing the multinomial stochastic complexity. Information Processing Letters, 103(6)2227-233 2007. ISSN 0020-0190.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovee, Interpretable explorable approximations of black box models, arXiv preprint 170.v:1707.01154 2017.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL URLhttp:///aan:
lecun. wom hoom/exdb/mtt hwtp///bbbb//////////m////
Ming Li and Paul Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. Peringer-Verlag, Berlin, Heidelberg, 1993. ISBN 0387940537.
10

---

https//dssillllpp/222000cccciiio Under review as a conference paper at ICLR 2021
Audun M. Oygard. Peeking inside convnets. peeking-insides convnets//ee/oocoocooooo....o..n June 2016.
Fabian Moerchen, Michael Thies, and Alfred Ultsch. Efficient mining of all margin-closed itemsets with applications temporal knowledge discovery and classification by compression. Knowledge and Information Systems, 29(1):55--0) 2011.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. 2015.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics Image Processing, pp. 722-729. IEFE, 2008
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: htps://distilll.ub/200 htt.///////feattuutissuuaaaiioii
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.23915/
distill.00010. https://distilllpubb/b 8/building
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction circuits. Distill, 2020. doi: 1033911//iiiil.00000
Lale Özbakır, Adil Baykasoğlu, and Sinem Kulluk. A soft computing-based approach for integrated training and rule extraction from artificial neural networks: Difaconn-miner. Applied Soft Com- puting. 10(1 (M1))304-311 2010.
Gabriëlle Ras, Marcel van Gerven, and Pim Haselager. Explanation methods in deep learning: Users, values, concerns and challenges. In Explainable and Interpretable Models in Computer Vision and Machine Learning, pp. 19-36. Springer, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-11444 1144, 2016.
Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465-771, 1978.
Jorma Rissanen. A universal prior for integers and estimation by minimum description length. The Annals of Statistics, 11(2):416-433 1983.
Marko Robnik- Shnnik-Sikojja and Igor Kononenko. Explaining classifications for individual instances. IEEE Transactions on Knowledge and Data Engineering, 2065);589-600, 2008
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei- Fei Li. Imagenet large scale visual recognition challenge. CoRR, abs/1409.0575, 2014. URL http://arxivv wtwp/iiirr////o org/or/g////.oo abs
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. arXiv preprint 1rXiv:1704.0268, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arXiv preprint arXiv: arXiv:1703.01365; 2017.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.619, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.

---

Under review as a conference paper at ICLR 2021
Ismail A Taha and Joydeep Ghosh. Symbolic interpretation of artificial neural networks. 1EEE Transactions on knowledge and data engineering. 11(3):448 463, 1999.
Sarah Tan, Rich Caruana, Giles Hooker, Paul Koch, and Albert Gordo. Learning global additive explanations for neural nets using model distillation. arXiv preprint arXiv:1801.08640 2018.
Son N. Tran and Artur dAvila Garcez. Deep logic networks: Inserting and extracting knowledge from deep belief networks. JEEE Transactions Neural Networks and Learning Systems, 29(2): 246 258, 2018. doi: 10.109/INNLSS2016266533
Geoffrey Webb. Self-sufficient itemsets: An approach to screening potentially interesting associ- ations between items. ACM Transactions on Knowledge Discovery from Data, 4(1):1-20, 2010.
Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran. Explainable deep learning: field guide for the uninitiated. arXiv preprint arXiv:2004. /4545, 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
A MDL FOR ROBUST RULES
In this section we will give extended examples on how to compute the MDL score for given database and set of rules, elaborate the eπrоr encoding for the rule tails, and give visual toy example on the impact of the extended pattern language for the rule head.
A.1 COMPUTING MDL FOR RULES
A
C D
D
Rind = (0 + A, B,
Ø + C,0 - D)
R = Rind U - BC3
Database
[D| 100
T TBCLA
[Tal = 40
TA
[Tgl 35
ITcl 33
[Tpl 35
[TeCiAl 30
Figure 6: Example database and model. A toy database D with blocks indicating where the items A. B, C, D occur in D. margins and relevant joint counts are given on the right. sensible rule set Mnd = A + BC U Mid is given on the right, the part of the database where the rule applies and holds indicated by a light respectively dark orange area.
For the given example in Fig. 6, will now compute the codelength L(D, L(M) L(D M) of transmitting the whole database D using M U M Here, we will stick with the simple encoding without matrices, make the process of computation more understandable. For
12

---

Under review as a conference paper at ICLR 2021
reference, we first compute the baseline model, which is given by
D
((D,Min)) = [Z] x DP) +
Σ
log
(Tyl)
TET
(100)
100
1100
4 x ppc (100) + log
+ 2 log
+ log
40
35
33
2 14.88 + 179.4 87 .93 = 375.92.
179.64
Thus, sending the data with just the baseline model costs 375.92 bits. Now, we will compute L(D, M U Mind), we will start with the costs of sending the data L(D M U Mind)
)
[Tx|
[12] (TH))
L(D | M U Mind)
Σ
log
+
Σ log
X-YEM
JEZ
(0
(100>
100
(100)
(100)
log
+ log
+ log
+
log
+ log
30
40
5
3
35
2 29.66 + 93.47 + 26.17 + 17.30 + 89.82 256.42.
The model costs composed of the parametric complexities for the (adapted) baseline rules, plus the costs of transmitting what the rule composed along with its parametric complexity. We thus get
L(M Mind) = [I] x Lpp( +
Σ
Lx([X)
) Ln (N + L(X) + L(Y) + Lpe
X X-YEM
40
35
33
4 (100) (1) + Lx(2) log
log
log
Li (40)
143
143
143
14.88 1.52 + 2.52 1.84 + 2.03 + 2.12
3.11 28.02.
Hence, the model with the complex rule has a smaller codelengh than the baseline, with L(D, M U Mind) 284.4 44 bits.
A.2 THE ERROR ENCODING FOR TAILS
For the error encoding for tails, which allow to discover rules in noisy settings (compare Fig. 7a, send where rule X Y approximately holds according some parameter k. which defines the number of items of the tail that have to be present in the transaction. The errors made by this ap- proximation are then accounted for by sending error correcting matrices X and Xx-y. which account for the destructive, respectively additive noise in the are where the rule applies (compare Fig. 7c).
Let us first assume we are given h. we will later show how we can optimize for k. We redefine (X C t) (JY nt) 2 k)), which corresponds to the
the transaction sets Tyix = (t transactions where the rule approximately holds. We will now slightly abuse notation and indicate the binary input matrix that correspond D by D. and we subset this matrix using the transaction id lists and item subsets. Both of these sets of indices that indicate which rows, respectively columns to use of the matrix. For example, the submatrix where X holds is given by D We can now define the error correcting matrices to be Xx = D(Trix. Y X [|Tyix [x]|] and Xx = D[Tx \ Tyxx, Y), where X is the element-wise XOR operator and 1'xj is matrix of size i x j filled with ones. The receiver, knowing Tx and Tyjx. can then reconstruct the original data D[ Y] [Xy]l|l| X X respectively D(Tx \ (T Y|= x
While this explains the concept of how original input, which hence define a lossless encoding, we are mainly interested in the codelength functions. To adapt the data costs, we now additionally send the two error matrices, which we can do using binomial codes. Hence, we get
correcting matrices
be used reconstruct the
L(D M)
Σ log
Txl ) + (EYixI)
Elog
[
X-EEM
Fet
[Tyixl x x|8l) (
[Tx
Trixl x (8]) |
+ +og
+ log
|
13

---

Under review as a conference paper at ICLR 2021
Rule A
BCDE
b
a
A
C
D
k
3
2
7
TECDE l4
Database
TA
C
B D
B D
B BCDE
TBCDE 14
XOR
XOR
Ta
Figure 7: Example of tail error encoding. For given database D given in a, where blocks indicate the occurrence of items, good rule is given by A + BCDE. The part of the database where the rule applies indicated by the orange area. In b show the part of the transaction were the rule holds for varying number of tail items that have to be present in a transaction, from all items on the left corresponding to a conjunction towards just a single item on the right, which corresponds to disjunction. In we visualize the eтrоr encoding used to transmit the data for & = 3. We first transmit the data where the rule holds, resulting in area that indicated by the gray block. XORing the eтrоr matrix X with this block, it is possible to reconstruct the original data for the part where the rule holds. Using X+, we reconstruct the original data in the area where the rule applies but does not hold.
with the second line providing the codelength of the error matrices, and | indicating the number of ones in X.
Our model M now not only consists of rules M Mind but also of the set of error correcting matrices. As the submatrix to which we need to apply the matrix is fully defined by Tx Typx and Y of the corresponding rule, also defining its size, the only adaptation need for the model costs the parametric complexities induced by the codes for transmitting the data. This yields
L(M) = [z] x Lp)( + Σ +) X-YEM
x (Y)) + Lpo AY X Y().
This completes the MDL costs for rules robust noise in the tail for given k. To optimize k, the crucial insight that the codelength of individual complex rules are independent, as is the data cost. That means optimize for each rule separately. Thus, for given rule X + Y we enumerate all Y many models for the different thresholds and let MDL decide which one fits the data best.
14

---

Under review as a conference paper at ICLR 2021
Rule ABC - D
b
a
k= 3
2
1
B
C
D
V
Database D
Figure 8: Example of the impact of noise. For a given database D given in where blocks indicate the occurrence items, a good rule given ABC > D. Due to high noise, the simple conjunc- where the rule should apply, visualized on
tive pattern language results in bad representation the left of b. More relaxed definitions towards disjunctions, where we only require items of the head to be present in the transaction, result in much more stable representation where the rule applics.
A.3 THE IMPACT OF THE EXTENDED PATTERN LANGUAGE
Extending the pattern language for rule heads is crucial to be applicable for tracing activation pat- terns through neural network. First of all, need start from labels, which are inherently activated mutually exclusive, as we only have a single label as classification. To find shared features of labels, it is essential be able to express disjunctions with rule heads. Furthermore, the data as well as activation patterns across the data are very noisy. Thus, determining where rule applies just based on conjunctions of features can give a very twisted look of the data at hand, as visualized in Fig. 8. That is the reason to introduce more flexible language similar to approximate rule tails, which solves these issues.
A.4 SEARCH COMPLEXITY
The size of the search space implied by our model class M is in O(2 41xl4)xx For two layers I,, Lj. we enumerate all possible rules by
12
41 K
14
1=0
Possibilities for head
Possibilities for tail
(
x 17,
k
k=O
| 244
21,12 |4,1 |4] 241++151
where the first sum enumerates all heads of size A, the binomial coefficient describes the ways of drawing heads of such size, and the term is the number of models given by the robust head encoding. Similarly, the second sum enumerates all tails of size the binomial coefficient describes the drawing of such tails, and the term / is the number of ways place the error correcting matrices
15

---

Under review as a conference paper at ICLR 2021
for the robust tail encoding. As in theory can have any subset of these rules model, thus get approximately 2011/x many different models.
A.5 ALGORITHM PSEUDOCODE
EXPLAINN explores the search space of rule sets in an iterative fashion, either generating new rules with a single item in the tail, merging two existing rules, thus generating more complex rules with multiple items in the tail. Using these two steps, we can generate all potential candidate rules to add to the model, and evaluate their respective gain in terms of MDL. For rule r', we will say model M' M @ r' is the refined model, with the refinement operator @ adding the rule r' = + Y to M, removing the merged rules that led to r', if any, and updating the singleton transaction lists TA for all items in the tail A E Y. Here, we will provide the pseudocode for the two candidate generation functions for rules and for merging rules in the general setting alongside the complete algorithm EXPLAINN.
For generating new rule with head using the extended pattern language described in the main paper, gathering all confidence values for a given neuron A in 17 for all potential head neurons I,. We keep all potential head neurons with confidence value beyond 0 in list HA sorted descending on confidence and merge the first / neurons in the list to form the head. Going over all 1.. HAl allows to greedily optimize for the best of all relevant heads for the given item..
the approach
Algorithm 1: GENCANDIDATENEEW
Input: Dataset D over layers I,, Ij, Model M, tail item A, threshold 0 Output: Best refinement M
HA < 0:
head items, in decreasing order of confidence
foreach E 7, do [T..Tal.
Oz,A TTl
// Compute conditional frequency
of > 0 then
insert (I, A..A) into HA:
Add neuron x to list
M' none;
Amin + 0;
|/ gain estimate in bits // Refine model with rule using first labels
for 1... HAl do
M' Me H l: - A}:
10
A, <- M) L(D, M');
if A, < A min then
12
A Amn + A;
M' t M @ (HA[:t) A):
I/ Update best rule set
return M
The key component is hidden in the gain estimate line 10, which for the given rule X - A determines the best value & of items in the head needed for rule to apply. That is, we test all for all transactions sets determining where the rule applies T = {t D [X U 2 k} which one gives the best gain. To generate mine rules with disjunctive heads, which means we only have to consider T corresponding to a disjunction- in the search process.
rules going from the output layer hidden layer, want to
To generate candidates from existing rules in M, we use an extended search scheme that allows to merge pairs of rules with approximately equal heads, hav-
16

---

Under review as a conference paper at ICLR 2021
ing up to μ dissimilar items, measured by the symmetric set differences between e.
Algorithm 2: GENCANDIDATEEMERGEES
Input: Dataset D, Model M, overlap threshold μ
Output: Candidates C' sorted by gain A
C t ;;
Candidate rule merges
for Ti X1 Y1 M, M, X2 Y2 M do
if e X2 S then
An L(D,M M (X1 X {uK)) I(D, M); // Gain of adding conjunction of heads
An < 0 then
insert (X, 0 X2 + Y1 YYUY, UY, An) into C Aut s L(D, M @ (Xl X + Yi (Y)) L(D, M); Gain of adding disjunction of heads
I/ Add to candidates
if Au < 0 then
insert (X1 2 UY, Au) into C:
I/ Add candidates
10 return
Using the candidate generation methods, we can now write down the pseudocode for EXPLAINN, which iteratively generates candidates and commits the candidate with highest gain, until there is no more candidate that yields any gain in terms of MDL.
Algorithm 3: EXPLAINN
Input: Dataset D over layers I,, J,. frequency threshold 0, overlap threshold j Output: Best model M
M + (0 - A E
Initialize model with baseline rule set
foreach A E I do
R' < DENCANDIDATENENWDD..
|| App. Alg.
M' M @ R':
L(D, M") < M) then
M M';
repeat
M M: C + GENCANNIDAAEMEERGRSSS M, p);
// App. Alg. 2 Keep track independence merged rules
10
+
foreach X + +Y y
12
M' < $ YY:
Refine model, test gain
if L(D, < L(D,M) then
M < Y
until M M; 16 return M
17

---

Under review as a conference paper at ICLR 2021
B EXPERIMENTS AND DATA
Here, we detail the setup and training of the individual networks, and provide further experimental results.
B. MNIST TRAINING
We trained CNN on the MNIST data set using the Keras framework, using 60000 images for training and 10000 images as hold out test set for evaluation. The network consists of convolutional layers, with 20 filters in the first layer and 40 filters in the second layer, each using 3x3 kernels and 2x2 maxpooling. The convolutional layers are followed by a Dropout layer with dropout rate 25, and the flattened outputs passed activations. Then follows a dropout layer with rate .5 and the output layer of size 10 with softmax activations. The network trained using AdaDelta with default parameters based categorical cross entropy loss over 12 epochs using a batch size of 128. We gathered binarized activations across all filters and applied EXPLAINN build rules from the output layer to the first respectively second convolutional layer.
fully connected layer with 64 nodes with ReLU
Figure 9: MNIST prototype. Prototype image for filter from the first convolutional layer.
18

---

Under review as a conference paper at ICLR 2021
2
7
8
(a) MNIST Average activations. Average activation maps across a class for filter 32.
(b) MNIST Prototype. Prototype image for filter 32.
2
3
7
(c) Horizontal edge detector: Discovered rules, feature groups found across classes share same colour.
Figure 10: Filter visualizations. Activation maps (a) for the classes, the prototype of the filter (b), and discovered rules (c), over the whole dataset for filter in the second convolutional layer.
19

---

Under review as a conference paper at ICLR 2021
2
5
6
7
8
Figure 11: The negative of a digit. Visualizations for filter 12 in the first convolutional layer. This filter seems to capture the 'negatives' the handwritten digits.
20

---

Under review as a conference paper at ICLR 2021
B.2
IMAGENET FURTHER RESULTS
B.2.1
GOOGLENET RESULTS
Labels
Curly Coated Retriever Chesapeake Bay Retriever Irish Water Spaniel Poodle
Peacock
Obelisk
Last layer
(a) Prototypes rules from output label last hidden layer.
(b) Samples for curly haired dog breeds. From left: Curly Coated Retriever, Chesapeake Bay Retriever, Irish Water Spanies,Fooole
Figure 12: GoogLeNel results on ImageNet. Vaiuaaiilss for the rules found between the labels and the last hidden layer in GoogLeNett. The labels in the rule heads written above the prototype images of the tail unit groups. Each rule tail captures some interesting features of the corresponding classes: In the first rule the characteristic curly hair of different dog breeds is captured, the second group encapsulates information about the typical colourful plumage of peacocks, the third captures the shape of obelisks. We provide example images of the curly haired dog breeds in (b).
21

---

Under review as a conference paper at ICLR 2021
B.2.2 VGG SHARED NEURONS
Japanese spaniel
Pekinese
Shih-Tzu
Labels
Lhasa
Affenpinscher
Red-breasted merg. Redshank Oystercatcher 1
Tusker Indian elephant African elephant
Viaduct Brabancon griffon Triumphal arch
Pug
FC7
Figure 13: Shared information across labels. Visualizations for the rules found between the labels and the last fully connected layer (FC7). The labels in the rule heads written above the prototype images of the tail unit groups. Each rule tail captures some interesting features of the corresponding classes: In the first rule the characteristic face of different dog breeds is captured, the second group encodes information about the arch structures present for both Vi aduct and Tri umphal a arch, the third captures the red beaks surrounded by blackish feather that are shared between different birds, and the fourth shows typical heads and tusks of elephants.
Visualization for the units in the tail individually
Visualization for the whole tail
Figure 14: The ugly face neurons. From the data for all dog breed categories, EXPLAINN discov- ered the rule between the labels (Japanese spaniel, Pekinese, Shih-Tzu, Lhasa, Affenping senhinscer, Pug, Brabancon griffon). and units from the FC7 layer, for which a prototype is given in the right image. The 5 units together capture the rather characteristic face of the mentioned breeds. Visualizing these 5 units one by one, given on the left, gives only little insight about the encapsulated information
We discovered many shared traits that the networks is able pick up across classes, which are encapsulated in groups of neurons in the last layer. For example, there are neurons that capture the red beaks of different birds, arch like structures of buildings, tusks of elephants and object itself, and the ugly face of a whole group of different dog breeds (see App. Fig. 13). Even we would visualize all neurons individually and handpick them, without the knowledge about the rule,
22

---

Under review as a conference paper at ICLR 2021
e.g. for the ugly dogs, it is very hard interpret what they encode, and only few of them give hints about what the information is (see App. Fig. 14). In practice, we only visualize class prototypes, which can be very misleading, shown in the next section.
B.2.3 RESOLVING THE MEANING OF CLASS PROTOTYPES
Class prototype for Great Danes
Row 1: Example images from the class Row 2: Rules in association with Great Danes
Figure 15:
The left image shows the visualization for the whole class Great Danes. This visu- alization could not highlight many characteristic features, since there is large diversity within the class. On the right side images from the dataset is shown, along with 3 rules that EXPLAINN finds in connection with the class label. We are able to pick up trends, that are not characteristic to the whole class, but only a subset.
A standard technique to capture traits that a neural network learns about class is prototyping the given class label. These can offer hints on what the network learns globally about the class, but very often lead to uninterpretable results. We provide one such example prototype for the Great Dane class in Imagenet of the VGG network in App. Fig. 15, which does not provide any clue what the network learns. The rules discovered by EXPLAINN however show, that different groups of neurons the last layer lead the Great Dane classification, each encoding a distinct type and pattern that appear with this breed. The class prototype is mixture of these different types, which explains the difficulty to interpret that prototype.
colour
23

---

Under review as a conference paper at ICLR 2021
FCG
CONVS
Side dogs. The visualization for the FC6 nodes resembles animals from side view, typical image positioning for dogs. the images generated for associated units in the convolutional layer, how the network uses different filters put this pattern together. The first depicted filter captures a dog head, typically found at the top left part of the image. The second and third filters focus on the legs. In the third row, activated with the feature, showing that there are typical positions for legs in both the left and right lower parts of the image. In the last three prototypes, recognize horizontal lines, which overall resembles the typical shape and position in the image where we would expect the back of an animal.
Figure 16:
that other units for these filters also commonly
24

---

Under review as a conference paper at ICLR 2021
Figure 17: Flower visualizations. For rules found between output and last fully connected layer, we visualize the neurons in the tail of the rule for the fine-tuned VGG-S network (first), the original VGG-S network (second), and example images for the flower classes (right).
B.3 OXFORD FLOWER DATA
25

---

Under review as a conference paper at ICLR 2021
B.4 ADDITIONAL RESULTS
In App. Fig. 18, 19, we provide additional results based on prototyping for rules found for ImageNet data and the VGG-S network. We focus on rules with multiple neurons in the tail, such class and multiclass prototypes can hardly be found by hand. Overall, we observed that the larger the number of neurons in the tail, the sharper and more interesting the resulting protoype. Furthermore, we found that for many prototypes spanning multiple classes, we discover multiple rules for some of these classes (e.g. Black Grouse) and the protoypes indicate that only fraction of information, such as patterns, a colored leg or beak, or a color patch, is used from each group of neurons such that together they arrive at the class prediction.
In App. Fig. 18, the first row of the panel are examples of neuron groups that learn typical shapes of objects, such as Sombrero or Gondola a. The second row contain groups of neurons capturing typical patterns and colors for individual classes, such as yellow patches on black skin of the F Fire Sal amaadrnerr, red caps with white dots of the Agaric mushroom, the typical leaf with red veins of Sorrel the wings of a Monarch butterfly. The third row contains common features between two classes that are together captured by the same group of neurons, like the arch-like structures and round rooftops found for certain Tri umphal Archs and Mosques, the layered and intertwined worm-like shapes of many Fur Coats and the Gyromira mushroom, or the characteristic tradi- tional covering of yurts and the front part of dogsleds.
In App. Fig. 19, groups of neurons that are shared between multiple classes are visualized both revealing surprising similarities, as well as confirming that the network learns similarities that we also use as human. In the first row, the neurons described by the first two images capture the typical shape and red color of the ears shared between the Red Fox and the Lesser Panda, respectively the insect legs and shiny turquoise color of the body of Tiger Beetles and Damsel flies. Intriguingly, the network also learns a roundish shape and distinct pattern between the Jackfruit and the Squirrel Monkey, At this point, we would like to invite the reader to look up how the top of the head of such a monkey looks like, it resembles surprisingly well the size, color, dotted
shape, and texture of Jackfruit. For the last picture in the first of this panel, wings that clearly are related with the associated labels Cabbage Butterfly, and Sulphut Butter f fly. But opposed visualizations related to other Butterflies (given in both panels), the wings are all oriented in a distinct way, which resemble the cap of a dotted mushroom, which might explain the association with the Agaric mushroom. In the second row, we observe that the network captures common features shared between similar classes in this case closely related animals with the same of neurons, which matches human intuition.
26

---

Under review as a conference paper at ICLR 2021
Sombrero 1
Gong 1
Barn
Gondola a
2
2
2
3
Europ.
Fire
Salam.
Agaric 4
Sorrel
Monarch
4
2
3
2
5
Rapeseed
Mosque Triumphal Arch 4
Pur Coat Gyromira
Dogsled Yurt
7
2
2
2
Figure 18: Diverse protolypes. Visualized are prototypes for rules found in the VGG-S network for ImageNet data between the output and last hidden layer. The class labels corresponding to the output given above each image, the size of the group of neurons that this picture generated from given the bottom right.
27

---

Under review as a conference paper at ICLR 2021
Red Fox
Tiger Beetle Damselfly
Jackfruit Squirrel Monkey Sulphur Butterfly
Cabbage Butterfly
Lesser Panda
Agaric
2
2
2
3
A. S. S. Terrier Bull Mastif Saint Bernard Black
Water Ouzel
Leopard Snow Leopard Cheetah
Ringlet Cabbage Butterfly Sulphur Butterfly
Black Grouse
Footed Ferret
3
3
7
8
Figure 19: More prototypes. Visualized are prototypes for rules found in the VGG-S network for ImageNet data between the output and last hidden layer. The class labels corresponding the output are given above each image, the size of the group of neurons that this picture was generated from is given in the bottom right.
28


Reviews:
[{"review_id": "RBUcnLQ05Yf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary\n\nThe authors propose a method to explore how neurons interact within a neural network and derive rules of interactions that can help interpret the inner workings of the neural network and open up the black box. The algorithm, EXPLAINN, identifies rules between successive layers where each rule represents a set of neurons that are activate simultaneously and conditionally based on the previous layer. Minimum Description Length principle is used to derive an objective that minimizes the number of bits used to encode the rules. The rule sets are identified using a greedy heuristic and improved until convergence of the objective. The algorithm is then evaluated to demonstrate the interpretation of images with MNIST, GoogLeNet and VGG-S. \n\n\nPositives\n* The problem at hand is clearly important and can help with interpretation of the neural networks.  This is particularly important in fields such as biology (genomics) where better interpretation is often times desired at a slight cost in performance \n* The formulation using Minimum Description Length Principle is definitely an interesting idea - particularly formulating the objective as set of independent and robust rules \n* The framework is flexible and allows for discovery of rules relevant for subsets / combinations of classes and not just individual examples or global rules across examples \n* The paper in general is well written and I particularly liked how the authors guide the readers through the different aspects of the algorithm evaluation. I particularly liked Appendix A since the concrete example made the objective function clear. The notations however can be better laid out and clarified. \n\n\nConcerns\n* My primary concern is the use of greedy heuristic to identify the rule set. This necessitates that each interaction individually carry some degree of information for a robust composite rule. While the rules identified through the heuristic could be informative, it is not clear whether they are the best set of rules since there are no bounds or guarantees of how the heuristic relates to the global optimum. \n* On a related note, I felt the evaluation presented by authors while extensive is rather qualitative in nature. While the prototypes of identified rules across different datasets look relevant and interesting, it is not clear whether they are the best set of rules. So, I believe this manuscript needs benchmarking in datasets constructed with known rules (possibly through simulation) to alleviate these two major concerns \n* The quantitative outputs of the ReLU activations are binarized for rule learning - the authors have not addressed how this impacts the accuracy of the rules since that quantitative information is used by the subsequent layers \n* The impact of the threshold parameters theta and mu (Algorithms 1 & 2) are also not addressed - this seems critical to me since these thresholds define the initial set of rule candidates.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MkD-jIg_DFr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to explainable supervised learning by extracting sets of rules for two individual layers within a neural network. The authors build their work on recent published work for patttern-based rule mining [0] to efficently find so-called robust rules. The authors evaluate the approach for image processing tasks with convolutional neural networks on MNIST, ImageNet and Oxford Flower by comparing generated rules against activation maps and prototypes.\n\nThe proposed approach is interesting, but I am left with several concerns and open questions with respect to contributions, available related works and the conducted evaluation. \n\nWhile the paper discusses quite nicely summarized several relevant related works in the introduction section, I am still wondering how the approach is related to OpenAI's Circuits [1], where already individual neurons and connections between neurons are studied with respect to interpretability/explainability of neural nets? This is not to say that prior work already learned sets of rules among layers, but - if relevant - it should be evaluated to what extent the proposed method is superior for explaining neural nets to end-users. A smaller comment to the related work is that there might be missing references for model destillation, such as [2].\n\nThe taken approach to rule generation follows recent work on association rule mining, which is sensible. To this end, the paper misses to clearly address the difference to GRAB, i.e. the referenced algorithm used for learning the rules using model description length (MDL). Could you therefore elaborate why GRAB cannot be applied to the rule mining task out-of-the-box or clearer state if this what you have actually done? I feel like the presentation of the paper could benefit from answering this question, one could  establish a background section and/or focus on novel aspects of the proposed algorithm. In addition, there might be more space for the evaluation, which should be a major part of the paper's contribution, especially explainable ML.\n\nThe idea of exploring neural connections among two layers is generally intriguing. The paper could benefit from better motivating this design choice, i.e. why use \"only\" two layers? Would it make sense to use more than two convolutional layers for deeper networks? Would this be computationally tractable? Are two layers better to interpret for humans?\n\nWhile the evaluation is insightful, I am not convinced that single-neuron-prototypes and activation maps are representative for all ongoing works on explainable and interpretable ML. More specifically with respect to activation maps, is it possible to rule out all activation map approaches for CNNs at once? What about perturbation-based approaches (e.g. [3])? To this end, I am wondering why you only compare to single neuron prototypes and not more complex prototypes of the individual class (for Sec. 3.1 and the MNIST experiment)? Lastly, I am also wondering if a user study would be helpful to confirm that the proposed explanations provide added value for end-users.  \n\nA minor comment is that it is not helpful for the reader that numerous references figures are in the appendix (e.g. the comparison of generated MNIST rules in Sec. 3.1 to the prototype in the appendix). A minor question would be why you define \"prototype\" as on a single-neuron-basis (in the introduction)? Is there a reference for the approach?\n\nReferences:\n[0] Fischer, J. and Vreeken, J., 2019, September. Sets of robust rules, and how to find them. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 38-54). Springer, Cham.\n[1] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Zoom In: An Introduction to Circuits. Distill, 5(3), pp.e00024-001.\n[2] Bastani, O., Kim, C. and Bastani, H., 2017. Interpreting blackbox models via model extraction. arXiv preprint arXiv:1705.08504.\n[3] Fong, R.C. and Vedaldi, A., 2017. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3429-3437).\n\n### Update after author response ###\nThanks again for the clarifications - After reading the author responses, the other reviewers' comments and the new version of the manuscript, I increase my score for the paper, as the authors now better state the relationship to Circuits and GRAB, and provide a significantly improved evaluation. The enhanced experimental section is now adequate for the paper's claims and offers additional insights into the usability of the generated rules. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EUOZOpJx8i", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work makes a convincing case that we need to trace information flow (or at least, to find clusters of neurons working together) within a deep neural network in order to get the clearest picture of how the network is working.  The approach is quite simple but also quite novel in that it uses concepts from coding theory which are not widely known in the deep learning community.  The paper is well written and presents examples from deep CNNs (~20 layers) trained on ImageNet.\n\nQuality:\nThe method is well thought-out and explained, and two different evaluations are used (MNIST and ImageNet).  It would be more impressive if the authors had included a problem from a different data modality since in principle the method is quite general.  The examples look very impressive, but my main concern is with whether the examples could have been cherry-picked, in the sense that most of the thousands of rules produced may not be useful.  Relatedly, I would like to know the reproducibility of the result.  If you train twice with different seeds, how similar are the results?  Or if you fit ExplainNN with different seeds on the same network?  And what is the danger of false positive findings?\n\nClarity:\nThe paper is very clear with regards to the problem setting, previous work, and the methods.  I am somewhat familiar with coding theory, having read much of MacKay's Information Theory, Inference, and Learning Algorithms, but I am by no means an expert in coding theory.  Still, I am confident that I understand the principles of the method.  However, I did not exactly follow how the authors carried out the tracing (pg 7 second paragraph.). Do you just apply ExplainN as usual and then filter for rules that (strictly/non-strictly) include Y?  This seems important to explain since the tracing, in my view, is the main contribution, given that there already exist tools for understanding the similarities of classes such as representational similarity (https://roberttlange.github.io/posts/2019/06/blog-post-3/).  \n\nOriginality:\nI have not previously seen the idea of mining association rules for deep neural networks, although it is a simple enough idea and I would not be surprised if the idea has appeared before.  However, the application of MDL to solving the problem for binarized activations is likely to be novel.\n\nSignificance:\nThe work is promising, but to me it is not conclusive that it will make a lasting impact.  There are a number of important practical questions to be addressed that could make or break the method as a tool for the field, such as the reproducibility and usability of the method (whether most rules produced are meaningful or significant manual filtering is required).  On the other hand, even if the method does not meet practical needs, it could still be of great utility for NN methods researchers interested in investigating, say, the redundancy of specific architectures. the true degree of similarity between two trained models, convergent dynamics between alternative architectures, and the value of overparameterization.  The paper could be even more significant if the authors could comment on the generalizability to other architectures such as RNNs, GNNs or transformers.  The method itself is interesting enough and the examples sufficiently compelling (even if cherry-picked) that I would recommend the paper to almost anyone interested in neural network interpretability.\n\nNote on rating:  In the face of limited details, I am willing to give the authors the benefit of the doubt that they did not cherry-pick overly aggressively and that the examples are representative of typical outputs. If it turns out that most rules do not look like the examples, then my rating would decrease.  On the other hand, if the authors can address my concerns about cherry-picking in the response, it is possible that I would raise my rating.\n\nPros:\n * important application\n * method is quite general\n * method is simple and intuitive\n\nCons:\n * evaluation of method performance limited to selected examples\n * reproducibility not addressed\n * control of false positives not addressed\n * method for tracking across layers not well-explained", "extra_scores": {}, "extra_text": {}}, {"review_id": "ECCIcwudWW4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to extract interpretable rules from a learned neural network. The authors claim that they are the first to propose rules connecting 1) multiple neurons together, and 2) do this at a dataset level. Their approach relies on using minimum description length and well known principles from the data mining community (e.g., downward closure lemma of apriori algorithm). The authors claim that experiments conducted on image data shows that their approach leads to more faithful, interpretable rules than other approaches such as prototyping or model distillation.\n\nCompiling rules from neural networks has been proposed before (e.g., see \"Deep Logic Networks: Inserting and Extracting Knowledge from Deep Belief Networks\" by Tran and d'Avila Garcez, 2018). I think the authors need to compare against such previous approaches to quantitatively show how their work extracts better rules. Otherwise, its difficult to appreciate the value of ExplaiNN. Also, the paper doesn't say anything about how faithful the rules are to the learned neural network. I mean, it seems possible that for some input the rules could produce a different output from the neural networks. There exist other works that also try to interpret neural networks consisting of affine layers with Relu activation that guarantee consistency (see \"Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution\" by Chu et al in KDD'18). The authors should at the very least compare and contrast with such works to highlight the pros and cons. Another undesirable property of ExplainNN seems to be that it relies on a dataset to derive its rules. Is it possible that when run with a different dataset and the same learned network, ExplaiNN would produce a different set of rules? Then how much faith do we place on ExplaiNN's output?\n\nWriting wise, the paper is presented well enough. There's a few paragraphs in the Experiments section where the authors point repeatedly to the Appendices. In the best case this makes reading a chore. I would advise the authors to refrain from using the main body of the paper as a listing of contents and simply pointing to the appendices. The pictures in the experiments section were difficult to make out. I couldn't figure out from the image whether the husky's pointed snout had been identified as a defining feature. I would hope the authors find more compelling ways to make their point.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to explore neuron interactions within a neural network by deriving rules for the activations of units at different layers. The rules can presumably help interpret the inner workings of the neural network. 
The reviewers have very different opinions on the paper and the views did not converge.  However, there is a common concern on the lack of quantitative evaluation on the faithfulness of the rules to the models. I therefore do not recommend accept. 

R1[5]: On a related note, I felt the evaluation presented by authors while extensive is rather qualitative in nature.
R2[3]: Given that I could provide you with a couple of references that you admit is relevant, and this was just off the top of my head, would you care to comment on a quantitative comparison with the referenced approaches?
R3[8]: The examples look very impressive, but my main concern is with whether the examples could have been cherry-picked, in the sense that most of the thousands of rules produced may not be useful.



Decision: Reject