What's in the Box? Exploring the Inner Life of Neural Networks with Robust Rules

1 INTRODUCTION . Neural networks provide state of the art performance in many settings . How they perform their tasks , how they perceive the world , and especially , how the neurons within the network operate in concert , however , remains largely elusive . While there exists a plethora of methods for explaining neural networks , most focus on the mapping between input and output ( e.g . model distillation ) or only characterize individual neurons within the network ( e.g . prototyping ) . In this paper , we introduce a new method that explains how the neurons in a neural network interact . In particular , we consider the activations of neurons in the network over a given dataset , and propose to characterize these in terms of rules X → Y , where X and Y are sets of neurons in different layers of the network . A rule hence represents that neurons Y are typically active when neurons X are . For robustness we explicitly allow for noise , and to ensure that we discover succinct and non-redundant rules we formalize the problem in terms of the Minimum Description Length principle . To discover good rule sets , we propose the unsupervised EXPLAINN method . The rules that we discover give clear insight in how the network performs its task . As we will see , they identify what the network deems similar and different between classes , show how information flows within the network , and show which convolutional filters it expects to be active where . Our rules are easily interpretable , give insight in the differences between datasets , show the effects of fine-tuning , as well as super-charge prototyping as they tell which neurons to consider in unison . Explaining neural nets is of widespread interest , and especially important with the emergence of applications in healthcare and autonomous driving . In the interest of space we here shortly introduce the work most relevant to ours , while for more information we refer to surveys ( Adadi & Berrada , 2018 ; Ras et al. , 2018 ; Xie et al. , 2020 ; Gilpin et al. , 2018 ) . There exist several proposals for investigating how networks arrive at a decision for a given sample , with saliency mapping techniques for CNNs among the most prominent ( Bach et al. , 2015 ; Zhou et al. , 2016 ; Sundararajan et al. , 2017 ; Shrikumar et al. , 2017 ) . Although these provide insight on what parts of the image are used , they are inherently limited to single samples , and do not reveal structure across multiple samples or classes . For explaining the inner working of a CNN , research mostly focuses on feature visualization techniques ( Olah et al. , 2017 ) that produce visual representations of the information captured by neurons ( Mordvintsev et al. , 2015 ; Gatys et al. , 2015 ) . Although these visualizations provide insight on how CNNs perceive the world ( M. Øygard , 2016 ; Olah et al. , 2018 ) it has been shown that concepts are often encoded over multiple neurons , and that inspecting individual neurons does not provide meaningful information about their role ( Szegedy et al. , 2013 ; Bau et al. , 2017 ) . How to find such groups of neurons and how the information is routed between layers remains unsolved . An orthogonal approach is that of model distillation , where we train easy to interpret white box models that mimic the decisions of a neural network ( Ribeiro et al. , 2016 ; Frosst & Hinton , 2017 ; Bastani et al. , 2017 ; Tan et al. , 2018 ) . Rules of the form ( if–then ) are easily interpretable , and hence a popular technique for model distillation ( Taha & Ghosh , 1999 ; Lakkaraju et al. , 2017 ) . Although some consider individual neurons to determine what rules to return ( Robnik-Šikonja & Kononenko , 2008 ; Özbakır et al. , 2010 ; Barakat & Diederich , 2005 ) all these techniques only yield rules that directly map input to output , and hence do not provide insight into how information flows through the network . Tran & dAvila Garcez ( 2018 ) mine association rules from Deep Belief Networks . Their approach , however , suffers from the pattern explosion in typical in frequency based rule mining , and is not applicable to state of the art networks . Chu et al . ( 2018 ) aim to explain piecewise linear NNs utilizing polytope theory to derive decision features of the network . Providing strong guarantees , this approach is however limited to PLNNs of extremely small size ( < 20 neurons in hidden layers ) . We instead propose to mine sets of rules to discover groups of neurons that act together across different layers in feed forward networks , and so reveal how information is composed and routed through the network to arrive at the output . To discover rules over neuron activations , we need an unsupervised approach . While a plethora of rule mining methods exists , either based on frequency ( Agrawal & Srikant , 1994 ; Bayardo , 1998 ; Moerchen et al. , 2011 ) or statistical testing ( Hämäläinen , 2012 ; Webb , 2010 ) , these typically return millions of rules even for small datasets , thus undermining the goal of interpretability . We therefore take a pattern set mining approach similar to GRAB ( Fischer & Vreeken , 2019 ) , where we are after that set of rules that maximizes a global criterion , rather than treating each rule independently . Providing succinct and accurate sets of rules , GRAB is however limited to conjunctive expressions . This traditional pattern language is too restrictive for our setting , as we are also after rules that explain shared patterns between clases , and are robust to the inherently noisy activation data , which both require a more expressive pattern language of conjunctions , approximate conjunctions , and disjunctions . We hence present EXPLAINN , a non-parametric and unsupervised method that discovers sets of rules that can model these types of rules . 2 THEORY . We first informally discuss how to discover association rules between neurons . We then formally introduce the concept of robust rules , and how to find them for arbitrary binary datasets , last , we show how to combine these ideas to reveal how neurons are orchestrated within feedforward networks . 2.1 PATTERNS OF NEURON CO-ACTIVATION . Similar to neurons in the brain , when they are active , artificial neurons also send information along their outgoing edges . To understand flow of information through the network , it is hence essential to understand the activation patterns of neurons between layers . Our key idea is to use recent advances in pattern mining to discover a succinct and non-redundant set of rules that together describe the activation patterns found for a given dataset . For two layers Ii , Ij , these rules X → Y , X ⊂ Ii , Y ⊂ Ij express that the set of neurons Y are usually co-activated when neurons X are co-activated . That is , such a rule provides us local information about co-activations within , as well as the dependence of neurons between layers . Starting from the output layer , we discover rules between consecutive layers Ij , Ij−1 . Discovering overlapping rules between layers X → Y and Y → Z , X ⊂ Ij , Y ⊂ Ij−1 , Z ⊂ Ij−2 , allows us to trace how information flows through the entire network . Before we can mine rules between two sets of neurons – e.g . layers – Ii and Ij of a network , we have to obtain its binarized activations for a given data set D = { dk = ( sk , ok ) } . In particular , for each sample sk and neuron set Ii , we take the tensor of activations φi and binarize it to φbi . For networks with ReLU activations , which binarize naturally at threshold 0 , we might lose some information about activation strength that is eventually used by subsequent layers , but it allows us to derive crisp symbolic , and directly interpretable statements on how neurons interact . Furthermore , binarization reflects the natural on/off state of biological neurons , also captured by smooth step functions such as sigmoid or tanh used in artifical neural networks . We gather the binarized activations into a dataset D where each row tk corresponds to the concatenation of φbi and φ b j of Ii and Ij for sk , i.e. , tk ∈ D is a binary vector of length |Ii|+ |Ij | . See Fig . 1 for a toy example . Next , given D , our goal is to find that set of rules that together succinctly describe the observed activations . The Minimum Description Length ( MDL ) principle lends itself as an objective to find such sets . MDL is a statistically well-founded and computable approximation of Kolmogorov Complexity ( Li & Vitányi , 1993 ) . First introduced by Rissanen ( 1978 ) , the essential idea is that the model M∗ ∈ M that best describes data D is the model that losslessly encodes D using the fewest bits M∗ = arg minM∈M L ( D , M ) . Here , our model classM is the superset of all possible rules over D , and by MDL we identify the best model M∗ as the one that compresses the data best . Traditionally , rule mining is restricted to conjunctions over items , which is not sufficient for our application ; neuron activations tend to be noisy , labels are inherently mutually exclusive , and hence we consider an extended language that allows for partial disjunctions of items ( neurons , labels ) and introduce a codelength function L ( D , M ) to instantiate MDL for our model class of rule sets . 2.2 MDL FOR ROBUST RULES . Our goal is to find a set of rules M that , in terms of description length L ( D , M ) , best describes a binary database D = { t | t ⊂ I } that consists of transactions t that are subsets of items I . Each rule is of the form X → Y , X , Y ⊂ I , and indicates that Y is strongly associated with , i.e . occurs mostly in transactions where X is present . We say a rule X → Y applies to a transaction t iff X ⊂ t and say a rule holds for t if additionally Y ⊂ t. We indicate these transactions sets as TX = { i | ti ∈ D. X ⊂ ti } , respectively TY |X = { i | ti ∈ TX . Y ⊂ ti } . Based on these definitions of rule transaction sets , we can now formally introduce our codelength function L ( D , M ) . Baseline model Our base model Mind = { ∅ → I | ∀I ∈ I } consists of singleton rules only , i.e . it models that all items I are generated independently . To send the n transactions of D using Mind , we simply send for each item I in which out of all transactions in the database it appears . We can do so optimally using a log binomial code , which is given by log ( |T∅| |TI|∅| ) = log ( n |TI | ) . To unambiguously decode , the recipient needs to know each |TI | , which we can optimally encode via the parametric complexities of the binomials , which are defined as Lpc ( n ) = log ( ∑n k=0 n ! ( n−k ) ! k ! ( k n ) k ( n−k n ) n−k ) , and can be computed in linear time ( Kontkanen & Myllymäki , 2007 ) . We thus have L ( D , Mind ) = ∑ I∈I ( log ( n |TI | ) + Lpc ( n ) ) . Mind serves as our baseline model , and its singleton rules are a required part of any more complex model as they ensure we can always send any data over I. Non-trivial models A non-trivial model M contains rules of the form X → Y , X , Y ⊂ I that are not part ofMind . The idea is that we first transmit the data for where these non-trivial rules hold , and then send the remaining data using Mind . To determine where such a rule applies , the receiver needs to know where X holds , and hence the data over X needs to be transmitted first . To ensure that we can decode the data , we only consider models M for which the directed graph G = ( I , E ) is acyclic , in which there exists an edge between two items i1 , i2 iff they occur in the head and tail of a rule , that is E = { ( i1 , i2 ) | ∃X → Y ∈M . i1 ∈ X ∧ i2 ∈ Y } . We thus get a codelength L ( D |M ∪Mind ) = ( ∑ X→Y ∈M log ( |TX | |TY |X | ) ) + ∑ ∅→I∈Mind log ( n |T ′I | ) , where T ′I = { t ∈ D | ( I ∈ t ) ∧ ( ∀X → Y ∈M . I ∈ Y =⇒ t 6∈ TY |X ) } is a modified transaction set containing all transactions with item I not covered by any non-trivial rule . In addition to the parametric complexities of the binomial codes , the model cost of a non-trivial model also includes the cost of transmitting the non-trivial rules . To transmit a rule X → Y , we first send the cardinalities of X resp . Y using the universal code for integers LN ( Rissanen , 1983 ) . For n ≥ 1 , this is defined as LN ( z ) = log∗ z + log c0 with log∗ ( z ) = log z + log log z + . . . , summing only over the positive components ( Rissanen , 1983 ) . To satisfy the Kraft equality we set c0 = 2.865064 . Knowing the cardinalities , we can then send the items of X resp . Y one by one using an optimal prefix code given by L ( X ) = − ∑ x∈X log |Tx|∑ I∈I |TI | . For a particular rule X → Y ∈M , the model costs for a rule , respectively the full model thus amount to L ( X → Y ) = LN ( |X| ) + LN ( |Y | ) + L ( X ) + L ( Y ) + Lpc ( |TX | ) , L ( M ∪Mind ) = |I| × Lpc ( n ) + LN ( |M | ) + ∑ X→Y ∈M L ( X → Y ) . We provide an example in App . A.1 . With these definitions , we have an MDL score that identifies the best rule set M∗ for data D as M∗ = arg min M∈M ( L ( M ∪Mind ) + L ( D |M ∪Mind ) ) , whereM is the space of all possible rule sets over the items in D. Robust Rules In real world applications , which are inherently noisy , we need a score that is robust and can approximate where rules hold and apply . In addition , as output neurons are only active mutually exclusively , rules need to be able to model disjunctions . The key problem with noisy data is that a single missing item in a transaction can cause a whole rule not to hold or apply . To discover rules that generalize well , we need to explicitly account for noise . The idea is to let rules apply , and hold , approximately . Specifying how many items l , and k , out of all items in the rule head , respectively tail , need to be part of a transaction , we relax the original rule definition to account for random noise ( see App . Figs . 7 , 8 ) . Setting l = 1 and k = 1 coincidentally corresponds to a disjunction of items in the head and tail of the rule X → Y , thus allowing to model output neurons correctly , and l = |X| and k = |Y | correspond to the original stringent rule definition . Varying between the two extremes accounts for varying levels of noise . The optimal l and k are those that minimize the MDL score . To ensure a lossless encoding , we need to make sure that the receiver can reconstruct the original data . Thus , for the relaxed head definition , for each rule we send the number of items l for the rule to apply using LN ( l ) bits . Knowing each l , the receiver can reconstruct where each rule applies . Sending where a rule holds now leaves the receiver with an approximation of the data . To be able to reconstruct the actual data , Fischer & Vreeken ( 2019 ) introduced error matrices that when XORed with the approximation yield the original data . These two matrices X+X→Y , and X−X→Y correct for the errors made in the part where the rule applies and holds , respectively applies but does not hold ( for an example see App . Fig . 7 ) . These error matrices are part of the model M and have to be transmitted with an adapted L ( D , M ) , which we review in App . A.2 . Complexity of the search To discover rules over the activations of two layers Ii , Ij , we have to explore all rules formed by subsets of neurons in Ii for the head , combined with any subset of neurons of Ij for the tail . There exist 2|Ii|×2|Ij | such rules , and hence 22 |Ii|+|Ij | distinct models would need to be explored . Fischer & Vreeken ( 2019 ) showed that the rule set search space does not lend itself to efficient search as it is neither monotone nor submodular , the counterexamples also holding for our model definition . In fact , for robust rules , we additionally have to consider where rules should apply respectively hold – optimizing k and l – which results in approximately 2|Ii|×|Ij |×2 |Ii|+|Ij | models ( detailed derivation in App . A.4 ) . Exhaustive search is therewith infeasible , which is why we present EXPLAINN , a heuristic algorithm to efficiently discover good sets of rules .

Summary: The authors propose a method to explore how neurons interact within a neural network and derive rules of interactions that can help interpret the inner workings of the neural network and open up the black box. The algorithm, EXPLAINN, identifies rules between successive layers where each rule represents a set of neurons that are activate simultaneously and conditionally based on the previous layer. Minimum Description Length principle is used to derive an objective that minimizes the number of bits used to encode the rules. The rule sets are identified using a greedy heuristic and improved until convergence of the objective. The algorithm is then evaluated to demonstrate the interpretation of images with MNIST, GoogLeNet and VGG-S.