Characterizing the Accuracy/Complexity Landscape of Explanations of Deep Networks through Knowledge Extraction

Abstract: Knowledge extraction techniques are used to convert neural networks into symbolic descriptions with the objective of producing more comprehensible learning models. The central challenge is to find an explanation which is more comprehensible than the original model while still representing that model faithfully. The distributed nature of deep networks has led many to believe that the hidden features of a neural network cannot be explained by logical descriptions simple enough to be understood by humans, and that decompositional knowledge extraction should be abandoned in favour of other methods. In this paper we examine this question systematically by proposing a knowledge extraction method using \textit{M-of-N} rules which allows us to map the complexity/accuracy landscape of rules describing hidden features in a Convolutional Neural Network (CNN). Experiments reported in this paper show that the shape of this landscape reveals an optimal trade off between comprehensibility and accuracy, showing that each latent variable has an optimal \textit{M-of-N} rule to describe its behaviour. We find that the rules with optimal tradeoff in the first and final layer have a high degree of explainability whereas the rules with the optimal tradeoff in the second and third layer are less explainable. The results shed light on the feasibility of rule extraction from deep networks, and point to the value of decompositional knowledge extraction as a method of explainability.

Reviews and author responses:

Thank you for your comments,

It's true that a rule based explanation may not be interpretable to a general audience. However, the aim of the paper is to explore to what extent a rule based explanation is even feasible in the first place. The results show that in certain layers of a CNN even rules which are very complex fail to accurately explain a hidden feature which suggests that other methods need to be employed in order to understand that hidden feature.

We plan on applying the method to a variety of datasets and architectures in order to gain insight into the conditions which may lead to simple and accurate rule based explanations of hidden features. Whether or not even a simple rule based explanation can be considered interpretable is a more general question for which one would have to consider both the audience and the dataset.

Thank you for your comments,

Our aim in this paper is not to produce a method which can reliably explain a neural network with rules but to explore to what extent it is possible to explain a hidden feature using rules of varying complexity.

We've included results on the DNA promoter dataset showing an explicit example of extracted rules as well as a fully illustrated example of rules of varying complexity extracted from a hidden neuron in the first layer of the .  Future experiments will involve multiple architectures and datasets in order to identify variables that make neural networks explainable with rule extraction.

Our method is novel in that it is a brute force approach which makes it infeasible for large networks but can be used on small or medium sized networks to provide a complete picture of the tradeoff that any extraction algorithm must make between accuracy and complexity. The possible benefit of the approach is twofold. First, it can give us insight into the conditions which make a network more or less explainable with rule extraction. Second, it can serve as a benchmark for other rule extraction algorithms to evaluate how close they are coming to the ideal complexity/error tradeoff.


Thank you for your comments,

We've now included a section with an example to fully illustrate how the method works. 

We've also run the experiments on the DNA promoter dataset and find a similar accuracy/complexity relationship. We will continue to run experiments on multiple datasets in the future. It should be noted that there are many possible variables which can influence the complexity of hidden neurons including network architecture, dataset, and learning algorithm. We plan on making the code publicly available so that similar experiments can be run to gain insight into how these variables effect the complexity of the representations learned by a neural network.

Although we use a relatively small number examples to evaluate the rules, repeating the experiments on a single neuron when evaluated on 5000 and 10000 examples produces very similar accuracy/complexity curves leading us to believe that the experiments give a fair representation of the relationship between the complexity and accuracy of extracted rules.

Thank you for your comments. 

As you describe, our intention for the experiments was to evaluate the complexity of the concepts learned by hidden neurons. In the binary case every neuron can be described exactly by a rule so any error from an extracted rule is the result of not finding the optimal solution. Usually the failure of an extraction algorithm to find an accurate solution is the result of not allowing rules of sufficient complexity. Our proposed method is meant to show the degree to which failures of an extraction method are due to restricting complexity.

With regards to the weight-ordering, for a binary network it can be shown that, when evaluated over the whole input space, the best M-of-N rule is the one which includes the N literals with highest weight magnitude. The same is true for a continuous network on an input space of the form [a,b]^n if the neurons are binarized using the same value to split them.

Since testing over the whole input space is infeasible we evaluate the rules on a subset of the training data which leaves open the possibility that a different ordering could result in more accurate rules on that subset but we feel that because of the reasons mentioned above the weight-ordering produces the most accurate M-of-N rules.

Of course the search space of M-of-N rules themselves is only a subspace of all possible rules. We've included additional experiments on the simple but well known DNA-promoter dataset and find that complex M-of-N rules can accurately approximate the network suggesting that M-of-N is a reasonable restriction of the search space and that the results are a good representation of the accuracy/complexity relationship for extracted rules.

With regards to the second point, in principle you can replace the whole network with a hierarchy of extracted rules as long as you choose consistent splits to binarize the neurons in each layer. As you point out though this will cause errors to accumulate. Our goal was to produce a baseline accuracy/complexity relationship, practical rule extraction algorithms might not achieve these results but the method allows us to gain insight into feasibility of rule extraction as a method of explanation.

The results for robustness were tentative and have been removed in order to make room for a more complete discussion of the method. Further experiments on knowledge extraction and robustness will be done in the future.

This paper is looking at how to extract knowledge from CNNs to help improve explainability and robustness against an adversarial attack.  It is using a known technical call M-of-N rules.

This problem of explainability of NN's is an important one and rules are a good step in that direction.  

+ The paper is generally well written

- The contribution seems to be relatively small
- The evaluation is limited, only 1 dataset and only 1 technique evaluated

General advice for work in AI explainability:
When one looks at the problem of AI explainability it is important to describe who the target audience is for the explanation.  Is it a machine learning expert, who wants to debug the model?  Is it an end user who wants to better understand why the prediction was made?  Is it a regulator who is trying to ensure the model's predictions are fair?

Each of these personas will come with different needs and different technical backgrounds, so an assumption that some artifact (a rule set?) is "explainable" may apply to one group, but not the other group.  For example, rules are likely to be more explainable to a ML expert, but may not be to an end user, unless they are very small.



The paper proposes to rewrite each neuron of a neural network as a M-of-N decision rule. An measure of rule complexity (which takes into the account the number of terms in the rule) is proposed, and an approximate rule induction algorithm which binarizes the neurons using an information gain criterion is provided.

The paper gives no evaluation of the accuracy of extracted rules on a test set. Instead, the fidelity to network is evaluated on the train set for individual neurons (see below).

Since even for moderately-sized networks the method would result in lots of rules, the authors propose instead to use the proposed algorithm and complexity criterion as a tool for understanding the complexity of concepts detected by a layer. The results, gathered in Figure 1, suggest that for some layers complex rules do approximate the behavior of neurons, while for other layers a neuron can't be replaced with a single but complex M-of-N rule.

The ideas presented in this paper seem rudimentary and require further exploration before being publishable. First of all, the main result of complexity-vs-layer doesn't differentiate between failures of the approximate rule induction algorithm (the terms in rules are considered for inclusion in a single order, reducing the search space) and the genuine complexity of the rules - this can be verified by evaluating a more exhaustive algorithm on at leas a few neurons (not necessarily on all of them).

Second, if the rules are extracted in a layerwise fashion,  their errors accumulate for deeper layers. However, Algorithm 1 suggests that each neuron is replaced by a rule independently from others, and moreover that it requires the true value of the neurons in the layer below, not of their rule counterparts. This means that the rules can't be combined and explains why the paper doesn't provide any measure for aggregate rule accuracy.

Similarly, the robustness of rules to adversarial examples is meaningless - it seems that applying the rules results in a system which is less accurate (only 8/10 of rules mimick their neurons with no rule complexity penalty) overall, but also makes fewer adversarial examples.

Minor remarks.
Please don't ever produce Figures such as Figure 1: no legend (description in text), color selection is not black and white friendly, font is so small that the axis labels are hard to read... In fact, the poor quality of the Figure by itself should be sufficient  to reject the paper for not abiding to the author guidelines (sec. 4.3: All artwork must be neat, clean, and legible.)

Summary: This paper proposes a novel knowledge extraction method using M-of-N rules
to help interpret hidden features in a Convolutional Neural Network (CNN). While the idea itself is interesting, I think that the paper is still in a very early stage and needs more work before it can be accepted. Detailed comments below.

Pros: 
1. The paper proposes a new algorithm to interpret CNNs.
2. The paper is reasonably well written.

Cons: 
1. The experimental evaluation is quite weak. The authors present their (partial) results on a single dataset and also seem to generalize some of the findings in a rather misleading way. 
2. The proposed method is not compared against any baseline though there are ample rule-based methods to understand NNs in literature.
2. It seems like the literals in the generated rules are actually outputs of previous stages of NNs. Are the rules even human understandable in that case? 

Detailed Comments:
1. I think that the paper is missing a very clear discussion on what is novel about the proposed method in contrast with recent work on explaining NNs (or black box models) using rule based approaches. Examples of relevant papers include "Anchors: High-Precision Model-Agnostic Explanations" by Ribeiro et. al. and "Interpretable & Explorable Approximations of Black Box Models" by Lakkaraju et. al. among others. 
2. Another important piece of discussion that is missing is how the proposed search technique for extracting M-of-N rules is novel compared to a lot of prior literature which deals with the same problem
3. I would strongly encourage the authors to experiment with at least three different datasets and multiple CNN architectures. 
4. I would really like to see the output of the proposed approach. What kinds of rules are being generated at each stage? It seems like the literals in the generated rules are actually outputs of previous stages of NNs. Are the rules even human understandable in that case? 
5. It would be good to do a simple user study to demonstrate that human users are able to understand something useful from the generated explanations. 

Overall, this is a interesting paper on an important topic: knowledge extraction from Neural Networks.
Even though the authors seem propose a novel approach to knowledge extraction, the paper would 
dramatically benefit from two additions:
- an empirical evaluation on at least two more datasets (as is, the paper uses a single dataset)
- an illustrative-but-realistic example of how at least one rule is extracted from each layer of the neural network 

Other comments:
- on page 4 (1st paragraph in 3.3), the authors talk about a "test set" that, it turns out, it is extracted from the actual training set (1st paragraph of 4.1); the authors should use a more careful terminology
- from the paper, it seems that the authors tried a single randomly chosen set 1000 random inputs in 4.1; they should most definitely try several such sets
- Figure 1 should have a legend in the image, rather than as a 2-line caption
 

Meta-review: The presented paper introduces a method to represent neural networks as logical rules of varying complexity, and demonstrate a tradeoff between complexity and error. Reviews yield unanimous reject, with insufficient responses by authors.

Pros:
+ Paper well written

Cons:
- R1 states inadequacy of baselines, which authors do not address.
- R3&4 raise issues about the novelty of the idea.
- R2&4 raise issues on limited scope of evaluation, and asked for additional experiments on at least 2 datasets which authors did not provide.

Area chair notes the similarity of this work to other works on network compression, i.e. compression of bits to represent weights and activations. By converting neurons to logical clauses, this is essentially a similar method. Authors should familiarize themselves with this field and use it as a baseline comparison. i.e.: https://arxiv.org/pdf/1609.07061.pdf 

Decision: rejected-papers