NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks
Authors: Isaac Ahern
Adam Noack
Luis Guzman-Nateras
Dejing Dou
Boyang Li
Jun Huan
Venue:  (2020)
Keywords: Machine Learning
Deep Learning
Interpretability
Feature Importance
Salience

Abstract: The problem of explaining deep learning models, and model predictions generally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model’s behavior. LIME develops multiple  interpretable models, each approximating a large neural network on a small region of the data manifold, and SP-LIME aggregates the local models to form a global interpretation.  Extending this line of research, we propose a simple yet effective method, NormLIME, for aggregating local models into global and class-specific interpretations.  A human user study strongly favored the class-specific interpretations created by NormLIME to other feature importance metrics. Numerical experiments employing Keep And Retrain (KAR) based feature ablation across various baselines (Random, Gradient-based, LIME, SHAP) confirms NormLIME’s effectiveness for recognizing important features.

Under review as a conference paper at ICLR 2020
NORMLIME: A NEW FEATURE IMPORTANCE METRIC FOR EXPLAINING DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of explaining deep learning models, and model predictions gener- ally, has attracted intensive interest recently. Many successful approaches forgo global approximations in order to provide more faithful local interpretations of the model's behavior. LIME develops multiple interpretable models, each ap- proximating large neural network on small region of the data manifold, and SP-LIME aggregates the local models to form a global interpretation. Extend- ing this line of research, we propose a simple yet effective method, NormLIME, for aggregating local models into global and class-specific interpretations. A hu- man user study strongly favored class-specific interpretations created by Norm- LIME to other feature importance metrics. Numerical experiments employing Keep And Retrain (KAR) based feature ablation across various baselines (Ran- dom, Gradient-basee based, LIME, SHAP) confirms NormLIMES effectiveness for rec- ognizing important features.
INTRODUCTION
As the applications of deep neural networks continue to expand, the intrinsic black-box nature of neural networks creates potential trust issue. For application domains with high cost of prediction error, such as healthcare |Phan et al.) (2017). it is necessary that human users can verify that model
learns reasonable representation of data and the rationale for its decisions are justifiable according
to societal norms (Koh & Liang 2017 Fong & Vedaldi 2018 Zhou et al. 2018 Lipton 2016 Langley 2019).
An interpretable model, such as linear sparse regression, lends itself readily to model explanation. Yet due to limited capacity, these interpretable models cannot approximate the behavior of neural networks globally. A natural solution, utilized by LIME Ribeiro et al. (016)) is to develop multiple interpretable models, each approximating the large neural network locally on a small region of the data manifold. Global explanations can be obtained by extracting common explanations from multiple local approximations. However, how to best combine local approximations remains an open problem.
Extending this line of research, we propose a novel and simple feature scoring metric, Norm which estimates the importance of features based on local model explanations. In this paper, we empirically verify the new metric using two complementary tests. First, we examine if the Norm- LIME explanations agree with human intuition. approach over three baselines (LIME, SmoothGrad and VarGrad) with NormLIME receiving 30% more votes than all the baselines combined. Second, we numerically examine if explanations created by Norm accurately capture characteristies of the machine learning problem at hand, using the same intuition proposed by |Hooeer al. (2018). Empirical results indicate that NormL.ME iden- tifies features vital to the classification performance more accurately than several existing methods. In summary, we find strong empirical support for our claim that NormLIME provides accurate and lumnrtaaddest explanations for deep neural networks.
user study, participants favored the proposed
The paper makes the following contributions:
We propose a simple yet effective extension of LIME, called NormLIME, for aggregating interpretations around local regions the data manifold create global and class-specific

---

Under review as a conference paper at ICLR 2020
Figure 1: Explanations for the MNIST digits 9 and 3. generated by: VarGrad (Adebayo et al. 2018 SmoothGrad Smilkov et al. 2017). LIME (Ribeiro et al. 2016), and NormLIME (ours).
interpretations. NormLIME outperforms LIME and other baselines in two complementary evaluations.
We show how feature importance from LIME can be aggregated create class-specific in- terpretations, which stands between the fine- ine-grained interpretation at the level of data points and the global interpretation at the level of entire datasets, enabling a hierarchical under- standing of machine learning models. The user study indicates that Norml IIM excels at this level of interpretation.
RELATED WORK
machine learning model can be interpreted from the perspective of how much each input feature contributes to given prediction. In computer vision, this type of interpretation is often referred to as saliency maps or attribution maps. A number of interpretation techniques, such as Smooth- Grad (Smilkov et al. 2017). VarGrad (Adebayo et al., 2018). Integrated Gradients (Sundararajan jet al. 2017). Guided Backpropagation (Springenberg et al. 2013). Guided GradCAM (Selvaraju al. 2017), and Deep Taylor Expansion (Montavon al. 2017). exploit gradient information, as
it provides a first-order approximation of the input's influence on the output (Simonyan et al.) [013)] Ancona al.] (0118). [Seo al. (2018) analyze the theoretical properties of SmoothGrad and Var- Grad. When the gradient is not easy to compute, [Bachrens et al. (2010) places Parzen windows around data points to approximate Bayes classifier, from which gradients can be derived. DeepLift (Shrikumar et al. 2017) provides gradient-fie free method for saliency maps. Though gradient- based techniques can interpret individual decisions, aggregating individual interpretations for global un- derstanding of the model remains a challenge.
Local interpretations are beneficial when the user interested understanding particular model decision. They become less useful when the user wants high-level overview of the model's be- havior. This necessitates the creation of global interpretations. LIME (Ribeiro et al. 2016) first builds multiple sparse linear models that approximate complex model around small regions on the data manifold. The weights the linear models can then be aggregated, and compared, in order to construct global explanation using Submodular Pick LIME (SP-LIME). Ribeiro al. (2018) introduce anchor rules to capture interaction between features. Tan et al. (2018) approximate the complex model using fluence of individual features and their high-order interactions. The proposed NormLIME technique paradigm. We directly modify the normalization for aggregated
of interpretable functions, such trees or splines, which capture the in-
fits into this "neighborhood-basee weights, rather than the function forms (such as rules or splines).
Ibrahim al. (201)) ranks feature importance and clusters data points based on their ranking corre- lation. The interpretations for cluster medoids are used in the place of single global interpretation. Instead of identifying clusters, in this paper, we generate interpretations for each class given dataset. The class-level interpretation provides an intermediate representation so that users can grasp behaviors of machine learning models at different levels of granularity.
Proper evaluation of saliency maps can be challenging. AAebayo et al. (2018) show that, although some techniques produce visually reasonable saliency maps, such maps may not faithfully reflect the behavior of the underlying model. Thus, visual inspection by itself is not reliable evaluation criterion. Kindermans et al. (2017) adopt linear classifiers as a sanity check. Feng & Graberr (2019) propose cooperative games, where humans can see the interpretation of their Al teammate, as a benchmark. [oooker et al. (2018) propose ablative benchmarks for the evaluation of feature importance maps. When features that are considered important are removed from the input, the
2

---

Under review as a conference paper at ICLR 2020
Explonation visualizaton
08 131
0
(15
(10 10)
(12 231
(8
06,00
[16
(1..1))
Figure 2: A LIME-based Local Explanation using the input pixels as features. On the right, green and red indicate pixels whose presence/absence offers support for the class label "3". On the left, we show the 20 pixels with the largest weights.
model should experience large drops in performance. The opposite should happen when features deemed unimportant removed. In this paper, evaluate the proposed technique using the Keep-and-Retrain (KAR) criterion from Hooker et al.)
3 METHODOLOGY
As method for explaining deep models, LIME (Ribeiro et al. 2016) first builds interpretable mod- els where each approximates the complex model around locality on the data manifold. After that, the local explanations are aggregated using SP-LIME. In this work, extend the general paradigm of LIME with a new method, which we call NormLIME, for aggregating the local explanations.
3.1 BUILDING LOCAL EXPLANATIONS
LIME constructs local interpretable models to approximate the behavior of a large, complex model within a locality on the data distribution. This process can be analogized with understanding how hypersurface fzz changes around To by examining the tangent hyperplane V fx(o).
Formally, for a given model f X' + V, we may learn an interpretable model g. which is local to the region around particular input a Gaussian probability distribution Tay centered around Zg. Repeatedly drawing æ' from Tao and applying f(.) yield new dataset X' {(w', f(x')), We then learn sparse linear regression g(x', 2o) = w.,1. x' using the local dataset X' by optimizing the following loss function with 22-)) measure of complexity.
X. To do this, we first sample from our dataset according
argmin C(S, g. ) l(Wmo)
(1)
where L(f.g, Tw.) is the squared loss weighted by Taa
CU mno))
= E wlorkmo
(s(x') - (f(x')-g(a)
g('', mo0))
(2)
For ((wn.), we impose an upper limit K for the number of non-zero components in Way- so that 2 (( oc 1 (1 llo > K). The optimization is intractable, but we approximate it by first selecting K features with LASSO regression and performing regression on only the top K features.
This procedure yields z()) which approximates the complex model f(x) around zp- The components the weight vector Wn, indicate the relative influence of the individual features of in the sample X' and serve the local explanation of f(x).. Figure 2 illustrates such local explanation.
3.2 NORMLIME
After a number local explanations have been constructed. we aim to create a global explanation. NMrmLIME
IME is method for aggregating and normalizing multiple local explanations and estimating
3

---

Under review as a conference paper at ICLR 2020
the global relative importance of all features utilized by the model. NormLIME gives more holistic explanation model than the local approximations of LIME.
We let e; denote the th feature, the 7th component in the feature vector Since the local explana- tion weights are sparse, not all local explanations utilize q₁. We denote the set of local explanations that do utilize c; ((ss), which is set of weight vectors Wxy computed at different locales æ.j In other words, for all w E E(,,), the corresponding weight component W; * O.
The NormLIME "importance score" of the feature c;, denoted by S(c;), is defined the weighted average of the absolute values of the corresponding weight wi Ve E E(ca.
S(c)) == E(G))
Σ
(We,, i) [aage
(3)
Way (k()))
where the weights are computed as follows.
Wzy.
Ww
(W,,, Σx W2y,
(4)
|way
Here, 7(W,,, i) represents the relative importance of the feature oi in the local model built around the data point.ry. Ty.. If a feature o; is not utilized in any local models, set its importance S(c,) to O.
We now introduce a slightly different perspective NormLIME, which helps us understand the dif- ference between this approach and the aggregation and feature importance approach used in LIME. Consider the global feature weight matrix M, whose rows are the local explanation w computed at different locales. Let w; be the 7thh column of the matrix, which contains the weights for the same feature in different local explanations. Let be the vector representing the L1 norms of the rows of M. We can express the NormLIME global feature importance function as
1
S(c))
|wo:|| w. diag(v" )w,,
where diag(e-)) denotes a matrix with w on the diagonal and zero everywhere else. || llo the LO norm, or the number of non-zero elements.
In comparison, the submodular pick method (SP-LIME) by Ribeiro et (0016) employs the 1.2 norm
S (c) = |
(6)
to measure the importance of G. Contrasting Eq. equation 5 and equation 6 it is apparent that the difference between the two methods lies in in the normalization of the column weights.
3.3 CLASS-SPECIFIC INTERPRETATIONS
As discussed above, NormLIME estimates the overall relative importance assigned to feature c; by the model. For binary classification problems, this is equivalent to a representation of the impor- tance the model assigns to the feature e; in distinguishing between the two classes, i.e., recognizing a class label. In multi-class problems, however, this semantic meaning is lost as the salience compu- tation above does not distinguish between classes, and class-relevant information becomes muddled together.
straightforward recover the salience information associated with individual class labels, by partitioning E(c;) based on the class label of the initial point zy that the local approximation is built around. The partition E,, (c;) contains the local explanation Way and only f() assigns the label y to ry. More formally,
(y( = {wi, E F(G) ((x,) = y)-
It is easy to see that if (c) # Ø, Vy, and f(x) is a single-label classification problem, then the family of sets Ey (c:) forms partition of E(c:):
E(c,) E,,C)) (g)(c) Egg (c:) Vyy # yk.
(8)
4

---

Under review as a conference paper at ICLR 2020
Computing salience of for given label is performed via
(y(c)) = (c)
r(Wny; i) Juanssl
Way (eg))))
Compared to global interpretations, the class-specific salience S (c:) yields higher resolution infor- mation about how the complex model differentiates between classes. We use (y(c;) as prediction- independent class-level explanations in the human evaluation, described in the next section.
4 HUMAN EVALUATION
In order to put the interpretations generated by NormLIME to test, we administered human user study across Amazon Mechanical Turk users. We compare class-specific explanations generated by standard feature importance functions and the proposed salience method on the MNIST dataset.
Consider Figure
From the following set of four images, please select the image that best highlights the pixels that distingulsh the number from the other nine possible digits.
Top: an example question from the user study.
Libe:
Label
Bottom: an example attention check from the user study. The correct choice is option 2. Op- tions and 4 are duplicates and cannot distinguish well between labels "3" and "8".
following set of images, please select image highlights pixels distinguish number from other nine possible digits.
Labet
Lbbi
Labet
4.1 SALIENCY MAP BASELINES
To avoid showing many op- tions to the human participants, which may cause cognitive over- load, we selected few base- line techniques that we consider to be the most promising for saliency maps (Seo et al. (2088). We select SmoothGrad and Var- Grad because they aim to reduce noise in the gradient, which should facilitate the aggregation of individual decisions to form class- level interpretation. The aggregation of these individual interpretations are performed by taking the mean importance
Figure 3: Example survey questions.
over interpretations from sample of datapoints corresponding each label. The details are discussed below.
4.1.1 SMOOTHGRAD
This technique (Smikov et al. 2017) generates more "interpretable" salience map by averaging out local noise typically present in gradient interpretations. We add random noise ~ (O, o²) to the input Here follow the default implementation using the "SmoothGrad Squared" formula, which is an expectation over yp:
0f + m) Ock
SGSQ (x)
noted in Hooker al. (2018) In practice, approximate the expectation with the average of 100 samples of 77 drawn from (0, o²) where a is 0.3. The class-level interpretation computed the average of the saliency maps for 10 images randomly sampled from the target class.
5

---

Under review as a conference paper at ICLR 2020
4.1.2 VARGRAD
Similar to SmoothGrad, VarGrad (Adebayo et al. 2018) uses local sampling of random noise ry ~ (O to reduce the noise of the standard gradient map interpretation. VarGrad perturbs an input x randomly via 7), and then computes the component wise variance of the gradient over the sample
(v) Var( 0f(z 7)
0ck
Similar to SmoothGrad, we compute the variance using 100 samples of r) from a normal distribution with zero mean and standard deviation of 0.3. We use the average saliency map over 10 randomly sample images the desired class as the class-level interpretation.
4.1.3 LIME
We compute importance (x) as in Eq. equation 6 but conditioned on the label y to capture feature that were positively correlated with the specific label: (SP(x (x. 1z)0)
For both LIME and NormLIME, only show the input features that are positively correlated with the prediction. That is, when JSP (SP(c)) (c:) or S(c,) is positive. The purpose is to simplify the instructions
given to human participants and avoid confusion, since most participants may not have sufficient background in machine learning.
4.2 EXPERIMENTAL DESIGN
The design of the study as follows: We administered a questionnaire featuring 25 questions, each containing four label-specific explanations for the same digit. We were able to restrict participants through Mechanical Turk users who had verified graduate degrees. Survey takers were instructed to evaluate the different explanations based on how well they captured the important characteristics of each digit in the model's representation. To account for response ordering bias, the order of the methods presented for each question was randomized. In order to catch participants who cheat by making random choices, we included 5 attention checks with a single acceptable answer that is relatively obvious. We only include responses that pass least 4 of the 5 attention checks.
We conducted the experiment on MNIST with 28 x 28 single-channel images. We trained a 5-layer convolutional network that achieves 99.05% test accuracy. This model consisted of two blocks of convolution plus max- max-pooling operations, followed by three fully connected layers with dropout in- between, and final softmax operation. The number of hidden units for the three layers 128, 128, and 10, respectively. Class-specific explanations were generated for the digits from to 9.
It is important to note that none of the explanations generated for the study represented particular prediction on a particular image, but instead represented how well the importance functions captured the important features for label (digit) in the dataset.
4.3 RESULTS AND DISCUSSION
After filtering responses that failed the attention check, we ended with 83 completed surveys. From their responses, the number of votes for each method were: 939 for NormLIME, 438 for LIME, 151 for VarGrad, and 132 for SmoothGrad. We analyzed the data by examining each user's response as a single sample of the relative proportions of the various explanation methods for that user, and performed standard one-way ANOVA test against the hypothesis that the explanations were preferred uniformly. We obtained a statistically significant result, with an statistic of 338 and p-value of 5.22 x 10-100, allowing us to reject the null hypothesis. We conclude that significant difference exist between how users perceived the explanations.
A subsequent Tukey HSD post hoc test confirms that the differences between NormLIME and all other methods are highly statistically significant (p < 0.001). also shows that the difference interpretations is statistically significant
between LIME and the gradient-basse conclude that overall, the NormLIME explanations were preferred over all other baseline techniques, including LIME and that NormLIME and LIME were preferred over the gradient- based methods.
0.001). We
Observing Figure I the interpretations of SmoothGrad and VarGrad do not appear to resemble anything semantically meaningful. This may be attributed to the fact that these methods are not

---

Under review as a conference paper at ICLR 2020
designed with class-level interpretation in mind. LIME captures the shape of the digits to some extent but cannot differentiate the most important pixels. In contrast, the normalization factor in NormLIME helps to illuminate the differences among the important pixels, resulting in easy-to-reas interpretations. This suggests that proper normalization is important for class-level interpretations.
NUMERICAL EVALUATION WITH KAR
Visual inspection alone may not be sufficient for evaluating saliency maps (Adebayo et al. 2018 In this section, we further evaluate NormLIME using technique akin to Keep And Retrain (KAR) proposed by Hooker et al. (2018). The underlying intuition KAR is that features with low impor- tance are less relevant to the problem under consideration. This gives rise a principled "hypothesis of least damage": removal of the least important features as ranked by a feature importance method should impact the model's performance minimally. Thus, can compare two measures of feature importance by comparing the predictive performance after removing the same number of features as ranked by each method the least important.
Specifically, we first train the same convolutional network as in the human evaluation with all input features and use one the importance scoring method to rank the features. We remove a number of least important features and retrain the model. Retraining is necessary as we want to measure the importance of the removed features to prediction rather than how much one trained model relies on the removed features. After that, measure the performance drop caused by feature removal; smaller performance drop indicates more accurate feature ranking from the interpretation method.
We perform KAR evaluation on two set of features. The first set is the raw pixels from the images. The second set of features are the output from the second convolutional layers of the network. The baselines and results discussed below.
5.1 BASELINES
We evaluated NormLIME against various baseline interpretation techniqes on the MNIST dataset [LeCun 1998). For NormLIME and LIME, use the absolute value of I(c:) as the feature impor- lance. In addition to those used in the human evaluation, we introduce the following baselines.
5.1.1 SHAP
The Shapley value measures the importance of a feature by enumerating all possible combinations of features and compute the average of performance drop when the feature is removed. The value well suited to situations with heavy interactions between the features. While theoretically appeal- ing, the computation is intractable. A number of techniques (Chen et al., 2019: Lundberg & Lee, 2017 Strumbelj Kononenko 2010) have been proposed to approximate the Shapley value. Here SHapley Additive exPlanations (SHAP), approximation based sampled least-squares [uuddberg & Lee 2017).
5.1.2 RANDOM
This baseline randomly assigns feature importance. This serves as a "sanity check" baseline. No- table, in the experiments Hooker et al. (2018), some commonly used saliency maps perform worse than random.
5.2 RESULTS AND DISCUSSION
Figure (a) shows the error gained after number of least important features are removed, averaged over independent runs. We use removal thresholds from 10% to 90%. When 50% of the features or less are removed, NormLIME performs better or similarly with the best baselines, though it picks up more error when more features are removed. The best prediction accuracy among all is achieved by NormLIME at 50% feature reduction with 0% error gain. This is matched by SHAP also 50% feature reduction and VarGrad at 20% feature reduction. All other basclines observe about at least 0.25% error gain 50% feature reduction,. SHAP and LIME perform better than other methods,

---

Under review as a conference paper at ICLR 2020
Random
NomLLIE 00%%)
SmoothGred NermLIME
2
(99.
21
Prediction Errer
Protfction
6.
50
(a)
(b)
Figure 4: Prediction errors on MNIST the least important features are gradually removed from: (a) the input. (b) the output of the last convolutional layers. The horizontal axis indicate the percentage of of features removed. The vertical axis shows absolute error in log scale.
including NormLIME, when 60% outperformed by NormLIME and LIME.
features removed. The gradient- maadeent-based methods,
Figure Suuneesshow shows the same measure the convolutional features. On these features, NormLIME out- performs the other methods by larger margins, compared to the input features. NormLIME achieves better results than the original model (at 99.05%) when 70% or less features removed, underscor- ing the effectiveness of dimensionality reduction. The best performance achieved at 30% removal with a classification accuracy of 99.31% %. The second best achieved by LIME at 99. 19.1% accuracy when filtering 40% of features, comparable with NormLIME performance at the same level.
When 80% of features removed, NormLIME demonstrates zero error gain, whereas the second best method, LIME, gains 0.3% absolute error. When 90% of features are removed, NormLIMe shows 0.45% error gain while LIME observes .6% error gain and all others receive at least 1.25%.
Overall, gradient ensemble methods SmoothGrad and VarGrad outperformed Random but compared unfavorably with "additive local model approximation" schemes (SHAP, LIME, and NormLIME:
The advantage of NormLIME is more pronounced when pruning the convolutional features than input features. Further, we can achieve better performance removing convolutional features, but the input features. This suggests that there is more redundancy in the convolutional features and NormL able to exploit that phenomenon.
CONCLUSIONS
Proper interpretation of deep neural networks crucial for state-of-the-art Al technologies gain the public's ' trust. In this paper, we propose new metric for feature importance, named NormLIME, that helps human users understand black-box machine learning model.
We extend the LIME / SP-LIME technique [Ribeiro et al. 2016), which generates local explanations of large neural network and aggregates them to form global explanation. NormLIME adds proper normalization to the computation of global weights for features. In addition, we propose label-based NormLIME, which provides finer-grained interpretation
multi-class setting compared to SP- LIME which focuses selecting an optimal selection of individual predictions to explain model.
Experimental results demonstrate that the NormLIME explanations agree with human intuition. The human evaluation study shows that explanations generated by NormLIME are strongly favored over comparable ones generated by LIME, SmoothGrad, and VarGrad with strong statistical significance. show that explanations formed by the NormL
Further, using Keep-And- Retrain evaluation, metric are faithful to the problem at hand, as it identifies input features and convolutional features whose removal is not only harmless, but may even improve prediction accuracy.
8

---

Under review as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
Intentionally omitted anonymize the manuscript.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems 31. 2018.
Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. Towards better understanding of gradient-bart based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11:1803-1831, 2010.
Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. In International Conference on Learning Rep- resentations, 2019.
Shi Feng and Jordan Boyd-Graber. What can Al do for me? evaluating machine learning inter- pretations in cooperative play. In 24th International Conference on Intelligent User Interfaces, 2019.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2018.
Sara Hooker, Dumitru Erhan, Pieter jan Kindermans, and Been Kim. Evaluating feature importance estimates. arXiv, 2018.
10758. .ddf]
Mark Ibrahim, Melissa Louie, Ceena Modarres, and John Paisley. Global explanations of neural networks:
Mapping the landscape of predictions. In Proceedings of the 2019 AAAI/ACM Con- ference AI, Ethics, and Society, AIES '19, pp. 279-287, New York, NY, USA, 2019. ACM. ISBN 978- 778-1-4503-63244
doi: (0.1145/3306618.33114444 10. N0.1145/3306611 3314230
URL htt.h:ttp//dd acm. acm.org/ org/
Pieter-Jan Kindermans, Kristof T. Schtt, Klaus-Robert Miler Maximilian Alber, Dumitru Erhan, Been Kim, and Sven Dahne. Learning how to explain neural networks: Patternnet and patternat- tribution. 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, pp. 1885-1894, 2017.
Pat Langley. Explainable, normative, and justified agency. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, 2019.
Yann LeCun. MNIST dataset, 1998. data retrieved from Yann Lecun's online archive, http:
//yann. www.n.eenn.. com/ ccom/exdb/// mnnp//niss. st/
Zachary C. Lipton. The mythos of model interpretability. In 2016 ICML Workshop on Human
Interpretability in Machine Learning, 2016.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30, pp. 4765-4774. 2017.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Rober Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recogn., 65(C):21 -22,, May 2017. ISSN 0031-3203, doi: 10. 10.t001ii. 2atcog URL
https::/doii. wwr.ooo..orgg// 100...11600 1016/1000000001 1011/1.aatcoooo.g...o.eoo 2016
008

---

Under review as a conference paper at ICLR 2020
Nhathai Phan, Dejing Dou, Hao Wang, David Kil, and Brigitte Piniewski. Ontology- basedd deep learning for human behavior prediction with explanations in health social networks. Information Sciences, 384:298 313, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Knowledge Discovery and Data Mining (KDD), 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: explanations. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
model-agnostic
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
Junghoon Sco, Jeongyeol Choe, Jamyoung Koo, Seunghyeon Jeon, Bcomsu Kim, and Taegyun Jeon. Noise-adding methods of saliency map series of higher order partial derivative. ICML Workshop, 2018. WtLhttts: www.iv..vv....//aia org/ http:///////// 1806. 03000. pdf
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning, pp. 3 3145-3153, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: networs.. Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.6033,, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vigas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. ICML Workshop, 2017. URL htttss://axiv.
abs/1700 03825
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. International Conference on Learning Representations, 2015.
Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, pp. 1-18, March 2010.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017.
Sarah Tan, Rich Caruana, Giles Hooker, Paul Koch, and Albert Gordo. Learning global additive explanations for neural nets using model distillation. ArXiv Preprint. ArXiv 1801.08640, 2018.
Bolei Zhou, Yiyou Sun, David Bau,, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. arXiv preprint. ArXiv: 02891, 2018.
10


Reviews:
[{"review_id": "rJxzAY-a9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for DNN interpretability based on LIME, itself based on ensembling of large numbers of low-complexity models called \"local explanation models\". This method allows to better capture the relative importance of each feature, and is also able to recover the class-specific signals that DNNs use to distinguish between a number of classes. \n\nScore: Weak Accept\nWhile this method seems like a good step forward, I am uncertain about its significance, mostly because of the choice of dataset and because it appears to be a minor change to a well known algorithm.\n\nI think some high-level aspects could be better motivated:\n- Why is interpreting black-box models a better avenue than building interpretable models?\n- Why should we expect this method to work beyond MNIST? Will it work on large images where translational invariance is a vital aspect? Will it work on non-image data?\n- Why does this relatively subtle change from LIME create such a dramatic visual difference? Do the authors have a stronger intuition than what is hinted at in the paper?\n\nOther comments:\n- The human evaluation is very interesting, and suggests that this method correlates much better with human judgement than previous ones. One key aspect that is missing, but hinted at, is where this difference comes from. It could be from the proposed change or from the class-specific information. (It should be easy to run KAR to test this?)\n- It's not always clear how this should be reproduced. For example, the authors specify the number of samples for SmoothGrad and VarGrad but not LIME nor NormLIME.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "Hygc2fIhtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Months ago, I read this article on arxiv (https://arxiv.org/pdf/1909.04200.pdf). It is an interesting work that tries to propose a simple yet effective interpretable model. I am not familiar with this research direction, and I try to make an educated guess.\n\nPros:\n-- As the author suggested, the method is simple and effective.\n-- The authors conducted user studies to demonstrate that the results generated by their proposed method is strongly favored over previous methods.\n\nCons:\n-- Subscripts in equations need improvement to make them consistent. For example, in Equation (7), we have E_{y}, but in Equation (8), we have E_{Y_j} and E_{Y_k}.\n-- Section 4, Figure 3, top, it seems obvious to choose the fourth one to distinguish the number 9? I feel this example is too easy and not convincing enough.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByxujsOoFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a novel scheme to aggregate local explanations\nfrom LIME to generate global and/or class-level interpretations in the\nform of feature importances. The technical novelty lies in the\nnormalization of the feature weights across local explanations where\nthe proposed scheme utilizes a well-motivated L0 normalization in\nplace of the existing L2 normalization in Ribeiro et al., 2016. The\nsparsity induced by the L0 normalization leads to more \"easy-to-read\"\nglobal (or class level) \"explanations\" since the number of features\nwith non-zero importances is reduced.\n\n\nThe simple idea of generating sparse aggregations of local\nexplanations to obtain global (or class-level) interpretations is well\nmotivated and seems fairly intuitive (and I consider simplicity a\nstrength) and the extremely favourable empirical results, I am leaning\ntowards a reject for the current version of the paper. The main\nreasons for this decision are as follows: \n\n- The proposed scheme is presented as closely tied to LIME, which\n  severely limits the scope of the proposed solution. However, I can\n  imagine that the high level idea of sparse aggregation of local\n  explanations can easily generalize to many local explanation\n  schemes. \n- The empirical evaluation needs to be improved (or better clarified)\n  in my opinion. There are many potential baselines that are also not\n  considered (see details below). Moreover, the evaluation is\n  completely limited to a single data set (which is good as a strawman\n  but not sufficient to make significant claims).\n\n\nClarifications re: baselines and empirical evaluations:\n\n- No comparison to Ibrahim, et al., 2019 -- you can cluster in\n  unsupervised setting but you can also aggregate within class in\n  supervised setting. Generating class-level interpretations is an\n  interesting idea but its novelty is somewhat unclear.\n- In terms of explanations, there are contrastive explanations [1]\n  which are not really discussed in this paper. This scheme could be\n  more useful for generating class level interpretations.\n- It is not clear why the negative correlations were removed in the\n  user study. In certain situations, such as when distinguishing\n  between digits 3 & 8, the absence of weights (potentially\n  corresponding to negative weights) can be crucial explanations.\n- It is not clear why in the SmoothGrad and VarGrad baselines, the\n  class level interpretations are generated with just 10 images per\n  class. Are the class level interpretations of LIME and NormLIME also\n  generated from just 10 images per class. If that is the case, 10\n  seems like a very small number and can potentially create noisy\n  explanations. If LIME and NormLIME used more than 10 images to\n  generate class level interpretations, the comparison does not seem\n  fair. Please explain this discrepancy.\n- For the evaluation in Sec. 5, it is not clear how the features are\n  removed. Are they removed based on the global explanations (feature\n  importances) of each baseline? If this is the case, are the global\n  explanations of each baseline generated using the same amount of\n  local explanations? Please clarify.\n- The KAR analysis results in Fig. 4(b) need to be further\n  investigated. First, it is not clear if this difference between the\n  different baselines something that happens in multiple datasets or\n  just this one. Moreover, it seems natural that something like\n  SmoothGrad and VarGrad would be able to capture the feature\n  redundancies if the redundancies were the actual cause for this\n  marked difference between the relative performances in Fig. 4(a) and\n  4(b). This could definitely use a better discussion.\n\n\n[1] Dhurandhar, Amit, et al. \"Explanations based on the missing:\nTowards contrastive explanations with pertinent negatives.\" Advances\nin Neural Information Processing Systems. 2018. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to extract the set of features explaining a class, from a trained DNN classifier.

The proposed approach relies on LIME (Ribeiro et al. 2016), modified as follows: i) around a point x, a linearized sparse approximation of the classifier is found (as in LIME); ii) for a given class, the importance of a feature aggregates the relative absolute weight of this feature in the linearized sparse approximations above; iii) the explanation is made of the top features in terms of importance.

This simple modification yields visual explanations that significantly better match the human perception than the SOTA competitors. 

The experimental setting based on the human evaluation via a Mechanical Turk setting is the second contribution of the approach. The feature importance measure is also assessed along a Keep and Retrain mechanism, showing that the approach selects actually relevant features in terms of prediction. 
Incidentally, it would be good to see the sensitivity of the method to parameter $k$ (in Eq. 1).

As noted by Rev#1, NormLIME is simple (and simplicity is a strength) and it demonstrates its effectiveness on the MNIST data. However, as noted by Rev#4, it is hard to assess the significance of the approach from this only dataset. 

It is understood that the Mechanical Turk-based assessment can only be used with a sufficiently simple problem.  However, complementary experiments on ImageNet for instance, e.g., showing which pixels are retained to classify an image as a husky dog, would be much appreciated to confirm the merits and investigate the limitations of the approach.

Decision: Reject