Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
Authors: Laura Rieger
Chandan Singh
W. James Murdoch
Bin Yu
Venue:  (2020)
Keywords: explainability
deep learning
interpretability
computer vision

Abstract: For an explanation of a deep learning model to be effective, it must provide both insight into a model and suggest a corresponding action in order to achieve some objective.  Too often, the litany of proposed explainable deep learning methods stop  at  the  first  step,  providing  practitioners  with  insight  into  a  model,  but  no way to act on it.  In this paper, we propose contextual decomposition explanation penalization (CDEP), a method which enables practitioners to leverage existing explanation methods in order to increase the predictive accuracy of deep learning models.  In particular, when shown that a model has incorrectly assigned importance to some features, CDEP enables practitioners to correct these errors by directly regularizing the provided explanations.  Using explanations provided by contextual decomposition (CD) (Murdoch et al., 2018), we demonstrate the ability of our method to increase performance on an array of toy and real datasets.

Under review as a conference paper at ICLR 2020
INTERPRETATIONS ARE USEFUL: PENALIZING EXPLA- NATIONS TO ALIGN NEURAL NETWORKS WITH PRIOR KNOWLEDGE
Anonymous authors Paper under double-blind review
ABSTRACT
For an explanation deep learning model be effective, must provide both insight into model and suggest corresponding action in order to achieve some objective. Too often, the litany of proposed explainable deep learning methods stop at the first step, providing practitioners with insight into a model, but no way to act on it. this paper, we propose contextual decomposition explanation penalization (CDEP), method which enables practitioners leverage existing explanation methods in order to increase the predictive accuracy of deep learn- ing models. In particular, when shown that model has incorrectly assigned im- portance to some features, CDEP enables practitioners to correct these errors by directly regularizing the provided explanations. Using explanations provided by contextual decomposition (CD) (Murdoch et al.] 2018), we demonstrate the ability of our method increase performance on an array toy and real datasets.
INTRODUCTION
In recent years, neural networks have demonstrated strong predictive performance across wide variety of settings. However, in order to achieve that accuracy, they sometimes latch onto spurious correlations, leading to undesirable behavior as a result of dataset bias (Winkler et al.) 2019), racial and ethnic stereotypes (Garg al. 2018), simply overfitting. While recent work into explaining neural network predictions (Murdoch of al. 2019 Doshi Velez & Kim 2017) has demonstrated an ability to uncover the relationships learned by model, it is still unclear how to actually alter the model order to remove incorrect, or undesirable, relationships.
We introduce contextual decomposition explanation penalization (CDEP), method which lever- ages existing explanation techniques for neural networks in order to prevent a model from learning unwanted relationships and ultimately improve predictive accuracy.
Given particular importance scores, CDEP works by allowing the user to directly penalize impor- tances of certain features, interactions. This forces the neural network to not only produce the correct prediction, but also the correct explanation for that prediction. While we focus on contextual decomposition (CD) (Murdoch al. 2018, (inghh et al. 2018), which allows the penalization of both feature importances and interactions, CDEP can be readily adapted for existing interpretation techniques, as long as they are differentiable. Moreover, CDEP is general technique, which can be applied to arbitrary neural network architectures, and is orders of magnitude faster and more memory efficient than recent gradient-basel methods, allowing its use on meaningful datasets.
In order to demonstrate the effectiveness of CDEP, we conducted experiments across a wide variety of tasks. the prediction of skin cancer from images, CDEP improves the prediction by teaching it to ignore spurious confounding variables present in the training data. In a colored MNIST task, CDEP allows the network to focus on a digit's 's shape rather than its color (with no extra human annotation needed). Finally, toy example using text classification shows how the penalization can help a network avoid bias towards particular words, such as those involving gender.
classifier

---

Under review as a conference paper at ICLR 2020
Prediction
Label
Desired Explanation
Explanation
Figure 1: CDEP allows a practitioner to penalize both a model's prediction and the corresponding explanation.
2 BACKGROUND
Explanation methods Many methods have been developed to help explain the learned relation- ships contained DNN. For local, prediction-level explanation, most prior work has focused on assigning importance to individual features, such as pixels in an image or words in a document. There are several methods that give feature-level importance for different architectures. They can be categorized as crudient-based (Springenberg et al. 20044; (Sundararaaan et al.) [017]] Selvaraju Rieger & Hansen 2019) , decompositiion-aasee (Murdoch &
al.
2016 Baehhrns et al.
2010
Szlam 2017 (Shrikumar et al.
2016
Bach et al. 2015 and others (Dabkowski & Gal 2017; Fong
Vedaldi|[201 Ribeiro et al.
2016 Zintgraf et 2011)) with many similarities among the meth-
ods (Ancona et al. 2018 Lundberg & Lee 2017). However, many of these methods have thus far been poorly evaluated (Adebayo et 20188 Nie et al. 2018), casting doubt on their usefulness. An- other line of work, which we build upon, has focused uncovering interactions between features, in addition to feature importances, (Murdoch et al.] (018). and using those interactions to create a hierarchy of features displaying the model's prediction process (Singh al. 2018).
Uses of explanation methods While much work has been put into developing methods for ex- plaining DNNs, relatively little work has explored the potential to use these explanations help build a better model. Some recent work proposes forcing models to attend to regions of the input which are known to be important (Burns et al. 2018 Mitsuhara et al.) (2019), although it is important to note that attention is often not the same as explanation (Jain & Wallace 2019). An alternative binary
line of work proposes penalizing the gradients of neural network match numan-providee annotations and shows the possibility to improve performance (Ross et al. 2017) and adversarial robustness (Ross Doshi Velei 2018). Two recent papers extend these ideas by penalizing attri- butions for natural language models (Liu & Avei 2019) and penalizing a modified gradient-base -aseed score produce smooth attributions (Erion et al. 2019 Predating deep learning, (Zaidan et al.) 2007) consider the use of "annotator rationales" " in sentiment analysis to train support vector ma- chines. This work on annotator rationales was recently extended to show improved explanations (not accuracy) in CNNs (Strout et al. 2019).
Other ways constrain DNNs While focus on the use of explanations to constrain the rela- tionships learned by neural networks, other approaches for constraining neural networks have also been proposed. A computationally intensive alternative to augment the dataset in order to prevent the model from learning undesirable relationships. through domain knowledge (Bolukbasi et al.) 2016), projecting out superficial statistics (Wang et al. 2019) or dramatically altering training im- ages (Geirhos et al., 2018). However, these processes computational cost or the difficulty constructing such an augmented data set. Adversarial training has also been explored (Zhang & Zhu 2019). These techniques often tied to particular datasets, and do not provide a clear link berween learning about a model's learned relationships through explanations, and subsequently correcting them.
often not feasible, either due to their
generally limited, they
3 METHODS
We now introduce CDEP, which penalizes the explanations of a neural network in order to align with prior knowledge about why model should make prediction. To do so, for each data point
2

---

Under review as a conference paper at ICLR 2020
it penalizes the CD scores of features, or groups of features, which user does want the model to learn to be important. While we focus on CD scores, which allow the penalization of interac- tions between features in addition to features themselves, this approach readily generalizes to other interpretation techniques, so long as they are differentiable.
3.1 AUGMENTING THE LOSS FUNCTION
Given particular classification task, want teach model to not only produce the correct prediction, but also to arrive at the prediction for the correct reasons. That is, we want the model to be right for the right reasons, where the right reasons are provided by the user and are dataset- dependent.
To accomplish this, CDEP modifies the objective function used train neural network, dis- played in [q[T] In addition to the standard prediction loss L. which teaches the model to produce the correct predictions, CDEP adds explanation error eepp, which teaches the model to produce the correct explanations for its predictions. In place of the prediction and labels fe(X) ) y, used in the prediction error L. the explanation error Lexpl uses the explanations produced by an interpretation method exple (X), along with targets provided by the user expl As is common with penalization, the two losses are weighted by hyperparameter À R:
i argmin C (fo())) + Lexpl (exply(X), explx)
0
Prediction enror
Explanation emor
The precise meanings of explx depend on the context. For example, the skin cancer image classification task described in Section 4 many of the benign skin images contain band-aids, but none of the malignant images. force the model to ignore band-aids making their prediction, in each image eXpl,(X) denotes the importance score of the band-aid and expl would be zero. These and more examples are further explored in Section
3.2 CONTEXTUAL DECOMPOSITION (CD)
In this work, we use the CD score as the explanation function. In contrast to other interpretation methods, which focus on feature importances, CD also captures interactions between features. CD was originally designed for LSTMs (Murdoch al. 22018) and subsequently extended to convolu- tional neural networks and arbitrary DNNs (Singh et al.) 2018). For a given DNN fxx) one can represent its output SoftMax operation applied to logits g(x). These logits, in turn, the composition L layers gs. such as convolutional operations or ReLU non-aincasssies.
f(x) = SoftMax(g(z)) = SoftMax(qu((g--11-
(2)
Given a group of features {zy Jes. the CD algorithm, g decomposes the logits g(x) into sum of two terms, 8(z) and 7(x). 3(x) is the importance score of the feature group {x, hess, and (xx) captures contributions to g(x) not included in (xx. The decomposition computed by iteratively applying decompositions CD( (I:) for each of the layers (1()).
(x) 9f ( (9-11 (...
CD
CD CD (x)))))) (8(x),
(0)
(3)
8(x) + (x0)== g(x)
(4)
3.3 CDEP OBJECTIVE FUNCTION
We now substitute the above CD scores into the generic equation in Eq arrive at the method used in this paper. While we use CD for the explanation method expl, other explanation methods could be readily substituted at this stage. In order convert CD scores probabilities, apply SoftMax operation to gC (x), allowing for easier comparison with the user-provided labels exply. We collect from the user. for each input I,, a collection feature groups Tg,s, x; E Rd. (1,... d), along with explanation target values expl_,e and use the || loss for Leepl.
3

---

Under review as a conference paper at ICLR 2020
0 = argmin
ΣΣ
Vi.e Vi,clog fo(xs)e + ΣΣ || expl
(5)
Classification error
Explanation error
In the above, indexes each individual example in the dataset, S indexes subset of the features for which we penalize their explanations, and с sums over each class. Updating the model parameters in accordance with this formulation ensures that the model not only predicts the right output but also does so for the right (aligned with prior knowledge) reasons.
3.4 COMPUTATIONAL CONSIDERATIONS
similar idea to Eq has been proposed in previous/coneurrent work, where the choice of explana- tion method uses gradient-han based attribution method (Ross et al.) 2017; Erion et al., (019). How- ever, using such methods leads to three main complications which are solved by our approach. The first complication is the optimization process. When optimizing over attributions from a gradient- based attribution method via gradient descent, the optimizer requires the gradient of the gradient, thus requiring that all network components be twice differentiable. This process is computationally expensive and indeed optimizing exactly involves optimizing over a differential equation. con- trast, CD attributions are calculated along with the forward pass of the network, and result be optimized plainly with back- propagation using the standard single forward-pass and backward-pass per batch.
A second complication solved by the use of CD in Eq iis the ability to quickly finetune a pre-trained network. many applications, particularly in transfer learning, last few layers of pre-trained neural network. Using CD, one can freeze early layers of the network and then finetune the last few layers of the network quickly the activations and gradients of the frozen layers are not necessary.
common finetune only the
Third, penalizing gradient- based methods incurs very large memory usage. Using gradient-based methods, training requires the storage of activations and gradients for all layers of the network as well as the gradient of input (which can be omitted in normal training). Even for the simplest version, based saliency, this more than doubles the required memory for given batch and network size. More advanced methods proved to be completely infeasible to apply to real-life dataset used, since the memory requirements were too high. By contrast, penalizing CD only requires small constant amount of memory more than standard training.
4
RESULTS
The results here demonstrate the efficacy of CDEP on a variety of datasets using diverse explanation types. Sec 4.1 shows results ignoring spurious patches in the ISIC skin cancer dataset (Codella et al. 2019). Sec 4.2
detaiis details experiments on converting a DNN's preference for color to preference for shape on a variant of the MNIST dataset (LeCun 1998), and Sec shows experiments on text data from the Stanford Sentiment Treebank (SST) (Socher et al. 2013)
4.1 IGNORING SPURIOUS SIGNALS IN SKIN CANCER DIAGNOSIS
In recent years, deep learning has achieved impressive results in diagnosing skin cancer, with predic- tive accuracy sometimes comparable to human doctors (Esteva et al.) (017). However, the datasets used to train these models often include spurious features which make it possible to attain high test accuracy without learning the underlying phenomena (Winkler et al. 2019). In partieular, pop- ular dataset from ISIC (International Skin Imaging Collaboration) has colorful patches present in approximately 50% of the cancerces images but not the cancerous images (Codella et al.) 2019). An unpenalized DNN learns to look for these patches as an indicator for predicting that an image is benign. We use CDEP to remedy this problem by penalizing the DNN placing importance on the patches during training.
All models were trained in PyTorch.
4

---

Under review as a conference paper at ICLR 2020
Benign
Malignant
half of data polluted with parchex
Figure 2: Example images from the ISIC dataset. Half of the benign lesion images include a patch in the image.
The task in this section is to classify whether an image of a skin lesion contains (1) benign melanoma or (2) malignant melanoma. The ISIC dataset consists of 21,654 images (19,372 benign), each diag- nosed by histopathology or a consensus of experts. For classification, we use a VGG architecture the ImageNet Classification task and freeze the
(Simonyan & Zisserman, 2014) pre-trained weights of early layers so that only the fully connected layers are trained. In order to use CDEP, the spurious patches are identified via imple image segmentation algorithm using color threshold (see S4p.
Table shows results comparing the performance of model trained with and without CDEP. We report results on two variants of the test set. The first, which we refer to as "no patches" only contains images of the test set that do not include patches. The second also includes images with those patches. Training with CDEP improves the AUC and F1-score for both test sets.
Table 1: Results from training a DNN on ISIC to recognize skin cancer (averaged over three runs). Results shown for the entire test set and for only the images the test set that do not include patches ("no patches"). The network trained with CDEP generalizes better, getting higher AUC and F1 on both. Std below 0.006 for all AUC and below 0.012 for all F1.
AUC (no patches)
F1 (no patches)
AUC (all)
FI (all)
Vanilla (excluded data)
0.86
0.59
0.92
0.59
Vanilla
0,85
0.56
0.92
0.56
With RRR
0.66
0.39
0.82
0.39
With CDEP
0.88
0.61
0.93
0.61
In the first row of Table [1, the model is trained using only the data without the spurious patches, and the second row shows the model trained on the full dataset. The network trained using CDEP achieves the best AUC, surpassing both unpenalized versions. Applying our method increases the ROC AUC as well as the best Fl score. We also compared our method against the method introduced in 2017 by Ross et al. (RRR). For this, restricted the batch size to 16 (and consequently use learning rate of 10 10-") due to memory constraints. Using RRR did not improve on the base AUC, implying that penalizing gradients is not helpful in penalizing higher-order features
Visualizing explanations Fig. 3 visualize GradCAM heatmaps Ozbulakk 2019; Selvaraju et al, 2017) for unpenalized DNN and a DNN trained with CDEP to ignore spurious patches. As expected, after penalizing with CDEP, the DNN attributes less importance to the spurious patches, regardless of their position in the image. More examples, also for cancerous images, shown in Sec
'Pre-trained model retrieved from torchvision
We were not able to compare against the method recently proposed in Frion et al. 2019 due the prohibitively slow training and large memory requirements.
5

---

Under review as a conference paper at ICLR 2020
Image
Vanilla
CDEP
CDEP climinates attention on spurious patch
Figure 3: Visualizing heatmaps for correctly predicted exampes from the ISIC skincancer test set. Lighter regions in the heatmap are attributed more importance. The DNN trained with CDEP cor- rectly captures that the patch is not relevant for classification.
4.2 COMBATING INDUCTIVE BIAS ON VARIANTS OF THE MNIST DATASET
In this section, we investigate whether we can alter which features DNN uses perform digit classification, using variants of the MNIST dataset (LeCun) 1998) and standard CNN architecture for this dataset retrieved from PyTorch]
4.2.1 COLORMNIST
Similar to a previous study (Li Vasconcelos 2019), we transform the MNIST dataset to include three color channels and assign each class distinet color, shown in Fig. 4 An unpenalized DNN trained on this biased data will completely misclassify test set with inverted colors, dropping to 0% accuracy (see Section 4.2.1) suggesting that it learns classify using the colors of the digits rather than their shape.
see if we can alter the DNN to focus on the shape of the digits rather than their color. Interestingly, this can be enforced by minimizing the contribution of pixels in isolation while maximizing the importance of groups of pixels (which can represent shapes). To do this, we add penalize the CD contribution of sampled single pixel values, following Eq S By minimizing the contribution of single pixels we effectively encourage the network focus more on groups of pixels, which can represent shape.
Here,
Training
2
3 4
Test
0 /
2
4 6
8 9
Figure 4: ColorMNIST: the test set shapes remain the same the training set, but the colors
inverted. A vanilla network trained on this training set will get 0% accuracy on the test set.
Section 4.2. shows that CDEP can partially change the network's 's focus on solely color to also focus on digit shape. compare CDEP two previously introduced explanation penalization techniques: penalization of the squared gradients (RRR) (Ross et al. (017)) and Expected Gradients (EG) (Erion et al. 2019). For EG we penalize variance between attributions of the RGB channels
Model and training code from htaff///sthhhceooooooommoemoeoee

---

Under review as a conference paper at ICLR 2020
(as recommended by the authors of EG in personal correspondence). None of the baselines are able to improve the test accuracy of the model on this task above the random baseline, while CDEP is able to significantly improve this accuracy to 31.0%. We show the increase of predictive accuracy with increasing penalization in Fig. S5
Table Results on ColorMNIST (test accuracy). All values averaged over thirty runs. CDEP the only method that captures and removes color bias.
Unpenalized
Baseline 9.8
CDEP
RRR
Expected Gradients
Test Accuracy
0.2 = 0.2
31.0 + 2.3
0.2 0.1
10.0 = 0.1
4.2.2 DECOYMNIST
For further comparison with previous work, we evaluate CDEP on an existing task: MecOyMNIST (Erion et al. 2019). DecoyMNIST adds class-indicative gray patch to random corner of the image. This task is relatively simple, as the spurious features are not entangled with any other feature and are always at the same location (the comnrr). Table shows that all methods perform roughly equally, recovering the base accuracy. Results are reported using the best penalization parameter X, chosen via gross-validttinn on the test accuracy. We provide details on the computation time, and memory usage in Table [S], showing that CDEP is similar to existing approaches. However, when freezing early layers of network and finetuning, CDEP very quickly becomes more efficient than other methods.
Table 3: Results on Grayscale Decoy set.
Unpenalized 60.1 5.1
CDEP
RRR
Expected Gradients 97.8 + 0.2
Test accuracy
97.2 0.8 99.0 E 1.0
4.3 FIXING BIAS IN TEXT DATA
To demonstrate CDEP's effectiveness on text, we use the Stanford Sentiment Treebank (SST) dataset (Socher et al. 2013), an NL. benchmark dataset consisting of movie reviews with binary sentiment (positive/negative. We inject spurious signals into the training set and train standard L.STM 5 to classify sentiment from the review.
Positive
pacino the best she's been years and keener marvelous she showcases davies as young woman of great charm, generosity and diplomacy shows she 's back in form, with an astoundingly rich film. proves once again that she's the best brush in the business
Negative
green ruins every single scene he's in, and the film, while it 's not completely wreaked, scriously compromised by that
i'm sorry to say that this should seal the deal - arnold is not, nor will he be, back this is sandler running empty, , repeating what he 's already done way too often
so howard appears to have had free rein to be pretentious he wanted
Figure Example sentences from the SST dataset with artificially induced bias on gender.
Model and training code from f httpl//w///iiinoooooooooooooooot

---

Under review as a conference paper at ICLR 2020
We create three variants of the SST dataset, each with different spurious signals which we aim to ignore (examples in See STT). In the first variant, we add indicator words for each class (positive: 'text', negative: 'video") ) at random location in each sentence. An unpenalized DNN will focus only on those words, dropping to nearly random performance on the unbiased test set. In the second variant, we use semantically similar words ('the', 'a') indicate the class by using one word only in the positive and one only in the negative class. In the third case, we use 'he" and 'she' to indicate class (example in Fig 5). Since these gendered words are only present in small proportion of the training dataset (~ 2%). for this variant, we report accuracy only on the sentences in the test set that do include the pronouns (performance on the test dataset not including the pronouns remains unchanged). Thole&sshows the test accuracy for all datasets with and without CDEP. In all scenarios,
CDEP successfully able to improve the test accuracy by ignoring the injected spurious signals.
Table 4: Results on SST. CDEP substantially improves predictive accuracy on the unbiased test set after training on biased data.
Unpenalized
CDEP
Random words
56.6 + 5.8
75.4 + 0.9
Biased (articles)
57.8 + 0.8
68.2 + 0.8
Biased (gender)
64.2 3.1
78.0 = 3.0
5 CONCLUSION
In this work we introduce novel method to penalize neural networks to align with prior knowledge. Compared to previous work, CDEP is the first of its kind that can penalize complex features and feature interactions. Furthermore, CDEP is more computationally efficient than previous work and does not rely on backpropagation, enabling its use with more complex neural networks. We show that CDEP can be used to remove bias and improve predictive accuracy on variety of toy and real data. The experiments here demonstrate a variety of ways to use CDEP to improve models both on real and toy datasets. CDEP is quite versatile and can be used in many more areas to incorporate the structure of domain knowledge (e.g. biology or physics). Of course, the effectiveness of CDEP depends upon the quality of the prior knowledge used to determine the explanation targets. Future work includes extending CDEP to more complex penalties. incorporating more fine-gre fine-grained explanations and interactions. We hope the work here will help push the field towards a more rigorous way to use interpretability methods, point which will become increasingly important as interpretable machine learning develops as a field (Doshi Velez & Kim 2017 Murdoch et al. 2019).
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(1 (((11):2274420 2012.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp- 9505-9515, 2018
Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In 6th International Conference on Learning Representations (ICLR 2018), 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Mlaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by wise relevance propagation. PloS one, 10(7):e0130140, 2015.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert MÅäller. How to explain individual classification decisions. Journal of Machine Learning Research, 1 1[Jun):1803-1 2010.
8

---

Under review as a conference paper at ICLR 2020
Tolga Bolukbasi, Kai- Wei Chang. James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pp. 3349-4355 2016.
Kaylee Burns, Lisa Anne Hendricks, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. arXiv preprini 18X3::/80000977 2018.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo. Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint porXivv9902 190..03322. 2019.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers, arXiv preprint arXiv:1705.07857, 2017.
Finale Doshi- and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: met::/70200.0. 2017.
Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Learning plainable models using attribution priors. arXiv preprint arXiv:1906. 2019.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-leve classification of skin cancer with deep neural networks. Nature, 542(7639) 2017.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. arXiv preprint vrXiv:174..03266 2017.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115 (16)E3335- 2018. ISSN 0027-8424. doi: 10.1073/pnas.172005 1720347115. URL https:
www. www.pppas.com httpaa/oorrrootttet////////////
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:/811.1222. 2018.
Sarthak Jain and Byron Wallace. Attention is not explanation. arXiv preprint arXiv: 1rXiv:1902.101800 2019.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. arXiv preprint arXiv: 0799/, 2019.
Frederick Liu and Besim Avci. Incorporating priors with feature attribution on text classification. arXiv preprint arXiv:1906. 09w.:0806.0088.... 2019.
Scott M Lundberg and Sn-In Lec. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4778-4777, 2017.
Masahiro Mitsuhara, Hiroshi Fukui, Yusuke Sakashita, Takanori Ogata, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Embedding human knowledge in deep neural network via attention map. arXiv preprint arXiv:1905.03540. 2019.
W James Murdoch and Arthur Szlam. Automatic rule extraction from long short term memory networks. arXiv preprint arXiv: / 7702002540 2017.
W James Murdoch, Peter J Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from Istms. arXiv preprint arXiv:1801.05453 2018
W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Interpretable machine learning: definitions, methods, and applications. arXiv preprint arXiv:1901 1901.0459,
04592, 2019.

---

Under review as a conference paper at ICLR 2020
Weili Nie, Yang Zhang, and Ankit Patel. A theoretical explanation for perplexing behaviors of backpropagation-batt based visualizations. arXiv preprint arXiv:1805.07033, 2018.
Utku Ozbulak. Pytorch enn visualizations.
2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144, ACM, 2016.
Laura Rieger and Lars Kai Hansen. Aggregating explainability methods for neural networks stabi- lizes explanations. arXiv preprint arXiv:1903. 2019.
Andrew Slavin Ross and Finale Doshi-Vekde Improving the adversarial robustness and interpretabil- ity of deep neural networks by regularizing their input gradients. In Thirty-second AAAI confer- ence on artificial intelligence, 2018.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez Right for the right reasons: Train- ing differentiable models by constraining their explanations. arXiv preprint 1r03::7703.03717 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. See https://arxiv. org/abs/1610.02333 v3, 7(8), 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.017/3. 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Chandan Singh, W James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. arXiv preprint 1006::/880.05350 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint 1rXiv:1412.6880 68066, 2014.
Julia Strout, Ye Zhang, and Raymond Mooney. Do human rationales improve machine explana- tions? arXiv preprint 1rXiv:1905.13714 2019.
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. ICML, 2017.
Haohan Wang, Zexue He, Zachary Lipton, and Eric P Xing. Learning robust representations by projecting superficial statistics out. arXiv preprint arXiv:1903.06256. 2019.
Julia K. Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Nofimann-Welleninn Luc Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, and Holger A. Haenssle. Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma RecognitionSur- gical Skin Markings in Dermoscopic Images and Deep Learning Convolutional Neural Network Recognition of MelanomaSurgical Skin Markings in Dermoscopic Images and Deep Learning Convolutional Neural Network Recognition of Melanoma. JAMA Dermatology, 08 2019. ISSN
10

---

Under review as a conference paper at ICLR 2020
2168-6068. doi: 10.1011/amddematl022191777... jamadermatol. 2tt.:ddee.att1.2.1..11... 2019..01....
URL https:://ac..rrg110..o
Omar Zaidan, Jason Eisner, and Christine Piatko. Using annotator rationales to improve machine learning for text categorization. In Human Language Technologies 2007: The Conference the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pp. 260-267, 2007.
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural net- works. arXiv preprint urXiv:1905.09999 2019.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv: 2017.

---

Under review as a conference paper at ICLR 2020
Supplement
SI ADDITIONAL DETAILS ABOUT SST TASK
shows the results for CDEP biased variants of the SST dataset. Here show exam- ples of the biased sentences (for task 2 and 3 we only show sentences where the bias was present) in For the first task, we insert two randomly chosen words in 100% of the sentences in the positive and negative class respectively. We choose two words ("text" for the positive class and "video" for the negative class) that were not otherwise present in the data but had a representation in Word2Vec.
Section 4.3
Figs. ST to S3
Positive
part of the charm of satin rouge is that it avoids the obvious with text humour and lightness. text screenplay more ingemiously constructed than 'memento' good text, good action, good acting, good dialogue, good pace, good cinematography. dramas like text this make human.
Negative
begins with promise, but runs aground after being video snared in its own tangled plot. the video movie is well done, but slow this orange has some juice, is, video too long and unfocused.
's video from fresh-squeezed.
Figure S1: Example sentences from the variant I of the biased SST dataset with decoy variables in each sentence.
For the second task, we choose replace two common words ('the" and "a") in sentences where they appear (27% of the dataset). We replace the words such that one word only appears in the positive class and the other world only in the negative class. By choosing words that are semantically almost replaceable, we ensured that the normal sentence structure would not be broken such with the first task.
Positive
comes off as touching, transcendent love story... most remarkable not because of its epic scope but because a startling intimacy
couldn't be better cruel but weirdly likable wasp matron uses humor and heartfelt conviction to tell that story about discovering your destination in life
Negative
to creep the living hell out of you holds its goodwill close. but is relatively slow to come to the point not the great monster movie.
consider the dvd rental instead
Figure S2: Example sentences from the variant of the SST dataset with artificially induced bias on articles ("the",, "a"). Bias was only induced on the sentences where those articles were used (27% of the dataset).
For the third task repeat the same procedure with two words ('he" and "she") that appeared in only 2% of the dataset. This helps evaluate whether CDEP works even if the spurious signal appears only in a small section of the data set.
12

---

Under review as a conference paper at ICLR 2020
Positive
pacino the best she's been years and keener marvelous she showcases davies young woman of great charm, generosity and diplomacy shows she 's back in form, with an astoundingly rich film. proves once again that she's the best brush the business
Negative
green ruins every single scene he's in, and the film, while it completely wreaked, is scriously compromised by that
i'm sorry to say that this should seal the deal aroold not, nor will he be, back
is sandler rumning on empty repeating what already done way too often.
howard appears have had free rein be as pretentious wanted
Example sentences from the variant 3 of the SST dataset with artificially induced bias on articles ("he", "she"). Bias was only induced on the sentences where those articles were used (2% of the dataset).
Figure S3:
S2 NETWORK ARCHITECTURES AND TRAINING
$2.1 NETWORK ARCHITECTURES
For the ISIC skin cancer task we used pretrained VGGI6 network retrieved from the PyTorch model zoo. We use SGD as the optimizer with a learning rate of 0.01 and momentum of 0.9. Preliminary experiments with Adam as the optimizer yielded poorer predictive performance.
or both MNIST tasks, we use standard convolutional network with two convolutional channels followed by max pooling respectively and two fully connected layers:
Conv(20,5,5) MaxPool() Conv(50,5.5) MaxPool - FC(256) FC(10). The models were trained
with Adam, using a weight decay of 0.001.
Penalizing explanations adds an additional hyperparameter, À to the training. À can either be set in proportion to the normal training loss or at a fixed rate. this paper did the latter. We expect that exploring the former could lead to more stable training process. For all tasks À was tested across a wide range between [10-1, 10*].
The LSTM for the SST experiments consisted of two LSTM layers with 128 hidden units followed by a fully connected layer.
$2.2 COLORMNIST
For fixing the bias in the ColorMNIST task, we sample pixels from the distribution non-zero pixels over the whole training set, as shown in Fig, S4
Figure S4: Sampling distribution for ColorMNIST
13

---

Under review as a conference paper at ICLR 2020
CDEP
Ross
30
EGradients
EGradients (var)
25
Random
Test Accuracy
20
15
10
5
0
10a
10a
10a
Regularization strength
Figure S5: Results on ColorMNIST (Test Accuracy). All averaged over thirty runs. CDEP is the only method that captures and removes color bias.
For Expected Gradients we show results when sampling pixels as well as when penalizing the vari- ance between attributions for the RGB channels (as recommended by the authors of EG) in Fig. [S5 Neither of them go above random accuracy, only achieving random accuracy when they are regular- ized constant prediction.
S3 RUNTIME AND MEMORY REQUIREMENTS OF DIFFERENT ALGORITHMS
This section provides further details on runtime and memory requirements reported in Table SI We compared the runtime and memory requirements of the available regularization schemes when implemented in Pytorch.
Memory usage and runtime were tested on the DecoyMNIST task with a batch size of 64. It is expected that the exact ratios will change depending on the complexity of the used network and batch size (since constant memory usage becomes disproportionally smaller with increasing batch size).
The memory usage was read by recording the memory allocated by PyTorch. Since Expected Gra- dients and RRR require two forward and backward passes, we only record the maximum memory usage. We ran experiments on single Titan X.
Table S1: Memory usage and run time were recorded for the DecoyMNIST task.
Unpenalized
CDEP
RRR
Expected Gradients
Run time/epoch (seconds) Maximum GPU RAM usage (GB)
4.7
17.1
11.2
17.8
0.027
0.068
0.046
0.046
S4 IMAGE SEGMENTATION FOR ISIC SKIN CANCER
To obtain the binary maps of the patches for the skin cancer task, we first segment the images using SLIC, common saage-segmentttion algorithm (Achanta et al. 20122). Since the patches look quite distinct from the rest of the image, the patches are usually their own segment.
Subsequently take the RGB and HSV values for all segments and filtered for segments which the mean was substantially different from the typical caucasian skin tone. Since different images different from the typical skin color in different attributes, filtered for those images
14

---

Under review as a conference paper at ICLR 2020
recursively. As example, in the image shown in Fig. [6 the patch has much higher saturation than the rest of the image. For each image we exported a map as seen in Fig. S6
Figure S6: Sample segmentation for the ISIC task.
15

---

Under review as a conference paper at ICLR 2020
S5 ADDITIONAL HEATMAP EXAMPLES FOR ISIC
We that the importance maps for the unregularized and regularized network very similar for cancerous images and non-cancerous images without patch. The patches are ignored by the network regularized with CDEP.
We show additional examples from the test set of the skin cancer task in Figs. S7 and S8
Image
Vanilla
CDEP
Figure S7: Heatmaps for benign samples from ISIC
16

---

Under review as a conference paper at ICLR 2020
Image
Vanilla
CDEP
Figure S8: Heatmaps for cancerous samples from ISIC
17

---

Under review as a conference paper at ICLR 2020
A different spurious correlation that noticed that proportionally more images showing skin cancer will have ruler next to the lesion. This is the case because doctors often want to show a reference for size if they diagnosed that the lesion is cancerous. Even though the spurious correlation less pronounced (in very rough cursory count, 13% of the cancerous and 5% of the benign images contain some sort of measure), the networks learnt recognize and exploit this spurious correlation. This further highlights the need for CDEP, especially in medical settings.
Image
Vanilla
CDEP
Both networks learni the nun penalized spurious correlation:
Ruler -h: Cancer
Figure S9: Both networks learnt that proportionally more images with malignant lesions feature ruler next to the lesion. To make comparison easier, we visualize the heatmap by multiplying it with the image. Visible regions are important for classification.
18


Reviews:
[{"review_id": "rklgSCRptr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a way of using generated explanations of model predictions to help prevent a model from learning \"unwanted\" relationships between features and class labels. This idea was implemented with a particular explanation generation method from prior work, called contextual decomposition (CD). For a given feature, the corresponding CD can be used to measure its importance. The proposed learning objective in this work optimizes not only the cross entropy loss, but also the difference between the CD score of a given feature and its explanation target value. Experiments show that this new learning algorithm can largely improve the classification performance.\n\nI like the high-level idea of this work and agree that there is not much work on using prediction explanations to help improve model performance. However, there are two major concerns of the model and experiment design. \n\nFirst, it seems like the proposed method requires whoever use it already know what the problem is. For example, \n\n- in section 3.3, the model inputs include a collection of features and the corresponding explanation target values.\n- in section 4.1, it is already known that some colorful patches only appear in some non-cancerous images but not in cancerous images. \n- it is even more obvious in section 4.2 and 4.3, because in both experiments, the training and test examples were altered on purpose to create some mismatch. \n\nMy question is that if we already know the bias or the mismatch, why not directly use this information in the regularization to penalize some features? Is it necessary to resort to some explanation generation methods?\n\nMy second concern is more like a personal opinion. In the experiment of section 4.2, if the colors are good indicators of these digits in the training set, I don't it is wrong for a model to capture these important features. However, the way of altering examples in the same class with different colors in training and test sets seems questionable, because now, the distributions of training and test images are different. On the other hand, if we already know color is the issue, why not simply convert the images into black-and-white? A similar argument can also be applied to the experiment in section 4.3\n\nOverall, I like the idea of using explanations to help build a better classifier. However, I am concerned about the value of this work. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkgqk1x6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method intended to allow practitioners to *use* explanations provided by various methods. Concretely, the authors propose contextual decomposition explanation penalization (CDEP), which aims to use explanation methods to allow users to dissuade the model from learning unwanted correlations. \n\nThe proposed method is somewhat similar to prior work by Ross et al., in that the idea is to include an explicit term in the objective that encourages the model to align with prior knowledge. In particular, the authors assume supervision --- effectively labeled features, from what I gather --- provided by users and define an objective that penalizes divergence from this. The object that is penalized is $\\Beta(x_i, s)$, which is the importance score for feature s in instance $i$; for this they use a decontextualized representation of the feature (this is the contextual decomposition aspect). Although the authors highlight that any differentiable scoring function could be used, I think the use of this decontextualized variant as is done here is nice because it avoids issues with feature interactions in the hidden space that might result in misleading 'attribution' w.r.t. the original inputs.\n\nThe main advantage of this effort compared to work that directly penalizes the gradients (as in Ross et al.) is that the method does not rely on second gradients (gradients of gradients), which is computationally problematic. Overall, this is a nice contribution that offers a new mechanism for exploiting human provided annotations. I do have some specific comments below.\n\n- I am not sure I agree with the premise as stated here. Namely, the authors write \"For an explanation of a deep learning model to be effective, it must provide both insight into a model and suggest a corresponding action in order to achieve some objective\" -- I would argue that an explanation may be useful in and of itself by highlighting how a model came to a prediction. I am not convinced that it need necessarily lead to, e.g., improving model performance. I think the authors are perhaps arguing that explanations might be used to interactively improve the underlying model, which is an interesting and sensible direction.\n\n- This work, which aims to harness user supervision on explanations to improve model performance, seems closely related to work on \"annotator rationales\" (Zaidan 2007 being the first work on this), but no mention is made of this. \"Do Human Rationales Improve Machine Explanations?\" by Strout et al. (2019) also seems relevant as a more recent instance in this line of work. I do not think such approaches are necessarily directly comparable, but some discussion of how this effort is situatied with respect to this line of work would be appreciated.\n\n- The experiment with MNIST colors was neat. \n\n- The authors compare their approach to Ross and colleagues in Table 1 but see quite poor results for the latter approach. Is this a result of the smaller batch size / learning rate adjustment? It seems that some tuning of this approach is warranted. \n\n- Figure 3 is nice but not terribly surprising: The image shows that the objective indeed works as expected; but if this were not the case, then it would suggest basically a failure of optimization (i.e., the objective dictates that the image should look like this *by construction*). Still, it's a good sanity check.\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxDW5Jl5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to add a regularizer to the loss function when training a prediction model. In particular, the regularizer considers explanations during the model training; if the explanations are not consistent with some prior knowledge, then explanation errors will be introduced. \n\nThe motivation for the proposed research is interesting and has some merit. However, I am a bit worried that the proposed approach is somewhat ad hoc. I can imagine there are various explanations that can be generated for the same model. There can also be different prior knowledge available for a particular problems. Which prior knowledge and explanations to use seem to affect a lot about the learned model. But there is no principled approaches for making the selection.\n\nIn some sense, standard regularizers such as L1 or L2 are are intrinsic regularizers, while the proposed regularizer is extrinsic regularizer. I think the extrinsic regularizer certainly has some merit, but it is also hard to regulate.\n\nFor instance, consider the example in Figure 2 about the presence of patches. Isn't that a too specific knowledge about the dataset, which in turn makes the proposed approach not general? I have doubts on how useful a method is if it relies on such specific prior knowledge about the data.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contains interesting ideas for giving simple explanations to a NN; however, the reviewers do not feel the contribution is sufficiently novel to merit acceptance.
Decision: Reject