Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper shows that interpretations for DNN decisions, e.g. computed by methods such as sensitivity analysis or DeepLift, are fragile: Visually (to a human) inperceptibly different image cause greatly different explanations (and also to an extent different classifier outputs). The authors perturb input images and create explanations using different methods. Even though the image is inperceptibly different to a human observer, the authors observe large changes in the heatmaps visualizing the explanation maps. This is true even for random perturbations. 

The images have been modified wrt. to some noise, such that they deviate from the natural statistics for images of that kind. Since the explanation algorithms investigated in this papers merely react to the interactions of the model to the input and thus are unsupervised processes in nature, the explanation methods merely show the model's reaction to the change.
For one, the model itself reacts to the perturbation, which can be measured by the (considarbly) increased class probability. Since the prediction score is given in probabilty values, the reviewer assumes the final layer of the model is a SoftMax activation. In order to see change in the softmax output score, especially if the already dominant prediction score is further increased, a lot of change has to happen to the outputs of the layer serving as input to the SoftMax layer.

It can thus be expected, that the input- and class specific explanations change as well, to an also not so small extent. The explanation maps mirror for the considered methods the model's reaction to the input. They are thus not meaningless, but are a measure to model reaction instead of an independent process. The excellent Figure 2 supports this point. Not the interpretation itself is fragile, but the model.
Adding a small delta to the sample x shifts its position in data space, completely altering the prediction rule applied by the model due to the change in proximity to another section of the decision hyperplane. The fragility of DNN models to marginally perturbed inputs themselves is well known. 
This especially true for adversial perturbations, which have been used as test cases in this work. The explanation methods are expected to highlight highly important areas in an image, which have been targetet by these perturbation approaches.

The authors give an example of an adversary manipulating the input in order to draw the activation to specific features to draw confusing/malignant explanation maps. In a settig of model verification, the explanation via heatmaps is exactly what one wants to have: If tiny change to the image causes lots of change to the prediction (and explanation) we can visualize the instability of the model not the explanation method.
Further do targeted perturbations not show the fragility of explanation methods, but rather that the models actually find what is important to the model. It can be expected, that after a change to these parts of the input, the model will decide differently, albeit coming to the same conclusion (in terms of predicted class membership), which reflects in the explanation map computed for the perturbed input.

Further remarks:
It would be interesting to see the size and position of the center of mass attacks in the appendix. The reviewer closely follows and is experienced with various explanation methods, their application and the quality of the expected explanations. The reviewer is therefore surprised by the poor quality and lack of structure in the maps obtained from the DeepLift method. Can bugs and suboptimal configurations be ruled out during the experiments? The DeepLift explanations are almost as noisy as the ones obtained for Sensitivity Analysis (i.e. the gradient at the input point). However, recent work (e.g. Samek et al., IEEE TNNLS, 2017 or Montavon et al., Digital Signal Processing, 2017) showed that decomposition-based methods (such as DeepLift) provide less noisy explanations than Sensitivity Analysis.

Have the authors considered training the net with small random perturbations added to the samples, to compare the "vanilla" model to the more robust one, which has seen noisy samples, and compared explanations? 
Why not train (finetune) the considered models using softplus activations instead of exchanging activation nodes?
Appendix B: Heatmaps through the different stages of perturbation should be normalized using a common factor, not individually, in order to better reflect the change in the explanation

Conclusion:
The paper follows an interesting approach, but ultimately takes the wrong view point:
The authors try to attribute fragility to explaining methods, which visualize/measure the reaction of the model to the perturbed inputs. A major rework should be considered.
review_title: Interesting work, but wrong conclusions

final_score: rating: 4: Ok but not good enough - rejection
review_content: The key observation is that it is possible to generate adversarial perturbations wherein the behavior of feature importance methods (e.g. simple gradient method (Simonyan et al, 2013), integrated gradient (Sundararajan et al, 2017), and DeepLIFT ( Shrikumar et al, 2016) ) have large variation while predicting same output.    Thus the authors claim that one has to be careful about using feature importance maps.

Pro:  The paper raises an interesting point about the stability of feature importance maps generated by gradient based schemes.

Cons:
The main problem I have with the paper is that there is no precise definition of what constitutes the stable feature importance map.  The examples in the paper seem to be cherry picked to illustrate dramatic effects.   The experimental protocol used does not provide enough information of the variability of the salience maps shown around small perturbations of adversarial inputs. The paper would benefit from more systematic experimentation and a better definition of what authors believe are important attributes of stability of human interpretability of neural net behavior.
review_title: The key point of the paper is that it is possible to get different salience maps for interpretability while retaining correct labeling. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors study cases where interpretation of deep learning predictions is extremely fragile. They systematically characterize the fragility of several widely-used feature-importance interpretation methods. In general, questioning the reliability of the visualization techniques is interesting. Regarding the technical details, the reviewer has the following comments: 

- What's the limitation of this attack method?

- How reliable are the interpretations? 

- The authors use spearman's rank order correlation and Top-k intersection as metrics for interpretation similarity. 

- Understanding whether influence functions provide meaningful explanations is very important and challenging problem in medical imaging applications. The authors showed that across the test images, they were able to perturb the ordering of the training image influences. I am wondering how this will be used and evaluated in medical imaging setting. 

review_title: Interpretation of Neural Network is Fragile

Meta-review: The paper tries to show that many of the state-of-the-art interpretability methods are brittle and do not provide consistent stable explanations. The authors show this by perturbing (even randomly) the inputs so that the differences are imperceptible to a human observer but the interpretability methods provide completely different explanations. Although the output class is maintained before and after the perturbation it is not clear to me or the reviewers why one shouldn't have different explanations. The difference in explanations can be attributed to the fragility of the learned models (highly non-smooth decision boundaries) rather than the explanation methods. This is a critical point and has to come out more clearly in the paper.
Decision: Reject