Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a method to check whether a model is under attack by using state of the art explainability model, SHAP. They evaluated their technique using CIFAR-10 and SVHN w.r.t. 5 baseline techniques. They showed their method outperforms all the other baselines with a significant margin.

+ Overall I think the paper made a valuable contribution to the adversarial ML literature. Using explainability to detect the presence of adversarial attacks seems like a nice intuitive idea and the results show that it indeed works. 

-	However, the contribution of the paper is rather incremental. They just used SHAPE to adversarial and negative examples. I do not see any insight while explaining the results. 

-	Why under PGD and FGSM attack under higher noise, the proposed technique is similar or slightly worse than Lid and Mohalonabis baselines?

-	I would also like to see how these results hold good for a complicated dataset like ImageNet

		



review_title: Official Blind Review #1995

final_score: rating: 1: Reject
review_content: Summary: The authors propose an explanation-based adversarial example detection algorithm. The main idea is to train a discriminator to detect whether the explanatory saliency map is consistent with the input. Experiments have been conducted on CIFAR10 and SVHN to validate the method.

Comments:
+ The idea is straightforward and easy to follow.

- The use of SHAP as the only explanation method is not well explained. There are a plenty of works on visual explanation methods, such as guided-backprop[1], excitation-backprop[2], integrated gradient[3], Grad-CAM[4], real-time saliency[5] and so on. And based on my expertise, SHAP cannot generate the most accurate saliency among these methods.  If the proposed framework is general, why not to conduct ablation study on the different choice of explainer?

- Doubts on the effectiveness of the proposed method. According to former works[6, 7], explanatory saliency methods  are vulnerable and unreliable with respect to input perturbations. But in this paper, the authors assume that the explanation saliency map for normal examples are perfectly correct and used as positive instances for training the discriminator. I think they only focus on target attack, in which the attacking target label is semantically distinct from the original label, and the resulting saliency map distribution is very different from the correct one. However, considering a tabby cat image is perturbed to become tiger cat, since two classes are very close, the resulting saliency maps should be similar and the detector may fail to detect the adversarial example. Therefore, I encourage the authors to provide more results on this challenging scenario (for example, conduct un-target attack on imagenet dataset).

- The reported results in Figure 2(e) is abnormal. First, the blue line (authors' method) is very close to AUC=1.0 across different noise levels, which means that the detector can perfectly classify all the adversarial examples in all the situation. Second, the reported values of other methods are not correct. For example, the black line (original Mahalanobis) is below AUC=0.5 across all the noise level. However, in the Table3 ResNet-CIFAR10 row of its original paper[8], the reported AUC under C&W attack is 95.84, which is much larger than those shown in the figure. Therefore, I think the comparison is invalid. Similar problems also appear in Figure 2(f).

[1] J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. (2015). Striving for simplicity: The all convolutional net. In ICLR (workshop track).
[2] J. Zhang, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff. (2016). Top-down neural attention by excitation backprop. In ECCV.
[3] Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. In ICML. 
[4] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV.
[5] Dabkowski, P., & Gal, Y. (2017). Real time image saliency for black box classifiers. In NeurIPS.
[6] Kindermans, P. J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., ... & Kim, B. (2019). The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (pp. 267-280). Springer, Cham.
[7] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In NeurIPS
[8] Lee, K., Lee, K., Lee, H., & Shin, J. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: A Simple method to detect adversarial examples, but needs more work.

#Summary:
The paper proposed a method that utilizes the model’s explainability to detect adversarial images whose explanations that are not consistent with the predicted class.  The explainability is generated by SHAP, which uses Shapley values to identify relative contributions of each input to a class decision. It designs two detection methods: EXAID familiar, which is aimed to detect the known attacks and EXAID unknown, which is against unknown attacks. Both of the two methods are evaluated on perturbed test data which are generated by FGSM, PGD and CW attack with perturbations of different magnitudes. Qualitative results also show that the proposed method can effectively detect adversaries, especially when the perturbation is relatively small.

#Strength
The method is easy to implement and using the idea of interpretation for detecting adversarial examples seems interesting.

Good results are demonstrated compared with other comparators.

#Weakness
The idea of this paper is based on the interpretation method of DNN. However, it has been shown that these interpretation methods are not reliable and easy to be manipulated [1][2]. Therefore, although the method is simple to design, it also brings other security concerns.
Unfortunately, the paper does not address these issues. In addition, the comparators listed in the experiments are not state-of-art or common baselines. It is either not clear why authors modified the existing method and develop their own “unsupervised” version. 
In the experiments, many details are omitted. For example, how is the “noise level” defined? Are they based on L1, L2 or L-inf perturbation? For PGD attack, how many iterations does the generation run and what is the step size? How many effective adversarial examples are generated for training and testing? And all the experiments are conducted in a relatively small dataset, it is also suggested to do experiments on large datasets, e.g. Imagenet.
In the evaluation part, it looks strange to me why the EXAID familiar performs worse than EXAID unknown in evaluating FGSM attack on SVHN since the EXAID familiar is trained using FGSM attack.

#Presentation
I think the authors used a wrong template to generate the article. The font looks strange and the headnote indicates it is prepared for ICLR2020. The paper contains many typos and even the title contains a misspelling. Poor coverage of citations. There are more works for detecting adversarial examples that are published, e.g. [3][4][5]. On the other hand, the paper does not have the literature review for work related to the model interpretation.

Overall, I think the paper is not good enough for publication at ICLR.
[1] Dombrowski, Ann-Kathrin, et al. "Explanations can be manipulated and geometry is to blame." arXiv preprint arXiv:1906.07983 (2019).
[2] Ghorbani, Amirata, Abubakar Abid, and James Zou. "Interpretation of neural networks is fragile." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
[3] Meng, Dongyu, and Hao Chen. "Magnet: a two-pronged defense against adversarial examples." In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 135-147. ACM, 2017.
[4] Liao, Fangzhou, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. "Defense against adversarial attacks using high-level representation guided denoiser." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787. 2018.
[5] Ma, Shiqing, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang. "NIC: Detecting Adversarial Samples with Neural Network Invariant Checking." In NDSS. 2019.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper suggests a method for detecting adversarial attacks known as EXAID, which leverages deep learning explainability techniques to detect adversarial examples. The method works by looking at the prediction made by the classifier as well as the output of the explainability method, and labelling the input as an adversarial example if the predicted class is inconsistent with the model explanation. EXAID uses Shapley values as the explanation technique, and is shown to successfully detect many standard first-order attacks.

Though method is well-presented and the evaluation is substantial, the threat model of the oblivious adversary is unconvincing. The paper makes the argument that oblivious adversaries are more prevalent in the real world, but several works [1,2,3,etc.] have shown that with only query access to input-label pairs from a deep learning-based system, it is possible to construct black-box adversarial attacks. Thus, it is unclear why an attacker cannot just treat the detection mechanism as part of this black box, and mount a successful query-based attack. 

Though I recognize that the task of detection is separate from the task of robust classification, in both cases the defender should at least operate in the case where the attacker has input-output access to the end-to-end system (including whatever detection mechanisms are present). In particular, it seems impossible to "hide" a detector from an end user (when the method detects an adversarial example, it will alert the user somehow that the input was rejected), and so the user will be able to use this information to fool the system. The authors should investigate the white-box accuracy of their detection system, or at the very least try black-box attacks against the detector. For this reason I do not recommend acceptance for the paper at this time.

[1] https://arxiv.org/abs/1804.08598
[2] https://arxiv.org/abs/1807.04457
[3] https://arxiv.org/abs/1712.04248
review_title: Official Blind Review #1

Meta-review: This paper proposes EXAID, a method to detect adversarial attacks by building on the advances in explainability (particularly SHAP), where activity-map-like explanations are used to justify and validate decisions. Though it may have some valuable ideas, the execution is not satisfying, with various issues raised in comments. No rebuttal was provided.
Decision: Reject