Title: Controlling Over-generalization and its Effect on Adversarial Examples Detection and Generation
Authors: Mahdieh Abbasi
Arezoo Rajabi
Azadeh Sadat Mozafari
Rakesh B. Bobba
Christian Gagné
Venue: ICLR 2019
Keywords: Convolutional Neural Networks
Adversarial Instances
Out-distribution Samples
Rejection Option
Over-generalization

Abstract: Convolutional Neural Networks (CNNs) significantly improve the state-of-the-art for many applications, especially in computer vision. However, CNNs still suffer from a tendency to confidently classify out-distribution samples from unknown classes into pre-defined known classes. Further, they are also vulnerable to adversarial examples. We are relating these two issues through the tendency of CNNs to over-generalize for areas of the input space not covered well by the training set. We show that a CNN augmented with an extra output class can act as a simple yet effective end-to-end model for controlling over-generalization. As an appropriate training set for the extra class, we introduce two resources that are computationally efficient to obtain: a representative natural out-distribution set and interpolated in-distribution samples. To help select a representative natural out-distribution set among available ones, we propose a simple measurement to assess an out-distribution set's fitness. We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. Finally, we show that generation of white-box adversarial attacks using our proposed augmented CNN can become harder, as the attack algorithms have to get around the rejection regions when generating actual adversaries.

TL;DR: Properly training CNNs with dustbin class increase their robustness to adversarial attacks and their capacity to deal with out-distribution samples.

Review discussion:
The paper propose to incorporate an additional class for adversarial and out-distribution samples in CNNs. The paper propose to incorporate natural out-distribution images and interpolated images to the additional class, but the problem of selecting the out-distribution images is itself an important problem. The paper presents a very simple approaches for selecting the out-distribution images that relies on many hidden assumptions on the images source or the base classier, and the interpolation mechanism is also too simple and there is the implicit assumption of low complexity images. There exists more principled approaches for selecting out-distribution images that has not considered here like those based on uncertainty estimation or recently proposed direct out-distribution detectors.
In summary, the quality of the paper is poor and the originality of the work is low. The paper is easily readable.
We are thankful for the reviewer 2 to provide us with his/her feedback,
The reviewer mentioned: "the interpolation mechanism is also too simple":
We would like to highlight that despite the simplicity of interpolated samples, there has been demonstrated the effectiveness of using such samples on developing more regularized and generalized neural networks (Zhang et al, 2018) as well as on making them more secure (Zhao et. al. 2018).
Thus, we believe that simplicity does not necessarily lead to ineffectiveness.
The reviewer mentioned “many hidden assumptions on the images source or the base classier”:
As this statement is not clear for us, we would appreciate if the reviewer could elaborate more on it. The only assumption we made is on the fact that the out-distribution samples should be statistically and semantically different than the in-distribution samples.
Then among such out-distribution sets, we propose a measurement for identifying the most representative one among those available.
The reviewer stated "There exists more principled approaches for *selecting* out-distribution images that has not considered here like those based on uncertainty estimation or recently proposed direct out-distribution detectors":
While there are the approaches that aim to detect out-distribution sets, they have not been designed for the selection purposes as we do. By mentioning this statement, if the reviewer means the missing of some principled approaches like ODIN in our comparisons, we would like to inform the inclusion of ODIN results in the revised version of the paper.
Reference:
- Zhao, Jake, and Kyunghyun Cho. "Retrieval-Augmented Convolutional Neural Networks for Improved Robustness against Adversarial Examples." arXiv preprint arXiv:1802.09502 (2018).
- Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. ICLR 2018 (arXiv preprint arXiv:1710.09412).
The idea of having a separate class for out-distribution is a very interesting idea but unfortunately previously explored.
In fact, in machine learning and NLP there is the OOV class which sometimes people in computer vision also use. Some of the claims in the paper can be further substantiated or explored.
For example in abstract there is a simple claim that is presented too strong: We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. This claim is bigger than just CNNs and needs to be studied in a theoretical framework not an empirical one.
Also, one simple way to stop these adversarial cases would be to explore using Sigmoid as opposed to softmax. In general it is very unlikely that you will be able to choose every variation of out-distribution cases. Much easier if you just try to solve the problem using a set of n Sigmoids (n total number of classes) and consider each output a probability distribution.
However, the studies in this paper are still valuable and I strongly recommend continuing on the same direction.
We appreciate the reviewer 1 for his/her feedback on our paper.
The reviewer mentioned: “In general it is very unlikely that you will be able to choose every variation of out-distribution cases”:
Actually, for training A-CNN (Augmented CNN), we did not train it on every variation of out-distribution cases, rather, we recognize a single representative out-distribution set among the available ones according to our measurement.
Then using it for training A-CNN with the aim of effectively controlling over-generalization.
The reviewer mentioned: “ Also, one simple way to stop these adversarial cases would be to explore using Sigmoid as opposed to softmax”:
We would be appreciated if the reviewer could provide us with the references that showing using only sigmoid could control such a challenging problem of adversaries.
Please note we did not aim to devise a method that is able to reject all adversaries.
Rather, we attempted to show that a CNN with less over-generalization is able to reject some of the adversaries while correctly classifies many of the remainder, particularly non-transferable attacks.
This paper proposed to add an additional label for detecting OOD samples and adversarial examples in CNN models. This research direction seems interesting, however, the idea of using an extra label for OODs is not new and was previously explored in different domains. I would expect the describe how their method is different, and keep the research from that point.
Additionally, there are several claims in this paper which I'm not convinced are true, such as the over-generalization of CNNs, the choice of OODs (recent studies have shown NNs are not well calibrated, so using softmax as the confidence might not be the best idea), etc.
Reg. the results, did the authors compare their method to existing adv. example detection methods, such as Ma, Xingjun, et al. ICLR (2018) "Characterizing adversarial subspaces using local intrinsic dimensionality." ? or some other method?
Moreover, in Table 2. I'm not sure what should I conclude from the "Naive Model Error" on OOD samples.
The originality of our paper is not on making use of a dustbin class (OOV). As mentioned by the reviewers, this has been proposed not only for NLP and vision tasks but also for adversaries detection (Gross et. al, 2017, Hosseini et. al, 2017) and out-distribution identification (Yu et. al. 2017).
Our main goal is rather to demonstrate the relationship between over-generalization induced by naive CNN and its sensitivity to **both** adversarial examples and out-distribution samples. To this end, we used Augmented CNN (A-CNN) as a tool for controlling over-generalization.
Indeed, to our knowledge, we are the first to demonstrate this relationship, i.e. over-generalization and sensitivity to adversaries (as well as out-distribution samples), in an extensive experimental setup.
Thus, the originality of our paper does not lie on re-proposing A-CNNs, but we used it as a tool to show that mitigating the effect of over-generalization on the development of more secure neural networks (e.g. CNN) in the presence of adversaries and out-distribution sets.
The immediate natural key question is how to acquire the training samples for the extra class (from a nearly infinite number of out-distribution samples) in order to cover properly the over-generalized regions induced by a naive CNN.
Instead of synthesizing artificial out-distribution samples using a hard-to-train generator (Lee. et al ICLR 2018, Jin et. al NIPS 2017, Yu et. al IJCAI 2017 ), we rather propose the use of a novel measurement for selecting a representative out-distribution dataset among the readily accessible natural ones for training an effective A-CNN. This not only maintains the accuracy on in-distribution samples but also results in identifying both adversaries and unseen out-distribution sets simultaneously.
Reference:
- Grosse, Kathrin, et al. "On the (statistical) detection of adversarial examples." arXiv preprint arXiv:1702.06280 (2017).
-Hosseini, Hossein, et al. "Blocking transferability of adversarial examples in black-box learning systems." arXiv preprint arXiv:1703.04318 (2017).
-Jin, Long, Justin Lazarow, and Zhuowen Tu. "Introspective classification with convolutional nets." Advances in Neural Information Processing Systems. 2017.
-Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detectingout-of-distribution samples. ICLR2018.
- Yu, Yang, et al. "Open-category classification by adversarial sample generation." Proceeding of the 26-th International Joint Conference on Artificial Intelligence (2017).

Meta-review:
role: AC
data: metareview: The reviewers agree the paper is not ready for publication at ICLR.

Decision: Reject