Controlling Over-generalization and its Effect on Adversarial Examples Detection and Generation

Convolutional Neural Networks ( CNNs ) significantly improve the state-of-the-art for many applications , especially in computer vision . However , CNNs still suffer from a tendency to confidently classify out-distribution samples from unknown classes into pre-defined known classes . Further , they are also vulnerable to adversarial examples . We are relating these two issues through the tendency of CNNs to over-generalize for areas of the input space not covered well by the training set . We show that a CNN augmented with an extra output class can act as a simple yet effective end-to-end model for controlling over-generalization . As an appropriate training set for the extra class , we introduce two resources that are computationally efficient to obtain : a representative natural out-distribution set and interpolated in-distribution samples . To help select a representative natural out-distribution set among available ones , we propose a simple measurement to assess an outdistribution set ’ s fitness . We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries . Finally , we show that generation of white-box adversarial attacks using our proposed augmented CNN can become harder , as the attack algorithms have to get around the rejection ( dustbin ) regions when generating actual adversaries . 1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) have allowed for significant improvements over the stateof-the-art in the last few years for various applications , and in particular for computer vision . Notwithstanding these successes , challenging issues remain with these models . In the following work , we specifically look at two concerns . First , CNNs are vulnerable to different types of adversarial examples ( Szegedy et al. , 2014 ; Kurakin et al. , 2016b ; Moosavi Dezfooli et al. , 2016 ; Carlini and Wagner , 2017b ) . These adversarial examples are created by deliberately modifying clean samples with imperceptible perturbations , with the aim of misleading CNNs into classifying them to a wrong class with high confidence . Second , CNNs are not able to handle instances coming from outside the task domain on which they are trained — the so-called out-distribution samples ( Liang et al. , 2017 ; Lakshminarayanan et al. , 2017 ) . In other words , although these examples are semantically and statistically different from the ( in-distribution ) samples relevant to a given task , the neural network trained on the task assigns such out-of-concept samples with high-confidence to the pre-defined in-distribution classes . Due to the susceptibility of CNNs to both adversaries and out-distribution samples , deploying them for real-world applications , in particular for security-sensitive ones , is a serious concern . These two issues have been treated separately in the past , with two distinct family of approaches . For instance , on the one hand , to handle out-distribution samples , some researchers have proposed threshold-based post-processing approaches with the aim of firstly calibrating the predictive confidence scores provided by either a single pre-trained CNN ( Liang et al. , 2017 ; Hendrycks and Gimpel , 2016 ; Lee et al. , 2017 ) or an ensemble of CNNs ( Lakshminarayanan et al. , 2017 ) , and then detecting out-distribution samples according to an optimal threshold . However , it is difficult to define an optimal and stable threshold for rejecting a wide range of out-distribution samples without increasing the false negative rate ( i.e. , rejecting in-distribution samples ) . On the other hand , researchers regarded adversarial examples as a distinct issue from the out-distribution problem and attempted to either correctly classify all adversaries through adversarial training of CNNs ( Tramèr et al. , 2017 ; Goodfellow et al. , 2015 ; Moosavi Dezfooli et al. , 2016 ) or reject all of them by training a separate detector ( Feinman et al. , 2017 ; Metzen et al. , 2017 ) . The performance of these approaches at properly handling adversarial instances mostly depends on having access to a diverse set of training adversaries , which is not only computationally expensive but also handling some possible future adversaries , which have not been discovered yet , most likely is difficult . It is known that deep neural networks ( e.g . CNNs ) are prone to over-generalization in the input space by partitioning it entirely into a set of pre-defined classes for a given in-distribution set ( task ) , regardless of the fact that in-distribution samples may only be relevant to a small portion of the input space ( Liang et al. , 2017 ; Spigler , 2017 ; Bendale and Boult , 2016 ) . In this paper , we highlight that the two aforementioned issues of CNNs can be alleviated simultaneously through control of over-generalization . To this end , we propose that an augmented CNN , a regular ( naive ) CNN with an extra class dubbed as dustbin , can be a simple yet effective solution , if it is trained on appropriate training samples for the dustbin class . Furthermore , we introduce here a computationally-efficient answer to the following key question : how to acquire such an appropriate set to effectively reduced the over-generalized regions induced by naive CNN . We note that our motivation for employing an augmented CNN is different from the threshold-based post-processing approaches that attempt to calibrate the predictive confidence scores of a pre-trained naive CNN without impacting its feature space . Our motivation in fact is to learn a more expressive feature space , where along with learning the sub-manifolds corresponding to in-distribution classes , a distinct extra sub-manifold for the dustbin class can be obtained such that the samples drawn from many over-generalized regions including a wide-range of out-distribution samples and various types of adversaries are mapped to this “ dustbin ” sub-manifold . As a training source for the extra class ( dustbin ) , one can consider using synthetically generated out-distribution samples ( Lee et al. , 2017 ; Jin et al. , 2017 ) or adversarial examples ( Grosse et al. , 2017 ) . However , using such generated samples is not only computationally expensive but also barely able to effectively reduce over-generalization compared to naive CNNs ( see Sec . 3 ) . Instead of such synthetic samples , there are plenty of cost-effective training sources available for the extra dustbin class , namely natural out-distribution datasets . By natural out-distribution sets we mean the sets containing some realistic ( not synthetically generated ) samples that are semantically and statistically different from those in the in-distribution set . A representative natural out-distribution set for a given in-distribution task should be able to adequately cover the over-generalized regions . To recognize such a representative natural set , we propose a simple measurement to assess its fitness for a given in-distribution set . In addition to the selected set , we generate some artificial out-distribution samples through a straightforward and computationally efficient procedure , namely by interpolating some pair of in-distribution samples . We believe a properly trained augmented CNN can be utilized as a threshold-free baseline for identifying concurrently a broad range of unseen out-distribution samples and different types of strong adversarial attacks . The main contributions of the paper are summarized as : • By limiting the over-generalization regions induced by naive CNNs , we are able to drastically reduce the risk of misclassifying both adversaries and samples from a broad range of ( unseen ) out-distribution sets . To this end , we demonstrate that an augmented CNN can act as a simple yet effective solution . • We introduce a measurement to select a representative natural out-distribution set among those available for training effective augmented CNNs , instead of synthesizing some dustbin samples using hard-to-train generators . • Based on extensive experiments on a range of different image classification tasks , we demonstrate that properly trained augmented CNNs can significantly reduce the misclassification rates for both 1 ) unseen out-distribution sets , and 2 ) for various types of strong black-box adversarial examples , even though they are never trained on any specific types of adversaries . • For the generation of white-box adversaries using our proposed augmented CNN , the adversarial attack algorithms frequently encounter dustbin regions rather than regions from other classes when distorting a clean samples , making the adversaries generation process more difficult . 2 PROPOSED METHOD . The key idea of this paper is to make use of a CNN augmented with a dustbin class , trained on a representative set of out-distribution samples , as a simple yet effective candidate solution to limit over-generalization . A visual illustration of this is given in Fig . 1 , which provides a schematic explanation of the influence of training samples used to learn the dustbin class on the out-distribution area coverage . This figure illustrates how the choice of training samples for the extra dustbin class plays a central role for achieving an effective augmented CNN . With a naive MLP ( no dustbin class ) , a decision boundary is separating the whole input space into two classes ( Fig . 1 ( a ) ) , working on the complete input space even in regions that are deemed irrelevant for the task at hand . As for augmented MLPs , the second plot ( Fig . 1 ( b ) ) shows results with dustbin training samples picked to be around the decision boundary , where many adversarial examples ( Moosavi Dezfooli et al. , 2016 ) are designed to be located . As it can be observed , such augmented MLP can only slightly reduce over-generalized regions . However , it might be able to classify some of adversaries as dustbin , and make generation of new adversaries harder since the adversarial attack algorithm should avoid the dustbin regions that are now located in between the two in-distribution classes . Thus , using solely such adversaries as training set of the dustbin class can not adequately cover the over-generalized regions . In another variation ( Fig . 1 ( c ) ) , the dustbin samples come from a out-distribution set , quite compact and located around the in-distribution samples from one specific class . Training an augmented MLP on this kind of out-distribution samples can not reduce over-generalization effectively . Accordingly , we argue that out-distribution training samples that are distributed uniformly w.r.t in-distribution classes can be regarded as a representative set for the extra dustbin class ( Fig . 1 ( d ) ) . Indeed , an augmented MLP trained on a representative set is able to classify a wide-range of unseen out-distribution sets and some of adversaries as its extra class , being more effective at controlling over-generalization . It is worth to note that coupling a representative out-distribution set with the samples drawn around the decision boundaries can further strengthen the augmented neural network against adversarial examples . There are many possible ways of acquiring some training samples for the extra class of augmented CNNs , ranging from artificially generated samples ( Jin et al. , 2017 ; Lee et al. , 2017 ) to natural available out-distribution sets . Instead of making use of a generator , which is computationally expensive and hard to train , we propose the use of two cost-effective resources for acquiring dustbin training samples in order to train effective augmented CNNs : i ) a selected representative natural out-distribution set and ii ) interpolated samples . 2.1 NATURAL OUT-DISTRIBUTION INSTANCES . A possible rich and readily accessible source of dustbin examples for training augmented models lies in natural out-distribution datasets . These sets contain natural samples that are statistically and semantically different compared to the samples of a given task . For example , NotMNIST and Omniglot datasets can be regarded as natural out-distribution sets when trying to classify MNIST digits . However , it is not clear how to select a sufficiently representative set from the ( possibly large ) corpus of available datasets in order to properly train augmented CNNs . We shed light on the selection of a representative natural out-distribution set by introducing a simple visualization metric . Specifically , we deem a natural out-distribution set as representative for a given in-distribution task if it is misclassified uniformly over the in-distribution classes . That is , if roughly an equal number of out-distribution samples are classified confidently as belonging to each of the in-distribution classes by the naive neural network . Accordingly , to assess the appropriateness of out-distribution sets for a given task ( or in-distribution set ) , we visualize the number of out-distribution samples that are misclassified to each of the in-distribution classes by using a histogram . In other words , a natural out-distribution set which has a more uniform misclassification distribution over the in-distribution classes appears better suited for training an effective augmented CNN . In Fig . 2 , the uniformity characteristics of SVHN vs CIFAR-100† as out-distribution sets for CIFAR10 , and LSUN vs DS-ImageNet† ( i.e . Down Scaled ImageNet ) for CIFAR-100 are shown1 . According to Fig . 2 ( a ) , most of SVHN samples are misclassified into a limited number of CIFAR-10 classes ( 5 classes out of 10 classes ) , while CIFAR-100† exhibits a relatively more uniform misclassifcation on CIFAR-10 classes . Therefore , compared with SVHN , we consider CIFAR-100 as a more representative natural out-distribution set for CIFAR-10 . A full comparison of these two out-distribution sets according to their ability to control over-generalization can be found in Table 3 of the Appendix . Similar behaviour can also be observed for LSUN vs DS-ImageNet as two out-distribution resources for training an augmented Resnet164 on CIFAR-100 ( as in-distribution ) . In this case DS-ImageNet† has a more uniform distribution when compared with LSUN . 1Throughout the paper , † indicates a modified out-distribution set by discarding the classes that have exact or semantic overlaps with the classes of its corresponding in-distribution set . For example , the super-classes of vehicle from CIFAR-100 is removed due to their semantic overlap with automobile and truck classes of CIFAR-10 . Refer to Appendix A for detail information .

Summary: The idea of having a separate class for out-distribution is a very interesting idea but unfortunately previously explored. In fact, in machine learning and NLP there is the OOV class which sometimes people in computer vision also use. Some of the claims in the paper can be further substantiated or explored. For example in abstract there is a simple claim that is presented too strong: We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. This claim is bigger than just CNNs and needs to be studied in a theoretical framework not an empirical one. Also, one simple way to stop these adversarial cases would be to explore using Sigmoid as opposed to softmax. In general it is very unlikely that you will be able to choose every variation of out-distribution cases. Much easier if you just try to solve the problem using a set of n Sigmoids (n total number of classes) and consider each output a probability distribution.