Title: CONTROLLING OVER-GENERALIZATION AND ITS EF-FECT ON ADVERSARIAL EXAMPLES GENERATION AND DETECTION

Abstract: Convolutional Neural Networks (CNNs) significantly improve the state-of-the-art for many applications, especially in computer vision. However, CNNs still suffer from a tendency to confidently classify out-distribution samples from unknown classes into pre-defined known classes. Further, they are also vulnerable to adversarial examples. We are relating these two issues through the tendency of CNNs to over-generalize for areas of the input space not covered well by the training set. We show that a CNN augmented with an extra output class can act as a simple yet effective end-to-end model for controlling over-generalization. As an appropriate training set for the extra class, we introduce two resources that are computationally efficient to obtain: a representative natural out-distribution set and interpolated in-distribution samples. To help select a representative natural out-distribution set among available ones, we propose a simple measurement to assess an outdistribution set's fitness. We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. Finally, we show that generation of white-box adversarial attacks using our proposed augmented CNN can become harder, as the attack algorithms have to get around the rejection (dustbin) regions when generating actual adversaries.

1. INTRODUCTION:

Convolutional Neural Networks (CNNs) have allowed for significant improvements over the stateof-the-art in the last few years for various applications, and in particular for computer vision. Notwithstanding these successes, challenging issues remain with these models. In the following work, we specifically look at two concerns. First, CNNs are vulnerable to different types of adversarial examples (Szegedy et al., 2014; Kurakin et al., 2016b; Moosavi Dezfooli et al., 2016; Carlini and Wagner, 2017b) . These adversarial examples are created by deliberately modifying clean samples with imperceptible perturbations, with the aim of misleading CNNs into classifying them to a wrong class with high confidence. Second, CNNs are not able to handle instances coming from outside the task domain on which they are trained -the so-called out-distribution samples (Liang et al., 2017; Lakshminarayanan et al., 2017) . In other words, although these examples are semantically and statistically different from the (in-distribution) samples relevant to a given task, the neural network trained on the task assigns such out-of-concept samples with high-confidence to the pre-defined in-distribution classes. Due to the susceptibility of CNNs to both adversaries and out-distribution samples, deploying them for real-world applications, in particular for security-sensitive ones, is a serious concern. These two issues have been treated separately in the past, with two distinct family of approaches. For instance, on the one hand, to handle out-distribution samples, some researchers have proposed threshold-based post-processing approaches with the aim of firstly calibrating the predictive confidence scores provided by either a single pre-trained CNN (Liang et al., 2017; Hendrycks and Gimpel, 2016; Lee et al., 2017) or an ensemble of CNNs (Lakshminarayanan et al., 2017) , and then detecting out-distribution samples according to an optimal threshold. However, it is difficult to define an optimal and stable threshold for rejecting a wide range of out-distribution samples without increasing the false negative rate (i.e., rejecting in-distribution samples). On the other hand, researchers regarded adversarial examples as a distinct issue from the out-distribution problem and attempted to either correctly classify all adversaries through adversarial training of CNNs (Tramèr et al., 2017; Goodfellow et al., 2015; Moosavi Dezfooli et al., 2016) or reject all of them by training a separate detector (Feinman et al., 2017; Metzen et al., 2017) . The performance of these approaches at properly handling adversarial instances mostly depends on having access to a diverse set of training adversaries, which is not only computationally expensive but also handling some possible future adversaries, which have not been discovered yet, most likely is difficult.

It is known that deep neural networks (e.g. CNNs) are prone to over-generalization in the input space by partitioning it entirely into a set of pre-defined classes for a given in-distribution set (task), regardless of the fact that in-distribution samples may only be relevant to a small portion of the input space (Liang et al., 2017; Spigler, 2017; Bendale and Boult, 2016) . In this paper, we highlight that the two aforementioned issues of CNNs can be alleviated simultaneously through control of over-generalization. To this end, we propose that an augmented CNN, a regular (naive) CNN with an extra class dubbed as dustbin, can be a simple yet effective solution, if it is trained on appropriate training samples for the dustbin class. Furthermore, we introduce here a computationally-efficient answer to the following key question: how to acquire such an appropriate set to effectively reduced the over-generalized regions induced by naive CNN. We note that our motivation for employing an augmented CNN is different from the threshold-based post-processing approaches that attempt to calibrate the predictive confidence scores of a pre-trained naive CNN without impacting its feature space. Our motivation in fact is to learn a more expressive feature space, where along with learning the sub-manifolds corresponding to in-distribution classes, a distinct extra sub-manifold for the dustbin class can be obtained such that the samples drawn from many over-generalized regions including a wide-range of out-distribution samples and various types of adversaries are mapped to this "dustbin" sub-manifold.

As a training source for the extra class (dustbin), one can consider using synthetically generated out-distribution samples (Lee et al., 2017; Jin et al., 2017) or adversarial examples (Grosse et al., 2017) . However, using such generated samples is not only computationally expensive but also barely able to effectively reduce over-generalization compared to naive CNNs (see Sec. 3). Instead of such synthetic samples, there are plenty of cost-effective training sources available for the extra dustbin class, namely natural out-distribution datasets. By natural out-distribution sets we mean the sets containing some realistic (not synthetically generated) samples that are semantically and statistically different from those in the in-distribution set. A representative natural out-distribution set for a given in-distribution task should be able to adequately cover the over-generalized regions. To recognize such a representative natural set, we propose a simple measurement to assess its fitness for a given in-distribution set. In addition to the selected set, we generate some artificial out-distribution samples through a straightforward and computationally efficient procedure, namely by interpolating some pair of in-distribution samples. We believe a properly trained augmented CNN can be utilized as a threshold-free baseline for identifying concurrently a broad range of unseen out-distribution samples and different types of strong adversarial attacks.

The main contributions of the paper are summarized as:

• By limiting the over-generalization regions induced by naive CNNs, we are able to drastically reduce the risk of misclassifying both adversaries and samples from a broad range of (unseen) out-distribution sets. To this end, we demonstrate that an augmented CNN can act as a simple yet effective solution.

• We introduce a measurement to select a representative natural out-distribution set among those available for training effective augmented CNNs, instead of synthesizing some dustbin samples using hard-to-train generators. both 1) unseen out-distribution sets, and 2) for various types of strong black-box adversarial examples, even though they are never trained on any specific types of adversaries. • For the generation of white-box adversaries using our proposed augmented CNN, the adversarial attack algorithms frequently encounter dustbin regions rather than regions from other classes when distorting a clean samples, making the adversaries generation process more difficult.

2. PROPOSED METHOD:

The key idea of this paper is to make use of a CNN augmented with a dustbin class, trained on a representative set of out-distribution samples, as a simple yet effective candidate solution to limit over-generalization. A visual illustration of this is given in Fig. 1 , which provides a schematic explanation of the influence of training samples used to learn the dustbin class on the out-distribution area coverage.

This figure illustrates how the choice of training samples for the extra dustbin class plays a central role for achieving an effective augmented CNN. With a naive MLP (no dustbin class), a decision boundary is separating the whole input space into two classes (Fig. 1 (a)), working on the complete input space even in regions that are deemed irrelevant for the task at hand. As for augmented MLPs, the second plot (Fig. 1(b )) shows results with dustbin training samples picked to be around the decision boundary, where many adversarial examples (Moosavi Dezfooli et al., 2016) are designed to be located. As it can be observed, such augmented MLP can only slightly reduce over-generalized regions. However, it might be able to classify some of adversaries as dustbin, and make generation of new adversaries harder since the adversarial attack algorithm should avoid the dustbin regions that are now located in between the two in-distribution classes. Thus, using solely such adversaries as training set of the dustbin class can not adequately cover the over-generalized regions. In another variation (Fig. 1(c )), the dustbin samples come from a out-distribution set, quite compact and located around the in-distribution samples from one specific class. Training an augmented MLP on this kind of out-distribution samples cannot reduce over-generalization effectively. Accordingly, we argue that out-distribution training samples that are distributed uniformly w.r.t in-distribution classes can be regarded as a representative set for the extra dustbin class (Fig. 1(d) ). Indeed, an augmented MLP trained on a representative set is able to classify a wide-range of unseen out-distribution sets and some of adversaries as its extra class, being more effective at controlling over-generalization. It is worth to note that coupling a representative out-distribution set with the samples drawn around the decision boundaries can further strengthen the augmented neural network against adversarial examples.

There are many possible ways of acquiring some training samples for the extra class of augmented CNNs, ranging from artificially generated samples (Jin et al., 2017; Lee et al., 2017) to natural available out-distribution sets. Instead of making use of a generator, which is computationally expensive and hard to train, we propose the use of two cost-effective resources for acquiring dustbin 0 1 2 3 4 5 6 7 8 9  0   5000   10000   0 1 2 3 4 5 6 7 8 9 training samples in order to train effective augmented CNNs: i) a selected representative natural out-distribution set and ii) interpolated samples.

2.1. NATURAL OUT-DISTRIBUTION INSTANCES:

A possible rich and readily accessible source of dustbin examples for training augmented models lies in natural out-distribution datasets. These sets contain natural samples that are statistically and semantically different compared to the samples of a given task. For example, NotMNIST and Omniglot datasets can be regarded as natural out-distribution sets when trying to classify MNIST digits. However, it is not clear how to select a sufficiently representative set from the (possibly large) corpus of available datasets in order to properly train augmented CNNs. We shed light on the selection of a representative natural out-distribution set by introducing a simple visualization metric. Specifically, we deem a natural out-distribution set as representative for a given in-distribution task if it is misclassified uniformly over the in-distribution classes. That is, if roughly an equal number of out-distribution samples are classified confidently as belonging to each of the in-distribution classes by the naive neural network. Accordingly, to assess the appropriateness of out-distribution sets for a given task (or in-distribution set), we visualize the number of out-distribution samples that are misclassified to each of the in-distribution classes by using a histogram. In other words, a natural out-distribution set which has a more uniform misclassification distribution over the in-distribution classes appears better suited for training an effective augmented CNN.

In Fig. 2 , the uniformity characteristics of SVHN vs CIFAR-100 † as out-distribution sets for CIFAR-10, and LSUN vs DS-ImageNet † (i.e. Down Scaled ImageNet) for CIFAR-100 are shown1 . According to Fig. 2(a) , most of SVHN samples are misclassified into a limited number of CIFAR-10 classes (5 classes out of 10 classes), while CIFAR-100 † exhibits a relatively more uniform misclassifcation on CIFAR-10 classes. Therefore, compared with SVHN, we consider CIFAR-100 as a more representative natural out-distribution set for CIFAR-10. A full comparison of these two out-distribution sets according to their ability to control over-generalization can be found in Table 3 of the Appendix. Similar behaviour can also be observed for LSUN vs DS-ImageNet as two out-distribution resources for training an augmented Resnet164 on CIFAR-100 (as in-distribution). In this case DS-ImageNet † has a more uniform distribution when compared with LSUN. Instead of generating adversarial examples for training, we propose an inexpensive and straightforward procedure for acquiring some samples around the decision boundaries. To this end, we interpolate some pairs of correctly classified in-distribution samples from different classes. An interpolated sample created from two samples with different classes aims to cover such regions (margins around decision boundaries) between two classes in order to assign them to out-distribution (dustbin) regions. Formally speaking, consider a pair of input images from two different classes of a K-classification problem, i.e. x i , x j ∈ R D i, j ∈ {1, ..., K}, where x j is the nearest neighbor of x i in the feature space of a CNN (its last convolution layer). An interpolated sample x ∈ R D is generated by making a linear combination of the given pair in the input space, x = α x i + (1 -α) x j . For all our experiments, we set α = 0.5. Some interpolated samples can be seen in Fig. 3 for MNIST and CIFAR-10. The reasons for finding the nearest neighbors in the feature space are twofold: computationally less expensive yet more accurate (Bengio, 2009) when compared to doing so in high-dimensional input space.

2.3. FEATURE SPACE OF AUGMENTED CNNS:

As an augmented CNN is trained in a end-to-end fashion, it allows learning of an extra sub-manifold corresponding to the added extra class (dustbin). Thus, if the augmented CNN is trained properly on a representative out-distribution set, it is able to map a large variety of out-distribution sets onto its extra sub-manifold, whether or not they have been seen during training. This should allow to learn a feature space that untangles the in-distribution set from the out-distribution samples. This is in contrast to the feature space of its naive counterpart, where the in-distribution and out-distribution samples are likely to be mixed or placed near each other.

Moreover, a proper trained augmented CNN is surprisingly able to map a large portion of black-box adversaries onto its extra manifold, even though it is never trained on any adversaries. Meanwhile some of the adversarial instances are mapped to their corresponding true class' sub-manifold. Therefore, this leads to a more engaging classifier for many practical situations (real-world applications) as some of adversaries are classified into dustbin (equivalent to the rejection option) while some of remaining ones are correctly classified as their true class (particularly non-transferable adversaries attacks, see Sec.3).

In Fig. 4 , we exhibit the feature spaces achieved from a naive CNN and its augmented counterpart for CIFAR-10 as an in-distribution task. Note CIFAR-100 is used as the training set for the extra class of the augmented CNN. As it can be visualized in Fig. 4 , the two out-distribution sets, including CIFAR-100 (green triangles) and Fast Gradient Sign (FGS) adversaries (Goodfellow et al., 2014) (shown with yellow triangles) are separated from CIFAR-10 samples in the feature space of the augmented CNN while they are mixed in the feature space of its naive counterpart. 

3. EVALUATION:

We conduct several experiments on three benchmarks, namely MNIST, CIFAR-10, and CIFAR-100 datasets, using three neural network architectures LeNet (LeCun et al., 1998) , VGG-16 (Simonyan and Zisserman, 2014) , and ResNet164 (He et al., 2016) . To assess robustness of the augmented versions of these CNNs , we consider five well-known strong attack algorithms: Fast Gradient Sign (FGS) (Goodfellow et al., 2014) , Iterative FGS (I-FGS) (Madry et al., 2017) , Targeted FGS (T-FGS) (Kurakin et al., 2016a) , DeepFool (Moosavi Dezfooli et al., 2016) , and C&W (Carlini and Wagner, 2017b) (see Appendix A.6 to learn about their hyper-parameter configurations). Note that we evaluate performance using three metrics: 1) accuracy (Acc.), which captures the rate or percentage of samples classified correctly as their true associated label; 2) rejection rate (Rej.), to measure the rate of samples correctly classified as dustbin (equivalent to rejection option); and 3) error rate (Err.), which captures the rate of samples that are neither correctly classified nor rejected.

3.1. BLACK-BOX ADVERSARIAL EXAMPLES:

It is widely known that many of adversarial examples generated from a learning model (e.g., CNN) can be transferred to attack other victim models (Papernot et al., 2017; Szegedy et al., 2014; Carlini and Wagner, 2017a ) -such attacks are called transferable black-box attacks. To evaluate robustness of the augmented CNNs on the aforementioned types of attacks generated in black-box setting, we generate adversarial samples corresponding to correctly classified clean test samples using a naive CNN, trained with different initial weights compared to the one under evaluation. Moreover, in order to demonstrate the influence of using different out-distribution sets for training the extra class on identifying adversaries, we employ four different sources for acquiring dustbin training samples: 1) adversarial samples generated by I-FGS; 2) only interpolated in-distribution data; 3) only a representative natural out-distribution set (selected according our proposed metric); and 4) both interpolated samples along with a representative natural out-distribution set (selected according our proposed metric).

To evaluate the generalization performance of the augmented CNNs on the in-distribution tasks, the in-distribution test accuracy rates are presented in Table 1 . Compared to the naive CNNs, we observe a slight drop in test accuracy rates of their augmented counterparts (except for that trained on I-FGS adversaries) while, interestingly, having also the error rates (i.e., the number of wrong decisions) reduced, leading to less error in decision making. This property can be highly beneficial for some security-sensitive applications, where making less error in some critical situations is vital.

For the augmented CNNs, rejection rate (i.e., assignments to dustbin) is reported in addition to accuracy (i.e., correct classifications) and error rates (i.e., misclassifications)2 . Comparing the augmented CNNs in A ). Accordingly, we emphasize that using the samples drawn from the vicinity of decision boundaries such as I-FGS adversaries as a single training source for the extra dustbin class of augmented CNN can not effectively control over-generalization. Contrary to I-FGS augmented CNN, augmented CNNs trained on a representative natural out-distribution set (selected according to our proposed metric) along with some interpolated samples consistently outperform their naive counterparts and the other augmented CNNs by achieving a drastic drop in error (misclassification) rates on all variants of adversaries, even though these augmented CNNs are not trained on any specific type of adversaries. This illustrates that if an augmented CNN is trained on a representative outdistribution set along with some interpolated samples, it can efficiently reduce over-generalization, resulting in generally well-performing model in the case of adversaries and various natural out-distribution samples. Due to space limitation, we place some results on the augmented CNNs trained with non-representative natural out-distribution sets in the Appendix A in Table 3 for illustrating the deficiency of such sets in controlling over-generalization.

To visualize and compare the classification regions in input space of our augmented CNN and its naive counterpart, we plot several church-windows (cross-sections) (Warde-Farley and Goodfellow, 2016) in Fig. 5 . The x-axis of each window is the adversary direction achieved by FGS or DeepFool using the naive network. For each adversary direction, we plot four windows by taking four random directions that are perpendicular to the given adversary direction (x-axis). As it can be observed, the fooling classification regions (spanned by the adversary direction and one of its orthogonal random directions) of the naive CNNs are occupied by dustbin regions (indicated by orange) in their augmented counterparts. 

3.2. WHITE-BOX ADVERSARIAL EXAMPLES:

White-box adversarial examples are generated by using directly the model on which they are applied. We further evaluate the robustness of our augmented CNNs on different types of white-box attacks, using the same parameter configurations as with the black-box experiments. For this purpose, we compute the percentage of visiting fooling classes (i.e., the classes different from dustbin and the true class associated to the clean samples) and the dustbin class when moving in the direction given by an attack method for a set of clean samples. Note that for generating some authentic white-box adversaries by the augmented CNNs, the attack algorithm should avoid dustbin regions to preclude generation of useless adversaries (those already recognizable as dustbin by the augmented CNN). In addition, the percentage of visiting the dustbin class when moving in a "legitimate" direction is also reported. By this we mean moving from a given sample x to its nearest neighbor x from the same class in the direction of their convex combination (1 -) x + x . Results for legitimate directions are computed with varying ∈ [0.1, 0.5].

To generate white-box adversaries using both a naive CNN and its augmented counterpart, MNIST and CIFAR-10 test sets are utilized. As seen in Fig. 6 , adversaries generated for the augmented CNNs (trained on a representative natural out-distribution set) encounter more often the dustbin class than a fooling class, indicating that generation of white-box adversaries using the augmented CNNs becomes harder. An adversarial algorithm needs to skip over some regions assigned to dustbin class, leading to a possible increase in the number of steps or the amount of distortions required for generating adversaries. Moreover, by moving in legitimate directions, the augmented CNNs appear to remain largely in the current true classes.

3.3. OUT-DISTRIBUTION SAMPLES:

The behavior of augmented CNNs is evaluated on several out-distribution sets across different indistribution tasks. For each in-distribution task, we consider several natural out-distribution datasets, both seen and unseen during the training of the augmented CNN. For comparison purposes, the rejection rates of two recent threshold-based approaches, including ODIN (Liang et al., 2017) , and Calibrated CNN (Lee et al., 2017) are considered3 . These approaches attempt to identify and reject out-distribution samples according to a specific threshold on the calibrated predictive confidence scores. For a fair comparison, the rejection rates (i.e. True Negative Rate) of these approaches are reported at the same True Positive Rates (TPR) as ours, where TPRs are considered 99%, 91%, and 95% for MNIST, CIFAR-10, and CIFAR-100, respectively.4 .

Moreover, for all ODIN experiments, we consider T = 1000 and ∈ {0, 5 × 10 -6 , 5 × 10 -5 , 5 × 10 -4 , 5 × 10 -3 , 1 × 10 -3 } is tuned for each pair of in-distribution and out-distribution validation sets such that the highest possible TNR at the specified TPR can be achieved. For "calibrated CNN" approach, its hyper-parameter (β > 0), which controls the effect of having the calibrated (uniform) predictions on the synthesized out-distribution samples, is tuned such that training of the calibrated CNN can be converged on the given in-distribution training set. We observe while the larger beta for MNIST and CIFAR-10 lead to better calibrated CNN on out-distribution samples, such a large beta for CIFAR-100 does not allow its training to converge. Considering this trade-off between calibration and convergence, in our experiments, the values of β are regarded 1 and 0.01 for CIFAR-10/MNIST and CIFAR100, respectively.

Table 2 compares the rejection rates (i.e TNR) of ours with ODIN and "calibrated CNN" as well as the error rates of naive and the augmented CNNs (trained on a representative natural out-distribution set and interpolated samples), where the error rate measures the number of the out-distribution samples classified with confidence higher than 50% as one of the in-distribution classes. These error rates by naive CNN aim to show the fact that a significant portion of out-distribution samples are confidently (confidence> 50%) misclassified by naive CNN. As it can be seen in Table 2 , the augmented CNNs, which is trained on one single but representative natural out-distribution set (as well as interpolated samples), almost outperforms "calibrated CNN", which is trained on a set of synthetic out-distribution samples, and ODIN, which its hyper-parameter is tuned for each pair of in-distribution and outdistribution validation set. It can demonstrate how controlling effectively over-generalization can lead to developing more robust CNNs in the presence of novel unseen out-distribution sets.

4. CONCLUSION:

In this paper we bridge two issues of CNNs that were previously thought of as unrelated: susceptibility of naive CNNs to various types of adversarial examples and incorrect high confidence prediction for out-distribution samples. We argue these two issues are connected through over-generalization. We propose augmented CNNs as a simple yet effective solution for controlling over-generalization, when they are trained on an appropriate set of dustbin samples. Through empirical evidence, we define an indicator for selecting an "appropriate" natural out-distribution set as training samples for dustbin class from among those available and show such selection plays a vital role for training effective augmented CNNs. Through extensive experiments on several augmented CNNs in different settings, we demonstrate that reducing over-generalization can significantly reduce the misclassification error rates of CNNs on adversaries and out-distribution samples, simultaneously, while their accuracy rates on in-distribution samples are maintained. Indeed, reducing over-generalization by such an end-to-end learning model (e.g., augmented CNNs) leads to learning more expressive feature space where these two categories of hostile samples (i.e., adversaries and out-distribution samples) are disentangled from in-distribution samples.

A APPENDIX A.1 DETAILS ON DATASETS AND EXPERIMENTS:

MNIST with NotMNIST MNIST consists of gray scale images of hand-written digits (0-9) and is made of 60k and 10k samples for training and testing, respectively. NotMNIST dataset5 , which involves 18,724 letters (A-J) printed with different font styles, is used as a source of out-distribution samples for MNIST. Images of both MNIST and NotMNIST datasets have the same size (28 × 28 pixels), with all pixels scaled in [0, 1]. LeNet, the CNN model we used comprised three convolution layers of 32, 32, and 64 filters (5 × 5), respectively, and one Fully Connected (FC) layer with softmax activation function6 . In addition, dropout with p = 0.5 is used on the FC layer for regularization. The augmented version of LeNet is trained with the 50k samples of MNIST, 10K randomly selected samples from NotMNIST for out-distribution samples and 15K interpolated samples (see Section2) generated from MNIST training samples. The remaining samples from NotMNIST (≈8K) are used together with MNIST test samples to evaluate the augmented CNN.

CIFAR-10 with CIFAR-100 † CIFAR-10 and CIFAR-100 represents low-resolution RGB images (32×32) of objects. CIFAR-10 contains 50k training and 10k testing instances over 10 classes. CIFAR-100 has the same characteristics except it is organized into 100 classes. For the experiments with CIFAR-10, out-distribution samples are taken from CIFAR-100 † . To avoid the semantic overlaps between the labels of CIFAR-10 and CIFAR-100, super-classes of CIFAR-100 conceptually similar to those of CIFAR-10 are ignored (i.e., vehicle 1, vehicle 2, medium-sized mammals, small mammals, and large carnivores excluded from CIFAR-100). Pixels are scaled in [0, 1], and then normalized by subtracting the mean of the image of the CIFAR-10 training set. VGG-16 (Simonyan and Zisserman, 2014) is used as CNN architecture for CIFAR-10, which has has 13 convolution layers of 3 × 3 filters and three FC layers. To train the augmented VGG-16, 15k samples are selected from CIFAR-100 † along with 15k interpolated samples from CIFAR-10 training set (both labeled as dustbin) and are appended to the CIFAR-10 training set.

CIFAR-100 with DS-ImageNet † Similar to CIFAR-10, training and test sets of CIFAR-100 contain 50K and 10K RGB images (32×32 pixels each). As out-distribution samples for CIFAR-100, we utilized down-scaled version of ImageNet dataset (called DS-ImageNet) (Chrabaszcz et al., 2017) (images are scaled to 32 × 32). To choose proper out-distribution samples for CIFAR-100, we utilized the samples from 62 classes of DS-ImageNet that have less conceptual overlap with CIFAR-100 labels. For creating the training set for out-distribution dustbin class, samples from those 62 classes are taken from training set of DS-ImageNet. Therefore, this training set has 79,856 samples in total, but we randomly selected 15K of them along with 15K interpolated samples to train our augmented CNNs. We utilized validation set of DS-ImageNet containing 50K images as the test out-distribution task. We use ResNet-164 (He et al., 2016) to train an augmented CNN on CIFAR-100 (as in-distribution task).

A.2 SELECTING NON REPRESENTATIVE OUT-DISTRIBUTION SET:

For a given in-distribution dataset, there are many possible candidate out-distribution datasets for training an augmented CNN. We argued in section 2 that an non-representative natural out-distribution set, can not effectively handle over-generalization. According to our measurement, a non-representative natural out-distribution set is the one that are only miscalssified as a limited-number of in-distribution classes by a nive CNN. Recall that, according to Fig. 2 (a), most of SVHN samples are misclassified into a limited number of CIFAR-10 classes (5 classes out of 10 classes) by the naive CNN, while CIFAR-100 † dataset exhibits a relatively more uniform misclassifcation on CIFAR-10 classes. Therefore, compared to SVHN, we consider CIFAR-100 † as a more representative natural out-distribution set for CIFAR-10. Similarly, for CIFAR-100, DS-ImageNet † dataset is more uniformly misclassified when compared with LSUN and is thus considered a more appropriate out-distribution dataset.

In Table 3 , we compare two types of out-distribution sets, representative vs non-representative across two classification tasks, and showing choosing a representative out-distribution set is a key factor for effectively reducing over-generalization such that a wide-range of other unseen out-distribution samples and adversarial examples can be confidently classified as dustbin (equivalent to rejection).

In comparison to the augmented VGG used SVHN, the augmented VGG-16 trained on CIFAR-100 † as outdistribution training samples performs significantly better at rejecting both adversaries and unseen out-distribution samples (Table 3 ). Similarly, when comparing two augmented Resnets (for CIFAR-100 as in-distribution), the one trained with LSUN as the source of out-distribution samples is less effective in reducing over-generalization when comparison to the other Resnet trained with DS-ImageNet † as the source of out-distribution samples. 1 (having the same architecture as our augmented CNNs except the number of outputs) are considered and the optimal value for the hyper-parameters of ODIN, i.e. ∈ {0, 5 × 10 -6 , 5 × 10 -5 , 5 × 10 -4 , 5 × 10 -3 , 1 × 10 -3 } and T = 1000 are utilized. The optimal value for is chosen separately for each set of black-box adversaries. For a fair comparison with ours, the ODIN results are reported at True Positive Rate (TPR) 91% and 95% for CIFAR-10 and CIFAR-100, respectively. As it can be seen from Table 5 , the error rates of our augmented CNN on most of the various types of black-box adversaries are lower than ODIN due to its higher adversaries rejection (classifying as dustbin) rates. For example, our method on I-FGS adversaries of CIFAR-100, which are highly transferable (it can be perceived from naive ResNet-164's low accuracy on I-FGS (i.e. 22.20% from Table 1 )), has error rate 37.45% as it rejects 45.75% of this I-FGS adversaries, while ODIN rejects only 5.75% of adversaries, resulting to error rate 73.7%. As seen in Figure 7 , we find that the over-generalization reduction leads to a more expressive feature space where all natural out-distribution samples along with many black-box adversarial examples are separated from in-distribution samples to be classified as belonging to the dustbin class. Further, some adversarial instances are even placed very close to their corresponding true class, leading the augmented CNNs to classify them correctly.

A.6 ADVERSARIAL GENERATION METHODS:

Generally, an adversarial generation method can either be targeted or untargeted. In targeted attacks, an adversary aims to generate an adversarial sample that makes a victim CNN misclassify it to an adversary selected target class (i.e., arg max F (x + δ) = y , where y is the targeted class and = y * the actual class). In an untargeted attack, an adversary aims to make the victim CNN to simply misclassify perturbed image to a class other than the true label (i.e., arg max F (x + δ) = y * , where y * is the true class). Here, we briefly explain some well-known targeted and untargeted attack algorithms. Targeted Fast Gradient Sign (T-FGS) (Kurakin et al., 2016a) : This targeted attack method tends to modify a clean image x so that the loss function is minimized for a given pair of (x, y ), where target class y is different from the input's true label (y = y * ). To this end, it uses the sign of gradient of loss function as follows:

x adv = x -.sign(∇J(F (x, θ), y )),

where J(F (x, θ), y ) is the loss function and as the hyper-parameter controls the amount of distortion. The transferability of T-FGS samples increases by utilizing larger at the cost of adding more distortions to the image. Moreover, the untargeted variant of this method called FGS (Goodfellow et al., 2014) is as follows:

x adv = x + .sign(∇J(F (x, θ), y * )).

(2)

Iterative Fast Gradient Sign (I-FGS) (Kurakin et al., 2017) : This method actually is an iterative variant of Fast Gradient Sign (also called Projected Gradient Descent (PGD) Madry et al. (2017) ), where iteratively small amount of FGs perturbation is added by using a small value for . To keep the perturbed sample in α-neighborhood of x, the achieved adversary sample in iteration should be clipped.

x 0 adv = x x k+1 adv = clip x,α {x k adv + .sign(∇J(F (x k adv , θ), y * ))},

Compared to FGS, I-FGS generates more optimal distortions.

DeepFool (Moosavi Dezfooli et al., 2016) : This algorithm is an iterative but fast approach for creation of untargeted attacks with very small amount of perturbations. Indeed, DeepFool generates sub-optimal perturbation for each sample where the perturbation is designed to transfer the clean sample across its nearest decision boundary.

Carlini Attack (C&W) (Carlini and Wagner, 2017b ): Unlike previous proposed methods which find the adversarial examples over loss functions of CNN, this method defines a different objective function which tends to optimize misclassification as follows:

EQUATION

Here Z(x) is the output of last fully connected (before softmax) layer and x is perturbed image x. Also κ denotes confidence parameter. A larger value for κ leads the CNN to misclassify the input more confidently, however it also makes finding adversarial examples satisfying the condition (having high misclassification confidence) difficult.

Hyper-parameters of Attack Algorithms: Each adversarial generation algorithm has a few hyperparameters as previously seen. We provide details on the hyper-parameters used in our experimental evaluation in Table 6 . To generate targeted Carlini attack (called C&W) (Carlini and Wagner, 2017b) , we used the authors' github code. Due to large time complexity of C&W, we considered 100 randomly selected images for each dataset. For each selected image, as was done in previous work (Xu et al., 2018) , two targeted adversarial samples are generated, where the target classes are the least likely and second most likely classes according to the predictions provided by the underlying CNN. Thus, in total 200 C&W adversarial examples are generated per dataset. To increase transferability of C&W, we utilized κ = 20 for MNIST and κ = 10 for CIFAR-10. For CIFAR-100, we used the same setting used for CIFAR-10 except for C&W, we used higher value for κ(= 20).

For other attacks (variants of FGS and DeepFool), we utilized 2K correctly classified test samples. 

Dataset:



:

Throughout the paper, † indicates a modified out-distribution set by discarding the classes that have exact or semantic overlaps with the classes of its corresponding in-distribution set. For example, the super-classes of vehicle from CIFAR-100 is removed due to their semantic overlap with automobile and truck classes of CIFAR-10. Refer to Appendix A for detail information.

From Table1we can obtain out-distribution detection performance, with true positive rate as (in-distribution accuracy + error rate), false positive rate as (1 -out-distribution rejection rate), and false negative as indistribution rejection rate. As the proposed approach is threshold-free, ROC curves and related measurements (e.g., AUC-ROC) are irrelevant.

The code made available on Github by the authors of those papers is used for our evaluation.

Note as our approach is independent of a threshold, we have only one fixed TPR for a given in-distribution set, which is computed from(Acc in-dist. + err. in-dist)

Available at http://yaroslavvb.blogspot.ca/2011/09/notmnist-dataset.html.

Details on the configuration are given in https://github.com/dnouri/cuda-convnet/blob/ master/example-layers/layers-18pct.cfg.

Figures and Tables:

Figure 1: Illustration of the influence of different sets of dustbin training samples on the overgeneralized regions. Two-moon classification dataset: (a) naive MLP trained only with in-distribution samples, (b-d) augmented MLPs trained with different out-distribution sets as dustbin. The MLP is made of three layers and ReLU activation functions.

Figure 2: Misclassification distribution over original classes: (a) CIFAR-100 † vs SVHN provided by a naive VGG trained on CIFAR-10; (b) DS-ImageNet † vs LSUN provided by a naive Resnet-164 trained on CIFAR-100.

Figure 3: Interpolated samples for CIFAR10 and MNIST. Third row for every dataset represents the interpolated samples that are composed of images from first (source) and second rows (target).

Figure 4: Visualization of data distribution in last convolution layer (i.e., feature space) of an augmented CNN trained on CIFAR-10 and CIFAR-100 as in-distribution and out-distribution sets, respectively. For visualization purposes, these feature spaces are reduced to 3D using PCA. For more results, refer to Fig 7 of the Appendix.

Figure 5: Church window plots for various data instances. Black dot corresponds to the clean sample position.

Figure 7: Visualization of some randomly selected test samples and their corresponding adversaries (FGS and T-FGS) in the feature spaces (the penultimate layer) learned by a naive CNN and an augmented CNN. To produce and manipulate the 3D plots refer to https://github.com/ mahdaneh/Out-distribution-learning_FSvisulization

003, α = 0.03, # of iterations = 20 C&W κ = 20

Table 1 across different classification tasks, we can find that the augmented Results for black-box adversaries attacks on three classification tasks. Values with * denotes best accuracy while boldface denotes lowest misclassification rate for each given dataset and attack method.CNNs trained on a set of I-FGS adversaries can reject (classifying as dustbin) almost all test variants of FGS adversaries (i.e., FGS, I-FGS and T-FGS), however they fail to reject non-FGS variants of adversaries (e.g., C&W and DeepFool), as well as the natural out-distribution sets (see Table4of Appendix

Table content: <table><tr><td>Augmented Models</td></tr></table>



Table content: <table><tr><td>Probability of visiting a class</td><td>0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0</td><td>DF</td><td>FGS</td><td>T-FGS</td><td colspan="2">I-FGS Legitimate</td><td>0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0</td><td>DF</td><td>FGS</td><td>T-FGS</td><td colspan="2">I-FGS Legitimate</td><td>True class Fooling class Dustbin Naive CNN</td></tr><tr><td colspan="13">Figure 6: Robustness of naive CNNs (LeNet for MNIST (left) and VGG-16 for CIFAR-10 (right)) and</td></tr><tr><td colspan="13">their augmented counterparts under different adversarial attack algorithms. Robustness to white-box</td></tr><tr><td colspan="13">attacks is measured by the percentage of visiting fooling classes and dustbin class by moving in</td></tr><tr><td colspan="5">adversarial directions.</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="2">Naive model</td><td colspan="3">Augmented model</td><td>ODIN</td><td>Calibrated CNN</td></tr><tr><td colspan="6">In-distribution train Out-distribution test</td><td colspan="2">Error (%)</td><td colspan="4">Error (%) Rejection (%) Rejection (%)</td><td>Rejection (%)</td></tr><tr><td/><td/><td/><td/><td colspan="2">NotMNIST (seen)</td><td colspan="2">93.15</td><td>0.01</td><td colspan="2">99.98</td><td>61.34</td><td>100</td></tr><tr><td colspan="3">MNIST</td><td/><td colspan="2">Omniglot  *  (unseen)</td><td colspan="2">95.19</td><td>0.00</td><td>100</td><td/><td>96.70</td><td>100</td></tr><tr><td/><td/><td/><td/><td colspan="2">CIFAR-10(gc) (unseen)</td><td colspan="2">64.26</td><td>0.00</td><td>100</td><td/><td>99.51</td><td>99.8</td></tr><tr><td/><td/><td/><td/><td colspan="2">CIFAR-100  † (seen)</td><td colspan="2">97.05</td><td>3.71</td><td>96.21</td><td/><td>48.1</td><td>29.64</td></tr><tr><td colspan="3">CIFAR-10</td><td/><td colspan="2">DS-ImageNet  † (unseen) SVHN (unseen)</td><td colspan="2">96.62 95.56</td><td>12.20 7.61</td><td>87.49 92.29</td><td/><td>54.27 60.68</td><td>79.63 57.42</td></tr><tr><td/><td/><td/><td/><td colspan="2">LSUN  *  (unseen)</td><td colspan="2">96.12</td><td>14.31</td><td>84.80</td><td/><td>69.26</td><td>71.5</td></tr><tr><td/><td/><td/><td/><td colspan="2">DS-ImageNet  † (seen)</td><td colspan="2">79.34</td><td>1.52</td><td>98.35</td><td/><td>32.06</td><td>15.77</td></tr><tr><td colspan="3">CIFAR-100</td><td/><td colspan="2">SVHN (unseen)</td><td colspan="2">81.19</td><td>67.75</td><td colspan="2">16.25</td><td>51.65</td><td>7.5</td></tr><tr><td/><td/><td/><td/><td colspan="2">LSUN  *  (unseen)</td><td colspan="2">96.12</td><td>0.01</td><td>99.99</td><td/><td>41.16</td><td>13.7</td></tr></table>

Comparing the performance of augmented CNNs trained on a representative v.s. nonrepresentative out-distribution sets.A.3 ADVERSARIAL TRAINING OF AUGMENTED CNNHere, we provide more results on the performance of augmented CNNs trained on the adversarial (I-FGS) samples labeled as dustbin to reject out-distribution samples. Although such augmented CNNs can reject perfectly the variants of FGS adversaries, they are not able to significantly reduce over-generalization as their error rates on a wide-range of out-distribution samples are considerably high (e.g., 94.77% error rate for CIFAR-100 images on CIFAR-10 classifier).

Table content: <table><tr><td/><td/><td/><td colspan="2">CIFAR-10 (VGG-16)</td><td colspan="2">CIFAR-100 (Resnet-164 )</td></tr><tr><td/><td>Out-dist. Training -&gt;</td><td colspan="2">SVHN</td><td>CIFAR-100  †</td><td>LSUN</td><td>DS-ImageNet  †</td></tr><tr><td>In-dist. Test</td><td/><td>Acc. Rej. Err.</td><td>91.71 0.07 8.22</td><td>88.58 5.69 5.73</td><td>74.68 0.03 25.29</td><td>74.75 0.93 24.32</td></tr><tr><td>Out-dist. Test</td><td>SVHN LSUN CIFAR-100  † DS-ImageNet  †</td><td>Rej. Err. Rej. Err. Rej. Err. Rej. Err.</td><td>99.94 0.06 0 92.71 3.82 92.27 0.55 95.36</td><td>93.44 6.36 85.72 13.70 95.30 4.52 65.00 33.93</td><td>0 81.46 100 0 --28.76 55.33</td><td>23.61 63.65 100 0 --97.37 2.46</td></tr><tr><td/><td/><td>Acc.</td><td>37.65</td><td>31.90</td><td>70.28</td><td>50.15</td></tr><tr><td/><td>FGS</td><td>Rej.</td><td>0.0</td><td>36.81</td><td>2.76</td><td>33.85</td></tr><tr><td/><td/><td>Err.</td><td>62.35</td><td>31.29</td><td>26.96</td><td>16</td></tr><tr><td/><td/><td>Acc.</td><td>52.50</td><td>54.27</td><td>29.75</td><td>16.90</td></tr><tr><td>Adversarial Examples</td><td>I-FGS T-FGS DeepFool</td><td>Rej. Err. Acc. Rej. Err. Acc. Rej. Err. Acc.</td><td>0.0 47.50 31.28 0.05 68.67 53.88 0.05 46.07 47.00</td><td>12.94 32.79 27.17 41.08 31.75 44.22 33.20 22.58 46.50</td><td>1.85 68.4 60.48 4.30 35.22 75.73 1.05 23.22 75.50</td><td>32.65 50.45 37.87 44.12 18.01 67.12 10.56 22.32 66.00</td></tr><tr><td/><td>C&amp;W (L2)</td><td>Rej.</td><td>7.00</td><td>18.50</td><td>0.5</td><td>16.50</td></tr><tr><td/><td/><td>Err.</td><td>46</td><td>35</td><td>24</td><td>17.5</td></tr><tr><td/><td/><td/><td/><td colspan="3">Augmented CNN (I-FGS)</td></tr><tr><td colspan="4">In-distribution train Out-distribution test</td><td colspan="2">Error (%)</td><td>Rejection (%)</td></tr><tr><td/><td colspan="3">NotMNIST (unseen)</td><td/><td>18.31</td><td>80.75</td></tr><tr><td>MNIST</td><td colspan="2">Omniglot (unseen)</td><td/><td/><td>0</td><td>100.0</td></tr><tr><td/><td colspan="3">CIFAR-10(gc) (unseen)</td><td/><td>0.09</td><td>99.89</td></tr><tr><td/><td colspan="3">CIFAR-100  † (unseen)</td><td/><td>94.77</td><td>0.82</td></tr><tr><td>CIFAR-10</td><td colspan="3">DS-ImageNet  † (unseen) SVHN (unseen)</td><td/><td>82.93 94.56</td><td>12.08 0.04</td></tr><tr><td/><td colspan="2">LSUN (unseen)</td><td/><td/><td>75.46</td><td>15.90</td></tr><tr><td/><td colspan="3">DS-ImageNet  † (unseen)</td><td/><td>67.30</td><td>12.11</td></tr><tr><td>CIFAR-100</td><td colspan="2">SVHN (unseen)</td><td/><td/><td>82.51</td><td>0.09</td></tr><tr><td/><td colspan="2">LSUN (unseen)</td><td/><td/><td>47.05</td><td>26.28</td></tr><tr><td colspan="7">Table 4: The performance of adversarial training of augmented CNN on a wide-range of unseen</td></tr><tr><td colspan="2">adversarial examples.</td><td/><td/><td/><td/><td/></tr><tr><td colspan="5">A.4 ODIN PERFORMANCE ON BLACK-BOX ADVERSARIES</td><td/><td/></tr></table>

The comparison of our approach with ODIN on black-box adversarial examples A.5 VISUALIZATION OF FEATURE SPACE

Table content: <table/>

Adversarial generation methods' hyper parameters

Table content: <table/>

Peer review (iclr2019):
 The idea of having a separate class for out-distribution is a very interesting idea but unfortunately previously explored. In fact, in machine learning and NLP there is the OOV class which sometimes people in computer vision also use. Some of the claims in the paper can be further substantiated or explored. For example in abstract there is a simple claim that is presented too strong: We also demonstrate that training such an augmented CNN with representative out-distribution natural datasets and some interpolated samples allows it to better handle a wide range of unseen out-distribution samples and black-box adversarial examples without training it on any adversaries. This claim is bigger than just CNNs and needs to be studied in a theoretical framework not an empirical one. Also, one simple way to stop these adversarial cases would be to explore using Sigmoid as opposed to softmax. In general it is very unlikely that you will be able to choose every variation of out-distribution cases. Much easier if you just try to solve the problem using a set of n Sigmoids (n total number of classes) and consider each output a probability distribution. However, the studies in this paper are still valuable and I strongly recommend continuing on the same direction.