How the Softmax Activation Hinders the Detection of Adversarial and Out-of-Distribution Examples in Neural Networks

1 INTRODUCTION . Thanks to their excellent performances , Neural Networks ( NN ) are now used to tackle important problems such as medical diagnosis ( Shen et al. , 2017 ) or pedestrian detection for autonomous cars ( Tian et al. , 2015 ) . However , regarding classification , one of the issues that prevent widespread adoption of such solutions is their overconfidence in their predictions ( Amodei et al. , 2016 ) . It has been observed that they can provide predictions with high confidence values for errors ( Guo et al. , 2017 ) , out-of-distribution ( OOD ) examples such as tailored noise ( Nguyen et al. , 2015 ) and adversarial examples ( Szegedy et al. , 2014 ) . The softmax layer has been identified as one potential culprit for these overconfident predictions ( Hendrycks & Gimpel , 2017 ; Neumann et al. , 2018 ) . The softmax function is defined as follows : σ ( zi ) = ezi∑C j=1 e zj for i = 1 , ... , C and z ∈ RC Hendrycks & Gimpel ( 2017 ) state that overconfident prediction are due to the use of the fast-growing exponential function in the softmax computation . It is thus designed to output high maximum values , even for small differences in the logits z . Solutions have been proposed for this specific issue , such as temperature scaling , which divides the logits z by a constant T > 1 in order to obtain ” softer ” probabilities . Several works showed that this method significantly improves the calibration of the probabilities ( Guo et al. , 2017 ; Neumann et al. , 2018 ) . However , another property of the softmax , which is less discussed , is that the outputs are normalized in order to obtain a probability distribution . These probabilities are indeed easier to interpret than logits , as humans can process probabilities intuitively ( Cosmides & Tooby , 1996 ) . However , the information about the logits absolute values is lost due to this normalization . To the best of our knowledge , only the very recent work of Ozbulak et al . ( 2019 ) adress this issue . In their work , they state that some adversarial examples , referred to as ” over-optimized ” adversarial examples , have very high logit values , but their magnitude is masked by the softmax function . They thus use this difference in logit distribution to detect this type of adversarial examples . Our work is aligned with theirs and extends their findings , as we show that logits can be used not only to detect over-optimized adversarial examples , but also ” regular ” adversarial examples and OOD data . We also propose new experiments to quantify the impact of the loss of the information contained in the logits on detection performances . Our main contrutions are : • We provide some intuition about the difference between logit and softmax distributions through experiments on MNIST . We show that there is a significant difference in logit distributions between regular MNIST images and OOD and adversarial images , but this difference is not present to the same extent after the softmax function is applied . • We propose 2 logit-based solutions inspired by state-of-the-art methods , each being suited for a different scenario : a NN trained on logit activations when the adversarial attacks and OOD datasets are known beforehand , and a kernel density estimation of the logit distribution when they are not . We achieve competitive performances for the detection of 4 kinds of adversarial attacks and 4 OOD datasets . These results remains true for 2 datasets , MNIST and CIFAR10 , and 2 NN architectures , a custom , relatively shallow one and a Wide Residual Network ( Zagoruyko & Komodakis , 2016 ) . It is noteworthy that few detectors are evaluated on both adversarial and OOD examples , as most work focus on only one problem . • Finally , we show that using the softmax probabilities in place of the logit activations results in significantly worse performances for both detectors . This highlights that relevant information is lost once the softmax function is applied , and raises questions about its role in the lack of robustness observed in NN . The ouline of the paper is as follows . In Section 2 , we present a brief summary of state-of-the-art methods for 3 related problems : adversarial example detection , out-of-distribution example detection and uncertainty estimation . Section 3 provides some qualitative insights about the difference in distribution between logit and softmax values on MNIST . In Section 4 , we show quantitatively the detrimental impact of the softmax on OOD and adversarial detectors performances . We summarize our findings and propose future directions in Section 5 . 2 RELATED WORK . 2.1 ADVERSARIAL EXAMPLE DETECTION . In response to the growing number of adversarial attacks that have appeared in the literature recently ( Chen & Jordan , 2019 ; Goodfellow et al. , 2015 ; Szegedy et al. , 2014 ; Moosavi-Dezfooli et al. , 2016 ) , detecting adversarial examples has also become a hot topic ( Feinman et al. , 2017 ; Lee et al. , 2018 ; Metzen et al. , 2017 ; Ozbulak et al. , 2019 ; Xu et al. , 2018 ) . Detection methods can be split into 3 categories : detecting changes in sample statistics ( Feinman et al. , 2017 ; Lee et al. , 2018 ; Ozbulak et al. , 2019 ) , training a detector ( Metzen et al. , 2017 ) and looking for prediction inconsistency ( Xu et al. , 2018 ) . Feinman et al . ( 2017 ) propose to use a combination of density estimates on the last hidden layer and bayesian uncertainty estimates ( computed using dropout during inference ( Gal & Ghahramani , 2016 ) ) to determine whether an exemple has been adversially perturbed or not . Metzen et al . ( 2017 ) train an adversarial example detector network using intermediate feature representations as inputs . Ozbulak et al . ( 2019 ) show that over-optimized adversarial examples have much higher logit values than legitimate samples , and thus propose to use a threshold on these values for detection . Xu et al . ( 2018 ) check the consistency of a NN predictions between a given input and a squeezed version of it to differentiate adversarial and regular images . Lee et al . ( 2018 ) use the Mahalanobis distance on NN learned representations to detect both adversarial and OOD examples . It is noteworthy that their detector is , to the best of our knowledge , one of the only other detectors that have been evaluated on these 2 tasks . 2.2 OUT-OF-DISTRIBUTION EXAMPLE DETECTION . Another related challenge that has recently received attention from the research community is the detection of OOD examples ( DeVries & Taylor , 2018 ; Hendrycks & Gimpel , 2017 ; Lee et al. , 2018 ; Liang et al. , 2017 ; Vyas et al. , 2018 ) . Indeed , it has been shown that a NN trained on ImageNet can provide predictions with almost 100 % confidence for images that are just noise to the human eye ( Nguyen et al. , 2015 ) . Hendrycks & Gimpel ( 2017 ) present a framework to evaluate misclassified and OOD example detectors and show that softmax probabilities can detect both of them to some extent , but that there is room for improvement . DeVries & Taylor ( 2018 ) propose to learn confidence estimates that discriminates between in and out-of-distribution examples by allowing NNs to ask for ” hints ” during training when the prediction is uncertain . Liang et al . ( 2017 ) introduce ODIN , a method that separates in and out-of-distribution examples by preprocessing the input using adversarial perturbation and then thresholding the softmax scores computed after temperature scaling . Vyas et al . ( 2018 ) train an ensemble of classifiers with a margin-based term added to the loss to detect OOD examples . 2.3 UNCERTAINTY ESTIMATION . A closely related and more general field of research is uncertainty estimation . It can be used in different settings such as semi-supervised learning ( Triguero et al. , 2015 ) , weighting an ensemble of classifiers ( Mandelbaum & Weinshall , 2017 ) , or misclassification detection ( Jiang et al. , 2018 ; Mandelbaum & Weinshall , 2017 ; Papernot & McDaniel , 2018 ) . Regarding this last task , it is noteworthy that all 3 works are strongly inspired by the KNN algorithm . Jiang et al . ( 2018 ) introduce the Trust Score , a specific 1-NN ratio applied on a filtered version of the training set in order to recognize misclassified examples . In the same spirit , Mandelbaum & Weinshall ( 2017 ) propose a confidence score based on the distance in the hidden representation space between an examples and its K nearest neighbors in the training set . Papernot & McDaniel ( 2018 ) compute a confidence score based on the homogeneity of the set of nearest neighbors computed in the representation space of each layer . 3 DIFFERENCE BETWEEN LOGIT AND SOFTMAX DISTRIBUTIONS . In this Section , we provide intuition about the difference between logit and softmax values distributions through experiments on the MNIST dataset ( LeCun et al. , 1998 ) . Our results show that there are significant differences in logit activations between MNIST , OOD and adversarial examples , but that these differences are less or not present at all once the softmax function has been applied . 3.1 SETUP . NN model : For these experiments , we train a simple custom NN using the Keras framework ( Chollet et al. , 2015 ) . The architecture is described in Table 1 . The training was done for 30 epochs . We used the RMSprop optimizer , a batch size of 64 and data augmentation ( i.e . slight rotations , zooms and shifts ) . OOD and adversarial examples : In order to study how logit and softmax values are distributed , we use or generate the following OOD and adversarial datasets : • For out-of-distribution examples , we study 4 different kinds of data . We use random normal and uniform noise images , as well as images from CIFAR-10 ( Krizhevsky et al. , 2014 ) and Fashion MNIST ( Xiao et al. , 2017 ) datasets . This allows to study 3 different kinds of OOD examples : noise images , natural images drawn from a significantly different distribution ( CIFAR-10 ) , and images drawn from a relatively close distribution ( Fashion-MNIST ) . • For adversarial examples , we use the CleverHans framework ( Papernot et al. , 2018 ) . We generate 4 adversarial datasets using different methods : FGSM ( Goodfellow et al. , 2015 ) , BIM ( Chen & Jordan , 2019 ) , DeepFool ( Moosavi-Dezfooli et al. , 2016 ) and the MaxConfidence attack ( Goodfellow et al. , 2019 ) . FGSM and BIM are two well known attacks , while DeepFool and MaxConfidence are complementary : DeepFool aims at finding the minimal perturbation which changes the prediction , while MaxConfidence aims at finding 3.2 ANALYSIS OF THE DISTRIBUTION OF LOGIT AND SOFTMAX VALUES . Figure 1 displays the distribution of the average logit values for MNIST , OOD and adversarial images . We can clearly see that logits values are higher for MNIST images than for any other dataset . These differences are statistically significant according to the Student ’ s t-test ( p < 0.0001 between MNIST and any other dataset ) . Whereas for softmax values , the average is always 1/C with C the number of classes , as the sum of the probabilities over all the classes is equal to 1 . Thus , we can see that because of the normalization of the softmax function , we lose the information about the average logit value , which appears to be discriminating . Figure 2 presents a scatterplot of the logit and softmax minimum and maximum values for the MNIST , Gaussian , CIFAR-10 , FGSM and MaxConfidence datasets . The Figure displays only these 5 datasets for the sake of visibility . Exhaustive results are presented in the Appendix . Qualitatively , we can clearly distinguish cluster of points representing each dataset . Hence , it appears that these 2 simple statistics for logits are enough to separate fairly well legitimate examples from OOD and adversarial ones . We can see that legitimate images tend to have higher minimum and/or maximum logit values , which correlates well with the distributions showed in Figure 1 . However , we can see that the discriminating information is once again lost when we apply the softmax function , as the exponential function combined with the normalization is pushing the highest value towards 1 and the lowest one towards 0 . These observations give some insights about the detrimental effect of the softmax function regarding OOD and adversarial example detection . The valuable information about the average of the logit value is lost due to the normalization , while the exponential function pushes minimum output values towards 0 and maximum ones toward 1 , making it difficult to determine if an example is legitimate or not .

Summary: This simple paper shows that the normalization of softmax causes a loss of information compared to using the unnormalized logits when trying to do OOD and adversarial example detection.  The main reason for this is of course the normalization used by the softmax. The paper is mostly empirical following this specific observation, and uses a number of examples on MNIST and CIFAR to show the improvement in performance by using unnormalized logits instead of softmax.