Confidence-Calibrated Adversarial Training: Towards Robust Models Generalizing Beyond the Attack Used During Training

1 INTRODUCTION . Deep neural networks have shown tremendous improvements in various learning tasks including applications in computer vision , natural language processing or text processing . However , the discovery of adversarial examples , i.e. , nearly imperceptibly perturbed inputs that cause mis-classification , has revealed severe security threats , as demonstrated by attacking popular computer vision services such as Google Cloud Vision ( Ilyas et al. , 2018a ) or Clarifai ( Liu et al. , 2016 ; Bhagoji et al. , 2017b ) . As the number of safety- and privacy-critical applications is increasing , e.g. , autonomous driving or medical imaging , this problem becomes even more important . In practice , adversarial training and its variants ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Madry et al. , 2018 ) , i.e. , training on adversarial examples , can be regarded as the state-of-the-art to obtain models robust against adversarial examples . In contrast to many other defenses , and to the best of our knowledge , adversarial training has not been broken so far . However , adversarial training is known to increase test error significantly . Only on simple datasets such as MNIST ( LeCun et al. , 1998 ) , adversarial training is able to preserve accuracy . This observation is typically described as a trade-off between robustness and accuracy ( Tsipras et al. , 2018 ; Stutz et al. , 2019 ; Raghunathan et al. , 2019 ; Zhang et al. , 2019 ) . Furthermore , while adversarial training leads to robust models for the employed threat model used during training , the obtained robustness does not translate to other threat models , e.g. , other Lp-balls ( Sharma & Chen , 2017 ; Song et al. , 2018 ; Madry et al. , 2018 ; Tramèr & Boneh , 2019 ; Li et al. , 2019a ; Kang et al. , 2019 ) , larger perturbations than the ones used during training or distal adversarial examples ( Hein et al. , 2019 ) . Contributions : We address both problems of adversarial training : the drop in accuracy on datasets such as Cifar10 ( Krizhevsky , 2009 ) and the poor “ generalization ” of robustness to larger perturbations or unseen threat models . To this end , we introduce confidence-calibrated adversarial training ( CCAT ) based on the idea that the predicted confidence for an adversarial example should decrease with its distance to the attacked example . Specifically , we bias the network to predict a convex combination of uniform and ( correct ) one-hot distribution for adversarial examples , which becomes uniform as the distance to the attacked test example increases . Thus , CCAT implicitly biases the network to predict the uniform distribution over classes beyond the adversarial examples seen during training . In contrast , standard adversarial training forces the network to classify adversarial examples correctly with maximal confidence but provides no guidance how to extrapolate beyond adversarial examples seen during training . Overall , CCAT allows to detect previously unseen adversarial examples , e.g. , large perturbations , different Lp attacks or distal adversarial examples , by confidence thresholding , i.e. , detection . We also discuss a worst-case evaluation methodology using an adaptive attack to compare our detection-based approach with adversarial training . First , we generalize the commonly used robust test error ( Madry et al. , 2018 ; Schott et al. , 2019 ) to our detection setting . Second , for each test example , the worst-case adversarial example , i.e. , with highest confidence , is selected from multiple white- and black-box attacks . Using an adapted attack based on ( Madry et al. , 2018 ; Dong et al. , 2018 ) , augmented with a novel backtracking scheme , we show that CCAT produces models with improved accuracy on Cifar10 and robustness to various unseen adversarial examples : larger perturbations , L2 , L1 and L0 attacks , distal adversarial examples and corrupted examples on MNIST ( -C ) , ( LeCun et al. , 1998 ; Mu & Gilmer , 2019 ) SVHN ( Netzer et al. , 2011 ) , Cifar10 ( -C ) ( Hendrycks & Dietterich , 2019 ) . We will make our code for CCAT and evaluation publicly available . Outline : After reviewing related work in Sec . 1.1 , we discuss adversarial training ( Madry et al. , 2018 ) and introduce CCAT in Sec . 2 . In Sec . 3 we present our confidence-thresholded robust test error and present experimental results in Sec . 4 . Finally , we conclude in Sec . 5 . 1.1 RELATED WORK . Adversarial Examples : Adversarial examples can roughly be divided into white-box attacks , i.e. , with access to the models , its weights and gradients , e.g . ( Goodfellow et al. , 2014 ; Madry et al. , 2017 ; Carlini & Wagner , 2017b ) , and black-box attacks , i.e. , only with access to the output of the model , e.g . ( Chen et al. , 2017 ; Brendel & Bethge , 2017 ; Su et al. , 2017 ; Ilyas et al. , 2018b ; Sarkar et al. , 2017 ; Narodytska & Kasiviswanathan , 2017 ) . Adversarial examples were also found to be transferable between similar models ( Liu et al. , 2016 ; Xie et al. , 2018 ) . While these attacks are intended to be ( nearly ) imperceptible , visible adversarial transformations , e.g. , rotations or rotations ( Engstrom et al. , 2017 ; 2019 ; Dumont et al. , 2018 ; Alaifari et al. , 2018 ; Xiao et al. , 2018 ) , or patches ( Liu et al. , 2018 ; Lee & Kolter , 2019 ; Brown et al. , 2017 ; Karmon et al. , 2018 ; Zajac et al. , 2019 ) have also been proposed and transferred to the physical world ( Athalye et al. , 2018b ; Li et al. , 2019b ; Kurakin et al. , 2016 ) . We refer to recent surveys ( Barreno et al. , 2006 ; Yuan et al. , 2017 ; Akhtar & Mian , 2018 ; Biggio & Roli , 2018a ) for more details .. Recently , white-box attacks utilizing projected gradient ascent to maximize cross-entropy loss or surrogate objectives , e.g. , ( Madry et al. , 2017 ; Dong et al. , 2018 ; Carlini & Wagner , 2017b ) , have become standard . Instead , we directly maximize the confidence in any but the true class , similar to ( Hein et al. , 2019 ; Goodfellow et al. , 2019 ) , to attack our proposed training procedure , CCAT . Adversarial Training : Many defenses against adversarial attacks have been proposed , see , e.g. , ( Yuan et al. , 2017 ; Akhtar & Mian , 2018 ; Biggio & Roli , 2018b ) , of which some have been shown to be ineffective , e.g. , in ( Athalye et al. , 2018a ; Athalye & Carlini , 2018 ) . Currently , adversarial training , i.e. , training on adversarial examples , is the de-facto standard to obtain robust models . While adversarial training was proposed in different variants ( Szegedy et al. , 2013 ; Zantedeschi et al. , 2017 ; Miyato et al. , 2016 ; Huang et al. , 2015 ; Shaham et al. , 2018 ; Sinha et al. , 2018 ; Madry et al. , 2017 ; Wang et al. , 2019 ) , the formulation by Madry et al . ( 2017 ) received considerable attention and has been extended in various ways ( Lee et al. , 2017 ; Ye & Zhu , 2018 ; Liu et al. , 2019 ) . For example , in ( Shafahi et al. , 2018 ; Pérolat et al. , 2018 ) , adversarial training is applied to universal adversarial example , in ( Cai et al. , 2018 ) , curriculum learning is used , and in ( Tramèr et al. , 2017 ; Grefenstette et al. , 2018 ) ensemble adversarial training is proposed . Adversarial training is also used in provable/certified defenses ( Kolter & Wong , 2017 ; Zhang & Evans , 2018 ) . The increased sample complexity of adversarial training ( Schmidt et al. , 2018 ) has been addressed in ( Lamb et al. , 2019 ) by training on interpolated examples or in ( Carmon et al. , 2019 ; Uesato et al. , 2019 ) using unlabeled examples . The dependence on the threat model used during training was addressed in ( Tramèr & Boneh , 2019 ) by using multiple threat models during training . Finally , the frequently observed trade-off between accuracy and robustness has been studied both theoretically and empir- ically ( Tsipras et al. , 2018 ; Stutz et al. , 2019 ; Zhang et al. , 2019 ; Raghunathan et al. , 2019 ) . CCAT differs from standard adversarial training in the used attack objective and the imposed distribution over the labels , which tends towards a uniform distribution for large perturbations . Detection : Instead of correctly classifying adversarial examples , as intended in adversarial training , several works ( Gong et al. , 2017 ; Rouhani et al. , 2018 ; Grosse et al. , 2017 ; Feinman et al. , 2017 ; Liao et al. , 2018 ; Ma et al. , 2018 ; Amsaleg et al. , 2017 ; Metzen et al. , 2017 ; Bhagoji et al. , 2017a ; Hendrycks & Gimpel , 2017 ; Li & Li , 2017 ; Pang et al. , 2018 ) try instead to detect adversarial examples . However , several detectors have been shown to be ineffective against adaptive attacks , i.e. , adversaries aware of the used detection mechanism ( Carlini & Wagner , 2017a ) . Recently , the detection of adversarial examples based on their confidence , similar to our approach with CCAT , has also been discussed ( Pang et al. , 2018 ) ; however , the authors build on ( Feinman et al. , 2017 ) which has been shown to be ineffective ( Carlini & Wagner , 2017a ) , as well . Goodfellow et al . ( 2019 ) , instead , focus on evaluating confidence-based detection methods using adaptive , targeted attacks maximizing confidence . Our attack , introduced in Sec . 2.2 , is similar in spirit , however , untargeted by nature and , thus , suitable for usage with CCAT . 2 CONFIDENCE CALIBRATION OF ADVERSARIAL EXAMPLES . Adversarial training , as described by Madry et al . ( 2018 ) , has become standard to train robust models . However , accuracy on challenging datasets such as Cifar10 is reduced and robustness does not generalize to larger perturbations , e.g. , with respect to the used L∞ constraint , or different threat models in other Lp norms . With CCAT , we intend to address both shortcomings : Instead of training models to correctly classify adversarial examples , we want to detect adversarial examples based on their confidence . In the following , we first review adversarial training , before discussing the modifications needed for our CCAT , thereby turning robustness against adversarial examples into a detection problem . Notation : We consider a classifier f : Rd → RK where K is the number of classes and fk denotes the confidence for class k. While we assume that the cross-entropy loss L is used for training , our approach can also be used with other losses . Given x ∈ Rd , classified correctly as y = argmax k fk ( x ) , an adversarial example x+ δ is defined as a “ small ” perturbation δ such that argmax k fk ( x + δ ) 6= y , i.e. , the classifier changes its decision . The strength of the change δ is measured by some Lp-norm with p ∈ { 0 , 1 , 2 , ∞ } ; p = ∞ is a popular choice in the literature as this leads to the smallest perturbation per feature/pixel .

Summary: This paper proposes the "confidence-calibrated adversarial training (CCAT)" to train robust DNNs against adversarial examples. The idea is to perform adversarial training with gradually smoothed network predictions (in an exponential or power rate to perturbation size). This new approach seems can help detection and defense at the same time to some extent. While the idea is quite interesting, the paper is poorly written and many times hard to read.