Intriguing class-wise properties of adversarial training

1 INTRODUCTION The existence of adversarial examples ( Szegedy et al. , 2014 ) reveals the vulnerability of deep neural networks , which greatly hinders the practical deployment of deep learning models . Adversarial training ( Madry et al. , 2018 ) has been demonstrated to be one of the most successful defense methods by Athalye et al . ( 2018 ) . Some researchers ( Zhang et al. , 2019 ; Wang et al. , 2019b ; Carmon et al. , 2019 ; Song et al. , 2019 ) have further improved adversarial training through various techniques . Although these efforts have promoted the progress of adversarial training , the performance of robust models is far from satisfactory . Thus we are eager for some new perspectives to break the current dilemma . We notice that focusing on the differences among classes has achieved great success in the research of noisy label ( Wang et al. , 2019a ) and long-tailed data ( Kang et al. , 2019 ) , while researchers in adversarial community mainly concentrate on the overall robustness . A question is then raised : How is the performance of each class in the adversarially robust model ? To explore this question , we conduct extensive experiments on six commonly used datasets in adversarial training , i.e. , MNIST ( LeCun et al. , 1998 ) , CIFAR-10 & CIFAR-100 ( Krizhevsky et al. , 2009 ) , SVHN ( Netzer et al. , 2011 ) , STL-10 ( Coates et al. , 2011 ) and ImageNet ( Deng et al. , 2009 ) , and the pipeline of adversarial training and evaluation follows Madry et al . ( 2018 ) and Wong et al . ( 2019 ) . Figure 1 plots the robustness of each class at different epochs in the test set , where the shaded area in each sub-figure represents the robustness gap between different classes across epochs . Considering the large number of classes in CIFAR-100 and ImageNet , we randomly sample 12 classes for a better indication , and the number of classes in each robustness interval is shown in Appendix A . From Figure 1 , we surprisingly find that there are recognizable robustness gaps between different classes for all datasets . Specifically , for SVHN , CIFAR-10 , STL-10 and CIFAR-100 , the class-wise robustness gaps are obvious and the largest gaps can reach at 40 % -50 % ( Figure 1 ( b ) -1 ( e ) ) . For ImageNet , since the model uses the three-stage training method ( Wong et al. , 2019 ) , its class-wise robustness gap increases with the training epoch , and finally up to 80 % ( Figure 1 ( f ) ) . Even for the simplest dataset MNIST , on which model has achieved more than 95 % overall robustness , the largest class-wise robustness gap still has 6 % ( Figure 1 ( a ) ) . Motivated by the above discovery , we naturally raise the following three questions to better investigate the class-wise properties in the robust model : 1 ) Is there any relations among these different classes as they perform differently ? 2 ) Are there any factors related to the above phenomenon ? 3 ) Is the class-wise performance related to the strength of the attack ? We conduct extensive analysis on the obtained robust models and gain the following insights : • Many examples from a certain class could only be maliciously flipped to some specific classes . As long as we remove those specific classes and re-train the model , these examples will not exist adversarial counterparts in bounded -ball . • The robustness of each class is near monotonically related to its norm of classifier weight in deep neural networks . • In both white-box and black-box settings ( Dong et al. , 2020 ) , stronger attacks are usually more effective for vulnerable classes ( i.e. , their robustness is lower than overall robustness ) . Furthermore , we propose a simple but effective attack called Temperature-PGD attack . It can give us a deeper understanding of how variants of Madry ’ s model work , especially for the robust model with obvious improvement in vulnerable classes ( Wang et al. , 2019b ; Pang et al. , 2020 ) . Thus , our work draws the attention of future researchers to watch out the robustness discrepancies among classes . 2 RELATED WORK Adversarial training . Adversarial training ( Madry et al. , 2018 ) is often formulated as a minmax optimization problem . The inner maximization applies the Projected Gradient Descent ( PGD ) attack to craft adversarial examples , and the outer minimization uses these examples as augmented data to train the model . Subsequent works are then proposed to further improve adversarial training , including introducing regularization term ( Zhang et al. , 2019 ; Wang et al. , 2019b ) , adding unlabeled data ( Carmon et al. , 2019 ; Uesato et al. , 2019 ; Zhai et al. , 2019 ) , and data augmentation ( Song et al. , 2019 ) . Since adversarial training is more time-consuming than standard training , several methods ( Shafahi et al. , 2019 ; Wong et al. , 2019 ) are proposed to accelerate the adversarial training process . Exploring the properties in adversarial training . A lot of researchers try to understand adversarial training from different perspectives . Schmidt et al . ( 2018 ) find more data can improve adversarial training . Tsipras et al . ( 2019 ) demonstrate that adversarial robustness may be inherently at odds with natural accuracy . Zhang & Zhu ( 2019 ) visualize the features of the robust model . Xie & Yuille ( 2019 ) explore the scalability . The work of Ortiz-Jimenez et al . ( 2020 ) is most relevant to ours . The difference is they focus on the distance from each example to the decision boundary , while we provide some new insights on the role of different classes in adversarial training . 3 EXPLORING THE PROPERTIES AMONG DIFFERENT CLASSES ON ADVERSARIAL TRAINING Inspired by the phenomenon in Figure 1 , we further conduct the class-wise analysis to explore the properties among different classes for a better understanding on adversarial training . Datasets . We use six benchmark datasets in adversarial training to obtain the corresponding robust model , i.e. , MNIST ( LeCun et al. , 1998 ) , CIFAR-10 & CIFAR-100 ( Krizhevsky et al. , 2009 ) , SVHN ( Netzer et al. , 2011 ) , STL-10 ( Coates et al. , 2011 ) and ImageNet ( Deng et al. , 2009 ) . Table 1 highlights that the classes of CIFAR-10 and STL-10 can be grouped into two superclasses : Transportation and Animals . Similarly , CIFAR-100 also contains 20 superclasses with each has 5 subclasses . See Appendix B for more details of all datasets . For ImageNet dataset , the pipeline of adversarial training follows Wong et al . ( 2019 ) , while the training methods of other datasets follow Madry et al . ( 2018 ) . The detailed experimental settings are : MNIST setup . Following Zhang et al . ( 2019 ) , we also use a four-layers CNN as the backbone . In the training phase , we adopt the SGD optimizer ( Zinkevich et al. , 2010 ) with momentum 0.9 , weight decay 2×10−4 and an initial learning rate of 0.01 , which is divided by 10 at the 55th , 75th and 90th epoch ( 100 epochs in total ) . Both the training and testing attacker are 40-step PGD ( PGD40 ) with random start , maximum perturbation = 0.3 and step size α = 0.01 . CIFAR-10 & CIFAR-100 setup . Like Wang et al . ( 2019b ) and Zhang et al . ( 2019 ) , we also use ResNet-18 ( He et al. , 2016 ) as the backbone . In the training phase , we use the SGD optimizer with momentum 0.9 , weight decay 2× 10−4 and an initial learning rate of 0.1 , which is divided by 10 at the 75th and 90th epoch ( 100 epochs in total ) . The training and testing attackers are PGD10/PGD20 with random start , maximum perturbation = 0.031 and step size α = 0.007 . SVHN & STL-10 setup . All settings are the same to CIFAR-10 & CIFAR-100 , except that the initial learning rate is 0.01 . ImageNet setup . Following Shafahi et al . ( 2019 ) and Wong et al . ( 2019 ) , we also use ResNet-50 ( He et al. , 2016 ) as the backbone . Specifically , in the training phase , we use the SGD optimizer with momentum 0.9 and weight decay 2×10−4 . A three-stage learning rate schedule is used as the same with Wong et al . ( 2019 ) . The training attacker is FGSM ( Goodfellow et al. , 2015 ) with random start , maximum perturbation = 0.007 , and the testing attacker is PGD50 with random start , maximum perturbation = 0.007 and step size α = 0.003 . For ImageNet dataset , a 14th epoch model is used to evaluate robustness as it did in Wong et al . ( 2019 ) . For other datasets , a 75th epoch model is used like it did in Madry et al . ( 2018 ) . These settings are fixed for all experiments unless otherwise stated . This paper mainly focuses on the adversarial robustness of the model , but the comparisons of the class-wise performance between the robust model and standard model are highlighted in Appendix C for saving space . 3.1 THE RELATIONS AMONG DIFFERENT CLASSES We first systematically investigate the relation of different classes under robust models . Figure 2 shows the confusion matrices of robustness between classes on all the six datasets . The X-axis and Y-axis represent the predicted classes and the ground truth classes , respectively . The grids on the diagonal line represent the robustness of each class , while the grids on the off-diagonal line represent the non-robustness on one class ( Y-axis ) to be misclassified to another class ( X-axis ) . Results Analysis . All the confusion matrices in Figure 2 roughly demonstrate one kind of symmetry , indicating that similar classes could easily be maliciously flipped to each other . Specifically , for SVHN , digits with similar shapes are more likely to be flipped to each other , e.g. , the number 6 and number 8 are similar in shape and the non-robustness between them ( number 6 is misclassified to be number 8 or vice versa ) is very high as shown in Figure 2 ( d ) . For CIFAR-10 and STL-10 , Figures 2 ( b ) and 2 ( e ) clearly show that the classes belonging to the same superclass1 have high probability to be craftily misclassified to each other , for example , both class 3 ( cat ) and class 5 ( dog ) in CIFAR10 belong to the superclass Animals , the non-robustness between them is very high in Figure 2 ( b ) . In contrast , there are few flips between superclasses , since the classes belonging to different superclasses are dissimilar with few mutual influence . For example , in STL-10 , the class 5 ( dog ) belongs to superclass Animals , while class 9 ( truck ) belongs to Transportation , and their non-robustness is almost 0 as shown in figure 2 ( e ) . A t-SNE ( Maaten & Hinton , 2008 ) visualization of CIFAR-10 is reported in Appendix D shows for further supporting our above analysis and conclusions on CIFAR10 . For CIFAR-100 and ImageNet , we can also observe symmetry properties of confusion matrix in Figure 2 ( c ) and Figure 2 ( f ) , indicating that some similar classes cloud easily be misclassified to each other . Overall , Figure 2 demonstrates that the classes with similar semantic would be easier misclassified ( with higher non-robustness ) to each other than those with different semantic ( e.g. , the classes belong to different superclasses ) in adversarial training . Removing the confound class . Inspired by the relations among different classes as shown in Figure 2 , in this subsection , we further investigate the class-wise properties of adversarial examples by removing the confound class . Specifically , for the example x from class i is attacked to the confound 1As we introduced in Table 1 , class 0,1,8,9 belong to superclass Transportation and class 2,3,4,5,6,7 belong to superclass Animals in CIFAR-10 . Similarly , class 0,2,8,9 belong to Transportation and class 1,3,4,5,6,7 belong to Animals in STL-10 . class j , we are curious if we remove confound class j ( i.e. , remove all examples of ground truth class j in the training set ) and re-train the model , will example x become a robust example WITHOUT being maliciously flipped to a new confound class ? Definition 1 . ( Confound Class ) The output class of the model to adversarial example x′ is defined as the confound class of this example x′ . ( This class must be different from ground truth class ) Definition 2 . ( Robust Example ) An example is defined as a robust example if it does not exist adversarial counterpart in bounded -ball , saying it would be correctly classified by the model . Definition 3 . ( Homing Property ) Given an adversarial example x′ from class i which is misclassified as the confound class j by a model , this example satisfies homing property if it becomes a robust example after we re-train the model via removing confound class j . To explore the above question , we conduct extensive experiments on the popular dataset CIFAR-10 as the case study . The results are reported in Figure 3 . Figure 3 ( a ) and Figure 2 ( b ) are similar , and the difference is that the values in Figure 3 ( a ) represent the number of examples instead of percentage , and the diagonal elements ( the number of examples correctly classified ) are hidden for better visualization and comparison . Thus this figure is called as the Misclassified confusion matrix . To check the homing property , we alternatively remove each confound class to re-train the model and plot the results in Figure 3 ( b ) , where the element in the ith row and jth column ( indexed by the classes starting from 0 ) indicates how many adversarial examples with ground truth class i and confound class j that satisfy homing property ( i.e. , these examples will become robust examples after removing the confound class j ) , so this figure is defined as the Homing confusion matrix . Results Analysis . Figure 3 clearly shows homing property is widely observed in many misclassified examples . For example , we can focus on the 9th row and the 1st column of Figure 3 ( a ) and 3 ( b ) . 147 in Figure 3 ( a ) means that 147 examples of class 9 are misclassified as class 1 , and 116 in Figure 3 ( b ) means that if we remove class 1 and re-train the model , 116 of 147 examples will home to the correct class 9 ( i.e. , become robust examples ) . This suggests that improving the robustness of class 9 only needs to carefully handle the relation with class 1 and has nothing to do with other classes . Interestingly , these group-based relations are commonly observed in CIFAR-10 , e.g. , class 0 ( airplane ) -class 8 ( ship ) and class 3 ( cat ) -class 5 ( dog ) . Conclusion and Suggestion . Classes can be divided into several groups , and intra-group classes are easily affected by each other . More importantly , many examples from a certain class are only attacked to some specific classes ( e.g. , most misclassified examples of class 9 are attacked to class 1 ) , which will become robust examples if we re-train the model by removing those specific classes . This discovery inspires us that the robustness of some classes can be handled in groups , e.g. , group by superclass or one-to-one grouping like class 1-class 9 in figure 3 ( b ) . 3.2 THE ROBUSTNESS OF CLASSES V.S . THE WEIGHT OF CLASSIFIER In this section , we attempt to explore whether some factors in the model are related to the unbalanced robustness among different classes in adversarial training . In fact , unbalanced performance is widely studied in the field of long-tailed data research ( Wang et al. , 2017 ) , that is , the number of training examples in each class varies greatly , and the class with a small number of examples can usually achieve lower accuracy since it can not be sufficiently trained . Recently , Kang et al . ( 2019 ) find there is a strong correlation between model parameters and the cardinality of classes , and they propose the state-of-the-art algorithm in long-tailed data research according to this property . Inspired by their work , we naturally curious whether the robustness of each class in adversarial training is related to the model parameters . For a better demonstration , we assume that the feature dimension of the penultimate layer is d and the total number of classes is C. Then the parameter of the last classifier can be represented by a classifier weight W = { wi } ∈ Rd×C and a classifier bias b = { bi } ∈ R1×C , where wi ∈ Rd is the classifier weight corresponding to class i . Similar to Kang et al . ( 2019 ) , we calculate the l2-norm of the classifier weight ‖wi‖2 corresponding to each class i ( i ∈ C ) . To clearly show the relation between the robustness of a class i and its corresponding classifier weight wi , we respectively normalize its robustness and l2-norm of wi , and report the results in Figure 4 . Results Analysis . From Figure 4 , we can find that for most classes , their robustness is positively correlated with their norm of classifier weight , i.e. , higher ( lower ) robustness corresponds to a higher ( lower ) norm of classifier weight . For example , the robustness of class reduced as decreasing of the norm of classifier weight across classes in CIFAR-100 as shown in 4 ( c ) . We also check this kind of correlation in standard training , but the experimental results show no significant correlation between the accuracy of each class and its corresponding norm of classifier weight in standard training . The main reason might be that these datasets in standard training are sufficient for most classes to be well trained , while adversarial training always requires abundant data for training ( Schmidt et al . ( 2018 ) ) , hence the insufficient adversarial data can not guarantee the classifier is well trained and lead to the above experimental observation results . Conclusion and Suggestion . The robustness of each class is near monotonically related to its norm of classifier weight . Inspired by this property , we believe that balancing the norm of classifier weight of each class is a possible way to alleviate the different robustness among classes , thereby improving overall model robustness . 3.3 THE CLASS-WISE ROBUSTNESS UNDER DIFFERENT ATTACKS The above analysis mainly concentrates on the performance under PGD attack . In this section , we investigate the class-wise robustness of state-of-the-art robust models against various popular attacks in the CIFAR-10 dataset . The defense methods we chose include Madry training ( Madry et al. , 2018 ) , TRADES ( Zhang et al. , 2019 ) , MART ( Wang et al. , 2019b ) and RST ( Carmon et al. , 2019 ) . We re-train WideResNet-32-10 ( Zagoruyko & Komodakis , 2016 ) following Madry et al . ( 2018 ) . Other defense methods directly use the models released by the authors . White-box attacks include FGSM ( Goodfellow et al. , 2015 ) , PGD ( Madry et al. , 2018 ) and CW∞ ( Carlini & Wagner , 2017 ) , and the implementation of CW∞ follows ( Carmon et al. , 2019 ) . Black-box attacks include a transfer-based and a query-based attack ( Dong et al. , 2020 ) . The former uses a standard trained WideResNet-32-10 as the substitute model to craft adversarial examples , and the latter uses N atacck ( Li et al. , 2019 ) . See Appendix E for the hyperparameters of all attacks . Results Analysis . As shown in Table 2 , in all models and attacks , there are remarkable robustness gaps between different classes , which further verifies our discovery in Figure 1 . Then we try to compare different attacks from a class-wise perspective . Interestingly , stronger attacks in whitebox settings are usually more effective for vulnerable classes , e.g. , comparing FGSM and PGD , the robustness reduction of the vulnerable classes ( e.g. , class 3 ) is obviously larger than that of robust classes ( e.g. , class 1 ) . In black-box settings , the main advantage of the query-based attack over the transfer-based attack is also concentrated in vulnerable classes . Additionally , we also notice that class 1 and class 3 are always the most robust and vulnerable class in all settings , which suggests the relative robustness of each class may have a strong correlation with the dataset itself . Conclusion and Suggestion . Unbalanced robustness is commonly observed in the defenses of state-of-the-art models against popular attacks , and stronger attacks are usually more powerful for vulnerable classes , e.g. , class 3 . We hope that future attack or defense works can report the results of class-wise robustness to better understand the proposed methods . 4 TEMPERATURE PGD ATTACK Some previous work has improved significantly in the vulnerable classes , such as MART ( Madry et al. , 2018 ) in Table 2 and HE ( Pang et al. , 2020 ) in Table 5 . One simple explanation is that they implicitly use instance-level ( a more fine-grained level than class-level ) information to improve model robustness . Specifically , MART achieves this by re-weighting misclassified examples ( mainly comes from vulnerable classes ) in the regular term . HE uses the re-scaled coefficient ( s in their paper ) to make the misclassified examples provide a larger gradient for the model ( more details can be found in Appendix G ) . Since the misclassified examples have received higher weight during the training process , the classification boundary of each class will be more complex , and the variance of the predicted probability of example x in each class i ( i ∈ C ) will be smaller , that is , the output distribution will be more smooth as shown in Figure 5 ( b ) -5 ( d ) ( MART ) . The vanilla PGD attacker may not be able to find the most effective direction in such a smooth probability distribution . Figure 5 ( a ) and Figure 5 ( e ) represents the mean value of the variance of the output probability distribution for each class . Specifically , the calculation process of class 0 in the Madry ’ s model is as follows : First , we calculate the variance of the prediction distribution of each examples with ground truth class 0 in CIFAR-10 , and then use these variance to obtain the mean value . This value can well reflect the smoothness of the output distribution of each class . Combine information from Table 2 , Figure 5 and Table 5 , the stronger model has a lower variance than the weak model , and the vulnerable class has a lower variance than the robust class . ( HE-LIS and HE-CMP are our improved methods based on HE which can boost robustness of vulnerable classes under vanilla PGD attack , see Appendix G for details ) . In order to find an effective direction in the extremely smooth distribution , we propose to use a temperature factor to change the smooth probability distribution , so as to create virtual power in the possible adversarial direction . For a better formulation , we assume that the DNN is f , the input sample is x , the number of classes in the dataset is C , then the softmax probability of this sample x corresponding to class i ( i ∈ C ) is S ( f ( x ) ) i = ef ( x ) i/T∑C k=1 e f ( x ) k/T ( 1 ) Using this improved softmax function , the adversarial perturbation crafted at tth step is δt+1 = ∏ ( δ t + α · sign ( ∇LCE ( S ( f ( x+ δt ) ) , y ) ) ) ( 2 ) Where ∏ is the projection operation , which ensures that δ is in -ball . LCE is the cross-entropy loss . α is the step size . y is the ground truth class . Figure 5 ( f ) -5 ( h ) is a good example of how Temperature-PGD works . This method allows us to arbitrarily control the absolute value of the gradient , and the idea of creating virtual power may The data in Table 3 is the result of comparing the performance of model robustness under Temperature-PGD attack with the vanilla PGD data in Table 2 . We find this method can reduce the overall robustness of the state-of-the-art models by 1.8 % -3.9 % , where the robustness of the vulnerable class ( i.e. , class 4 ) can be reduced by 4.2 % -10.2 % , which is consistent with our previous findings . Since the model output of Madry ’ s model is relatively more certain ( Figure 5 ( b ) -5 ( d ) ) , the effectiveness of Temperature-PGD is not obvious . See Appendix F for more ablation studies . In general , Temperature-PGD is a powerful tool for evaluating the defense which explicitly or implicitly use instance-wise information to improve model robustness . More importantly , it can give researchers a new perspective of how variants of Madry ’ s model work . We speculate that the robustness improvement of many current state-of-the-art models may be due to this phenomenon . For example , label-smoothing-based defenses ( Goibert & Dohmatob , 2019 ; Cheng et al. , 2020 ) may not be able to defense Temperature-PGD attack , since these methods explicitly flat the distribution of predicted probabilities . 5 CONCLUSION In this paper , we conduct a class-wise investigation in adversarial training based on the observation that robustness between each class has a recognizable gap , and reveal three intriguing properties among classes in the robust model : 1 ) Group-based relations between classes are commonly observed , and many adversarial examples satisfy the homing property . 2 ) The robustness of each class is positively correlated with its norm of classifier weight . 3 ) Stronger attacks are usually more effective for vulnerable classes , and we propose an attack to better understand the defense mechanism of some state-of-the-art models from the class-wise perspective . Based on these properties , we propose three promising guidelines for future improvements in model robustness : 1 ) Deal with different classes by groups . 2 ) Balance the norm of classifier weight corresponding to each class . 3 ) Pay more attention to classes suffering from poor robustness . We believe these findings can promote the progress of adversarial training and help to build the state-of-the-art robustness model . REFERENCES Anish Athalye , Nicholas Carlini , and David Wagner . Obfuscated gradients give a false sense of security : Circumventing defenses to adversarial examples . In ICML , pp . 274–283 , 2018 . Yoshua Bengio , Jérôme Louradour , Ronan Collobert , and Jason Weston . Curriculum learning . In ICML , pp . 41–48 , 2009 . Nicholas Carlini and David Wagner . Towards evaluating the robustness of neural networks . In S & P , pp . 39–57 . IEEE , 2017 . Yair Carmon , Aditi Raghunathan , Ludwig Schmidt , John C Duchi , and Percy S Liang . Unlabeled data improves adversarial robustness . In NeurIPS , pp . 11192–11203 , 2019 . Minhao Cheng , Qi Lei , Pin-Yu Chen , Inderjit Dhillon , and Cho-Jui Hsieh . Cat : Customized adversarial training for improved robustness . arXiv preprint arXiv:2002.06789 , 2020 . Adam Coates , Andrew Ng , and Honglak Lee . An analysis of single-layer networks in unsupervised feature learning . In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pp . 215–223 , 2011 . Jia Deng , Wei Dong , Richard Socher , Li-Jia Li , Kai Li , and Li Fei-Fei . Imagenet : A large-scale hierarchical image database . In CVPR , pp . 248–255 . Ieee , 2009 . Yinpeng Dong , Qi-An Fu , Xiao Yang , Tianyu Pang , Hang Su , Zihao Xiao , and Jun Zhu . Benchmarking adversarial robustness on image classification . In CVPR , pp . 321–331 , 2020 . Morgane Goibert and Elvis Dohmatob . Adversarial robustness via adversarial label-smoothing . arXiv preprint arXiv:1906.11567 , 2019 . Ian J Goodfellow , Jonathon Shlens , and Christian Szegedy . Explaining and harnessing adversarial examples . ICLR , 2015 . Sven Gowal , Jonathan Uesato , Chongli Qin , Po-Sen Huang , Timothy Mann , and Pushmeet Kohli . An alternative surrogate loss for pgd-based adversarial testing . arXiv preprint arXiv:1910.09338 , 2019 . Yandong Guo and Lei Zhang . One-shot face recognition by promoting underrepresented classes . arXiv preprint arXiv:1707.05574 , 2017 . Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image recognition . In CVPR , pp . 770–778 , 2016 . Bingyi Kang , Saining Xie , Marcus Rohrbach , Zhicheng Yan , Albert Gordo , Jiashi Feng , and Yannis Kalantidis . Decoupling representation and classifier for long-tailed recognition . In ICLR , 2019 . Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images . 2009 . Yann LeCun , Léon Bottou , Yoshua Bengio , and Patrick Haffner . Gradient-based learning applied to document recognition . Proceedings of the IEEE , 86 ( 11 ) :2278–2324 , 1998 . Yandong Li , Lijun Li , Liqiang Wang , Tong Zhang , and Boqing Gong . Nattack : Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks . In ICML , pp . 3866–3876 , 2019 . Weiyang Liu , Yandong Wen , Zhiding Yu , and Meng Yang . Large-margin softmax loss for convolutional neural networks . In ICML , volume 2 , pp . 7 , 2016 . Laurens van der Maaten and Geoffrey Hinton . Visualizing data using t-sne . Journal of machine learning research , 9 ( Nov ) :2579–2605 , 2008 . Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu . Towards deep learning models resistant to adversarial attacks . In ICLR , 2018 . Yuval Netzer , Tao Wang , Adam Coates , Alessandro Bissacco , Bo Wu , and Andrew Y Ng . Reading digits in natural images with unsupervised feature learning . 2011 . Guillermo Ortiz-Jimenez , Apostolos Modas , Seyed-Mohsen Moosavi-Dezfooli , and Pascal Frossard . Optimism in the face of adversity : Understanding and improving deep learning through adversarial robustness . arXiv preprint arXiv:2010.09624 , 2020 . Tianyu Pang , Xiao Yang , Yinpeng Dong , Kun Xu , Hang Su , and Jun Zhu . Boosting adversarial training with hypersphere embedding . NeurIPS , 2020 . Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al . Pytorch : An imperative style , highperformance deep learning library . In NeurIPS , pp . 8026–8037 , 2019 . M.J.D . Powell . Nonlinear Programming . New York : Academic Press , 1970 . Rajeev Ranjan , Carlos D Castillo , and Rama Chellappa . L2-constrained softmax loss for discriminative face verification . arXiv preprint arXiv:1703.09507 , 2017 . Mengye Ren , Wenyuan Zeng , Bin Yang , and Raquel Urtasun . Learning to reweight examples for robust deep learning . arXiv preprint arXiv:1803.09050 , 2018 . Ludwig Schmidt , Shibani Santurkar , Dimitris Tsipras , Kunal Talwar , and Aleksander Madry . Adversarially robust generalization requires more data . In NeurIPS , pp . 5014–5026 , 2018 . Ali Shafahi , Mahyar Najibi , Mohammad Amin Ghiasi , Zheng Xu , John Dickerson , Christoph Studer , Larry S Davis , Gavin Taylor , and Tom Goldstein . Adversarial training for free ! In NeurIPS , pp . 3358–3369 , 2019 . Chuanbiao Song , Kun He , Jiadong Lin , Liwei Wang , and John E Hopcroft . Robust local features for improving the generalization of adversarial training . In ICLR , 2019 . Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Goodfellow , and Rob Fergus . Intriguing properties of neural networks . ICLR , 2014 . Antonio Torralba , Rob Fergus , and William T Freeman . 80 million tiny images : A large data set for nonparametric object and scene recognition . TPAMI , 30 ( 11 ) :1958–1970 , 2008 . Dimitris Tsipras , Shibani Santurkar , Logan Engstrom , Alexander Turner , and Aleksander Madry . Robustness may be at odds with accuracy . In ICLR , number 2019 , 2019 . Jonathan Uesato , Jean-Baptiste Alayrac , Po-Sen Huang , Robert Stanforth , Alhussein Fawzi , and Pushmeet Kohli . Are labels required for improving adversarial robustness ? arXiv preprint arXiv:1905.13725 , 2019 . Yisen Wang , Xingjun Ma , Zaiyi Chen , Yuan Luo , Jinfeng Yi , and James Bailey . Symmetric cross entropy for robust learning with noisy labels . In CVPR , pp . 322–330 , 2019a . Yisen Wang , Din Zou , Jinfeng Yi , James Bailey , Xingjun Ma , and Quanquan Gu . Improving adversarial robustness requires revisiting misclassified examples . In ICLR , 2019b . Yu-Xiong Wang , Deva Ramanan , and Martial Hebert . Learning to model the tail . In NeurIPS , pp . 7029–7039 , 2017 . Eric Wong , Leslie Rice , and J Zico Kolter . Fast is better than free : Revisiting adversarial training . In ICLR , 2019 . Cihang Xie and Alan Yuille . Intriguing properties of adversarial training at scale . In ICLR , 2019 . Sergey Zagoruyko and Nikos Komodakis . Wide residual networks . arXiv preprint arXiv:1605.07146 , 2016 . Runtian Zhai , Tianle Cai , Di He , Chen Dan , Kun He , John Hopcroft , and Liwei Wang . Adversarially robust generalization just requires more unlabeled data . arXiv preprint arXiv:1906.00555 , 2019 . Hongyang Zhang , Yaodong Yu , Jiantao Jiao , Eric Xing , Laurent El Ghaoui , and Michael I Jordan . Theoretically principled trade-off between robustness and accuracy . In ICML , 2019 . Tianyuan Zhang and Zhanxing Zhu . Interpreting adversarially trained convolutional neural networks . In ICML , pp . 7502–7511 , 2019 . Bolei Zhou , Agata Lapedriza , Aditya Khosla , Aude Oliva , and Antonio Torralba . Places : A 10 million image database for scene recognition . TPAMI , 40 ( 6 ) :1452–1464 , 2017 . Martin Zinkevich , Markus Weimer , Lihong Li , and Alex J Smola . Parallelized stochastic gradient descent . In NeurIPS , pp . 2595–2603 , 2010 . A THE NUMBER OF CLASSES IN EACH ROBUSTNESS INTERVAL OF CIFAR-100 AND IMAGENET Due to the large number of classes in CIFAR-100 and ImageNet , we randomly sample 12 classes for analysis in the above paper . For the sake of experimental completeness , the number of classes in different robustness intervals is shown in Figure 6 . Obviously , the robustness of the classes is distributed at multiple intervals , which is consistent with the results shown in Figure 1 . B INTRODUCTION TO THE DATASETS USED IN THE EXPERIMENT A variety of datasets are used for research on adversarial training . Here , we introduce in detail the six datasets used in our experiment . MNIST . MNIST ( LeCun et al. , 1998 ) is a handwritten digit dataset , containing numbers 0 to 9 . The dataset consists of 60,000 training images and 10,000 test images , with 6,000 and 1,000 images per digit . All images are fixed size ( 28×28 pixels ) with a value of 0 to 1 , and these digits are located in the center of the image . This dataset is widely used in adversarial training ( Madry et al. , 2018 ; Zhang et al. , 2019 ; Wang et al. , 2019b ; Carmon et al. , 2019 ) . CIFAR-10 & CIFAR-100 . CIFAR-10 & CIFAR-100 ( Krizhevsky et al. , 2009 ) are labeled subsets of the 80 million tiny images dataset ( Torralba et al. , 2008 ) . CIFAR-10 consists of 50,000 training images and 10,000 test images in 10 classes , with 5,000 and 1,000 images per class . CIFAR-100 has the same total number of images as CIFAR-10 , but it has 100 classes . Thus CIFAR-100 has only 500 training images and 100 test images per class . All images in these two datasets are 32×32 threechannel color images . As mentioned in Section 3 , CIFAR-10 can be grouped into 2 superclasses and CIFAR-100 can be grouped into 20 superclasses . CIFAR-10 is the most popular dataset for adversarial training ( Dong et al. , 2020 ) and all proposed methods are evaluated in this dataset . CIFAR-100 is more challenging than CIFAR-10 , Shafahi et al . ( 2019 ) and Song et al . ( 2019 ) evaluate their defense methods in CIFAR-100 . SVHN . SVHN ( Netzer et al. , 2011 ) is similar in flavor to MNIST and both of them contain 10 digits . SVHN contains 73,257 labeled digits for training , 26,032 labeled digits for testing and over 600,000 unlabeled digits images for semi-supervised or unsupervised training . In order to maintain MNIST-like style , SVHN crops the image to a size of 32×32 . As a result , many of the images do contain some distractors at the sides . Due to SVHN is obtained from house numbers in the real world , its data distribution is more complicated than MNIST . Uesato et al . ( 2019 ) and Zhai et al . ( 2019 ) use this dataset for semi-supervised adversarial training . STL-10 . STL-10 ( Coates et al. , 2011 ) is inspired by the CIFAR-10 and sampled from images in the ImageNet ( Deng et al. , 2009 ) . STL-10 also contains 10 classes and each class has only 500 training images and 800 test images . The size of all images is 96×96 . As mentioned in Section 3 , STL-10 can be grouped into 2 superclasses . Song et al . ( 2019 ) use this dataset to evaluate their proposed method . ImageNet . ImageNet ( Deng et al. , 2009 ) is a high-resolution image dataset with 1000 classes . It contains 1,281,167 training images , 50,000 validation images and 100,000 test images . Since the test set has no labels , the validation set is often used to evaluate model robustness . The speed of training a robust model on ImageNet is intolerable , so using this dataset to evaluate model robustness usually requires some accelerated training techniques ( Shafahi et al. , 2019 ; Wong et al. , 2019 ) . C COMPARE CLASS-WISE GAPS BETWEEN ADVERSARIAL TRAINING AND STANDARD TRAINING In the above paper , we mainly focus on the robustness gap in the model obtained by adversarial training . Since Wang et al . ( 2019a ) also report the emergence of unbalanced accuracy in the standard model , we compare this phenomenon with that in the robust model to highlight the differences between adversarial training and standard training . Experimental setup . We use the pre-trained model of Pytorch ( Paszke et al. , 2019 ) as the standard model for ImageNet . For other datasets , the experimental settings of standard models are the same as the robust models mentioned in Section 3 , but adversarial examples are not added to the training set . Figure 7 shows the class-wise accuracy in standard training and class-wise robustness in adversarial training . The slashed part in each sub-figure represents the largest gap in accuracy/robustness among different classes , and the classes in the bracket represent the highest and lowest accuracy/robustness class . Note that natural test images are evaluated for standard training , while adversarial test images are used for adversarial training . Results Analysis . As illustrated in Figure 7 , the relative order of accuracy/robustness of different classes is almost the same in standard training and adversarial training , but the class-wise gap is enlarged in adversarial training . For example , in CIFAR-10 and SVHN datasets , their largest classwise accuracy gap in standard training are 10.7 % ( Figure 7 ( b ) ) and 5.4 % ( Figure 7 ( d ) ) , but these indicators are enlarged to 52.8 % ( Figure 7 ( b ) ) and 37.0 % ( Figure 7 ( d ) ) in adversarial training . In more complex datasets , such as STL10 , CIFAR-100 and ImageNet , although standard models also have imbalanced accuracy between classes , these gaps in adversarial training are still larger . Conclusion . The performance discrepancies among classes in the robust model are larger than that of the standard model . e.g. , the largest gap in CIFAR-10 is enlarged from 10.7 % in the standard model to 52.8 % in the robust model . D T-SNE VISUALIZATION OF CIFAR-10 In the evaluation phase , we show the t-SNE ( Maaten & Hinton , 2008 ) visualization of CIFAR-10 to further demonstrate the property of the two superclasses Transportation and Animals in this dataset . The results of Figure 8 ( a ) and Figure 8 ( b ) show that the datapoints between Transportation ( class 0,1,8,9 ) and Animals ( class 2,3,4,5,6,7 ) can be clearly distinguished , while the datapoints between the inner classes in Animals are completely confused , which confirms our analysis in Section 3.1 . E HYPERPARAMETERS FOR POPULAR ATTACKS ON THE STATE-OF-THE-ART MODELS FGSM setup . Random start , maximum perturbation = 0.031 . PGD setup . Random start , maximum perturbation = 0.031 . For RST model , step size = 0.01 and steps α = 40 , following Carmon et al . ( 2019 ) . For other models , step size = 0.003 and steps α = 20 . CW∞ setup . Binary search steps b = 5 , maximum perturbation times n = 1000 , learning rate lr = 0.005 , initial constant c0 = 0.01 , τ decrease factor γ = 0.9 . Similar to Carmon et al . ( 2019 ) , we randomly sample 2000 images to evaluate model robustness , and 200 images per class . Transfer-based attack setup . All settings are the same to PGD for the substitute standard model . N attack setup . Random start , maximum perturbation = 0.031 , population size npop = 300 , noise standard deviation σ = 0.1 and learning rate lr = 0.02 . Similar to Li et al . ( 2019 ) , we randomly sample 2000 images to evaluate model robustness , and 200 images per class . F ABLATION EXPERIMENT OF TEMPERATURE-PGD ATTACK The data in Table 4 is similar to Table 3 , except that 1/T is different . Combined with the results of Table 3 , we can find the robustness of vulnerable classes ( e.g. , class 4 ) in TRADES , MART and RST has a significant decrease in all 1/T settings . When 1/T is set to 2 , the overall robustness of the model trained by Madry et al . ( 2018 ) is no exception reduced by 0.26 % , with the most significant decrease by 2.8 % in class 4 . Furthermore , the decline of the overall robustness in Madry ’ s model is indeed lower than that in other models , One possible explanation is that these improved robust models may obfuscate gradients Athalye et al . ( 2018 ) in vulnerable classes , and theoretical analysis is left to the future . Recently , Pang et al . ( 2020 ) combine feature normalization ( FN ) ( Ranjan et al. , 2017 ) , weight normalization ( WN ) ( Guo & Zhang , 2017 ) and angular margins ( AM ) ( Liu et al. , 2016 ) to propose a defense method that can boost model robustness . In their paper , they believe that using FN and WN to limit embeddings on the hypersphere is the key to robustness improvement , so they call their method hypersphere embedding ( HE ) . However , we find that the factor ( temperature factor in our paper ) for scaling WN is the real key to improve robustness . We try to understand this from a point of view they overlooked . Our analysis mainly focuses on FN and WN , following Pang et al . ( 2020 ) . For a better formulation , we assume the input sample is x , the extracted feature of an example in the penultimate layer is z ∈ Rd , the number of classes in the dataset is C , the scale factor is s ( corresponds to the 1/T in our paper ) and the parameter of the last classifier is W = { wi } ∈ Rd×C , where wi ∈ Rd is the classifier weight corresponding to class i . Then FN operation is z̃ = z/‖z‖2 , WN operation is w̃i = wi/‖wi‖2 and the probability Pi of this sample x corresponding to class i ( i ∈ C ) after the softmax function S is Pi = S ( z ) = ew̃i T·z̃·s∑C k=1 e w̃k T·z̃·s = ew̃i T·z̃/T∑C k=1 e w̃k T·z̃/T ( 3 ) The gradient of cross-entropy loss LCE corresponding to wi is ∇wiLCE = { ( Pi − 1 ) · z i = y Pi · z i 6= y ( 4 ) Equation ( 3 ) and Equation ( 4 ) suggest we can change Pi by adjusting T and finally control∇wiLCE . e.g. , the maximum probability for an example is class l , then Pl = 1 if 1/T →∞ and Pl = 1/C if 1/T → 0 . For better demonstration , suppose that there is a three classification task ( Figure 9 ) . The top three figures represent equation ( 3 ) and the bottom three figures represent the corresponding equation ( 4 ) . The light-colored bars in the top three figures indicate the sum of the probabilities should be equal to 1 . The bars corresponding to the pink arrows in Figure 9 ( b ) and Figure 9 ( e ) have the same length , and Figure 9 ( c ) and Figure 9 ( f ) are similar . Specifically , we consider an adversarial example , i.e. , the highest probability of this example is usually the incorrect class . Figures 9 ( a ) and 9 ( d ) represent the probability and gradient of this example after softmax . When 1/T > 1 ( Figures 9 ( b ) and 9 ( e ) ) , this example is considered by the model as a harder example ( the probability of the adversarial class becomes larger ) , so that the model can update more gradients based on this example . Similarly , when 1/T < 1 ( Figures 9 ( c ) and 9 ( f ) ) , this example becomes simpler for the model and obtains a smaller gradient . Overall , these figures clearly show that we can adjust the gradient through the temperature factor 1/T . According to the above analysis and learn from the ideas of curriculum learning ( Bengio et al. , 2009 ) , we believe the gradient should be gradually enlarged ( by increasing 1/T ) during adversarial training , instead of using a fixed 1/T like Pang et al . ( 2020 ) . Specifically , the following two schedules to adjust 1/T are proposed : Linear Interpolation Schedule ( LIS ) . We use a simple linear interpolation schedule to adjust 1/T . Therefore , the 1/T of nth epoch is 1 Tn = 1 T0 + n nFI ( 1 TFI − 1 T0 ) ( 5 ) In our implementation , the initial temperature factor 1/T0 = 1 and the final temperature factor of the interpolation 1/TFI = 75 , where the subscript FI is short for final interpolation . The final epoch of the interpolation is equal to the total training epoch nFI = ntot = 100 . The training pipeline follows Pang et al . ( 2020 ) , but we do not use angular margins . Other hyperparameters are the same to Section 3 . Control Maximum Probability ( CMP ) . At each epoch , we can accurately calculate the required 1/T according to equation ( 3 ) to control the maximum probability . Since equation ( 3 ) is a nonlinear function , Powell ’ s dogleg method ( Powell , 1970 ) is used to solve this function . Therefore , the maximum probability Pmax of all examples in nth epoch isP max n = P max 0 + n nFI ( PmaxFI − Pmax0 ) 0 ≤ n < nFI Pmaxn = 1 nFI ≤ n < ntot ( 6 ) In our implementation , the initial maximum probability Pmax0 = 0.2 and the final maximum probability of the interpolation PmaxFI = 1 . The final epoch of the interpolation nFI = 90 , while the total training epoch ntot = 100 . For the last 10 epochs , the maximum probability is always controlled at 1 . Other settings are the same to Linear Interpolation Schedule . We use the above methods to train ResNet-18 on CIFAR-10 and choose vanilla PGD20 attack to evaluate model robustness . For a fair comparison , we also re-train ResNet18 following Madry et al . ( 2018 ) and Pang et al . ( 2020 ) . As shown in Table 5 , our methods HE-LIS and HE-CMP have further boosted the model robustness . Especially , the improvement of vulnerable classes ( e.g. , class 3 and class 4 ) is impressive . This seems to be an exciting result because the robustness gaps between classes are largely reduced . Unfortunately , these improved methods can not defend the Temperature-PGD attack . As shown in Table 6 , the numbers in the first row indicate the different 1/T . The improvement disappears when 1/T in Temperature-PGD is large enough . We have analyzed it in Section 4

Summary: The authors study the robustness of adversially-trained models across different classes. They find that classes tend to have largely non-uniform robust accuracy--i.e., some are less robust then others. Moreover, certain datapoints can only be misclassified as specific classes and removing these classes during training can make these datapoints robust. Next, the authors investigate how this robustness discrepancy across classes relates to the norm of the last fully-connected layer of the model, as well as to the strength of the attack used. Finally, the authors propose a new adversarial attack that they evaluate against existing models.