Intriguing class-wise properties of adversarial training
Authors: Qi Tian
Kun Kuang
Fei Wu
Yisen Wang
Venue:  (2021)
Keywords: adversarial training
class-wise properties
robustness
adversarial example

Abstract: Adversarial training is one of the most effective approaches to improve model robustness against adversarial examples. However, previous works mainly focus on the overall robustness of the model, and the in-depth analysis on the role of each class involved in adversarial training is still missing. In this paper, we provide the first detailed class-wise diagnosis of adversarial training on six widely used datasets, $\textit{i.e.}$, MNIST, CIFAR-10, CIFAR-100, SVHN, STL-10 and ImageNet. Surprisingly, we find that there are $\textit{remarkable robustness discrepancies among classes}$, demonstrating the following intriguing properties: 1) Many examples from a certain class could only be maliciously attacked to some specific semantic-similar classes, and these examples will not exist adversarial counterparts in bounded $\epsilon$-ball if we re-train the model without those specific classes; 2) The robustness of each class is positively correlated with its norm of classifier weight in deep neural networks; 3) Stronger attacks are usually more powerful for vulnerable classes. Finally, we propose an attack to better understand the defense mechanism of some state-of-the-art models from the class-wise perspective. We believe these findings can contribute to a more comprehensive understanding of adversarial training as well as further improvement of adversarial robustness.

Under review as a conference paper at ICLR 2021
INTRIGUING CLASS-WISE PROPERTIES OF ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial training is one of the most effective approaches to improve model robustness against adversarial examples. However, previous works mainly focus on the overall robustness of the model, and the in-depth analysis on the role of each class involved in adversarial training still missing. In this paper, we pro- vide the first detailed class-wiss wise diagnosis of adversarial training six widely used datasets, Lee, MNIST, CIFAR-10, CIFAR-100, SVHN, STL-10 and Ima- geNet. Surprisingly, we find that there are remarkable robustness discrepancies among classes, demonstrating the following intriguing properties: 1) Many ex- amples from certain class could only be maliciously attacked to some specific semantic-similee classes, and these examples will not exist adversarial counter- parts in bounded e-ball if re-train the model without those specific classes; 2) The robustness of each class is positively correlated with its norm of classifier weight in deep neural networks; 3) Stronger attacks usually more powerful for vulnerable classes. Finally, we propose an attack to better understand the defense mechanism of some state-of-the-art models from the class-wise perspective. We believe these findings can contribute to a more comprehensive understanding of adversarial training as well as further improvement of adversarial robustness.
INTRODUCTION
The existence of adversarial examples (Szegedy et al. 2014) reveals the vulnerability of deep ral networks, which greatly hinders the practical deployment of deep learning models. Adversarial training (Madry et 2018) has been demonstrated be one of the most successful defense meth- ods by Athalye et al. (2018). Some researchers (Zhang et al. 2019 Wang et al. 2019b; Carmon al. 2019: Song et al. (201)) have further improved adversarial training through various tech- niques. Although these efforts have promoted the progress of adversarial training, the performance of robust models is far from satisfactory. Thus we are eager for some new perspectives to break the current dilemma.
We notice that focusing on the differences among classes has achieved great success in the research of noisy label (Wang al. 2019a) and toig-tailed data (Kang al. 2019), while researchers in adversarial community mainly concentrate on the overall robustness. A question is then raised:
How is the performance of each class in the adversarially robust model?
To explore this question, we conduct extensive experiments six commonly used datasets in adver- sarial training, i.e.. MNIST (LeCun al., 1998), CIFAR-10 10 & CIFAR- 100 (Krizhevsky al., 2009),
SVHN (Netzer et al. 2011). STI-10 (Coates et al. 2011 and ImageNet (Deng et al.. 2009), and the pipeline of adversarial training and evaluation follows Madry et al.) (2018) and Wong et al. (2019). Figure plots the robustness of each class at different epochs in the test set, where the shaded area in each sub-figure represents the robustness gap between different classes across epochs. Consider- ing the large number of classes in CIFAR-100 and ImageNet, randomly sample 12 classes for better indication, and the number of classes in each robustness interval is shown in Appendix A
From Figure I surprisingly find that there recognizable robustness gaps between different classes for all datasets. Specifically, for SVHN, CIFAR- 10, STL-10 and CIFAR-100, the class-wist robustness gaps are obvious and the largest gaps reach 40%-50% (Figure T(b) I(e)). For

---

Under review as a conference paper at ICLR 2021
Cla
Cln't Cla
Claso-wion roheudtoos hoot NXT
Claioeei rohesdnon tn
80-
80
80
60-
8
-00
Nomber spochs
spochs
spochs
(a) MNIST
(b) CIFAR-10
(c) CIFAR-100
Clac
Clas
Chaneriio rohecction 100 501
Cla3
Clac
Clan
80
80
80
60
80
20-
30
20
12
Nomber spochs
spochs
Number of epochs
(d) SVHN
(c) STL-10
(f)
ImageNet
Figure 1: Class-wise robustness at different epochs in test set
2019). its class- wise ImageNet, since the model uses the shree-stage training method (Wong et al. robustness gap increases with the training epoch, and finally up to 80% (Figure I(O) Even for the simplest dataset MNIST, on which model has achieved more than 95% overall robustness, the largest class-wisc robustness gap still has 6% (Figure I(a)). Motivated by the above discovery, we naturally raise the following three questions better investigate the class-wise properties in the robust model:
1) Is there any relations among these different classes as they perform differently?
2) Are there any factors related to the above phenomenon?
3) Is the class-wise performance related to the strength of the attack?
We conduct extensive analysis on the obtained robust models and gain the following insights:
Many examples from certain class could only be maliciously flipped to some specific classes. As long as we remove those specific classes and re-train the model, these examples will not exist adversarial counterparts in bounded e-ball.
The robustness of each class is near monotonically related to its norm of classifier weight in deep neural networks.
In both white-box and black-box settings (Dong et al. 2020), stronger attacks are usually more effective for vulnerable classes (i.e., their robustness is lower than overall robustness).
Furthermore, we propose a simple but effective attack called Temperature-Per PGD attack. can give us a deeper understanding of how variants of Madry's 's model work, especially for the robust model with obvious improvement vulnerable classes (Wang et al. 2019b Pang et al. 2020). Thus, our work draws the attention of future researchers to watch out the robustness discrepancies among classes
RELATED WORK
Adversaiial training. Adversarial training (aadry et al., 2018) is often formulated as a min- max optimization problem. The inner maximization applies the Projected Gradient Descent (PGD) attack to craft adversarial examples, and the outer minimization uses these examples as augmented data to train the model. Subsequent works are then proposed to further improve adversarial training, including introducing regularization term (Zhang et al.] 2019 Wang et al. (2016)), adding unlabeled data (Carmon et al. 2019 Uesato et al. 2019 Zhai et al. 2019), and data augmentation (Song al.,
2

---

Under review as a conference paper at ICLR 2021
2019).
Since adversarial training is more time- cone-conuuming than standard training, several methods Shafahi et al. 2019 Wong et al. 2019) are proposed to accelerate the adversarial training process.
Exploring the properties in adversarial training. A lot of researchers try understand adversar- ial training from different perspectives. [Schmidt et al. (2018) find more data can improve adversarial training. TTipras et al. (2119) demonstrate that adversarial robustness may be inherently at odds with natural accuracy. Zhang & Zhu (2019) visualize the features of the robust model. Xie & Yuille (2119) explore the scalability. The work of Ortiz-Jimenez et al. (2020) most relevant ours. The difference is they focus on the distance from each example to the decision boundary, while we provide some new insights on the role of different classes in adversarial training.
EXPLORING THE PROPERTIES AMONG DIFFERENT CLASSES ON ADVERSARIAL TRAINING
Inspired by the phenomenon in Figure ! further conduct the class-wise analysis to explore the properties among different classes for a better understanding on adversarial training.
Datasets. We six benchmark datasets in adversarial training obtain the corresponding bust model, ie.. MNIST (LeCun et al.) 1998), CIFAR-10 & CIFAR-100 (Krizhevsky et al. 2009), SVHN NNtzer et al. 2011), STL-I0 (Coates al. 2011) and ImageNet (Deng et al., (2009). Ta- ble D highlights that the classes of CIFAR- 10 and ST can be grouped into two superclasses: Transportation and Animals. Similarly, CIFAR- also contains 20 superclasses with each has subclasses. See Appendix for more details of all datasets.
Table I: Superclasses in CIFAR- 10 and STL-10.
Dataset
Tramsporationn
Animals
CIFAR-10
Airplane(0)
Automohile(1)
Ship(8) Truck(9)
Bird(2)
Car(3)
Deer(4)
Dog(5)
Prog(6)
Horse(
STL-10
Airplane(O)
Car2))
Ship(8)
Truck(9)
Bird(l)
Car(3)
Deer(4)
Dog(5)
Horset6)
Monkey(7)
The number in brackets represents the numeric label of the class in the dataset.
For ImageNet dataset, the pipeline of adversarial training follows Wong et al. (2019). while the training methods of other datasets follow Madry et al. (2018). The detailed experimental settings
are:
Following Zhang et al. 2019), we also use a four-layers CNN as the backbone. In the training phase, adopt the SGD optimizer (Zinkevich al. 2010) with momentum 0.9, weight decay 2 x 10-4 and an initial learning rate of 0.01, which is divided by 10 at the 55th 75th and 90th epoch (100 epochs in total). Both the training and testing attacker are 40-step PGD (PGD with random start, maximum perturbation = 0.3 and step size = 0.01.
MNIST setup.
CIFAR-10 & CIFAR-100 setup. Like Wang et (019b) and Zhang al. 2019) we also use ResNet-18 18 (He et al. 2016) as the backbone. In the training phase, we use the SGD optimizer with momentum 0.9, weight decay 2 x 10-4 and an initial learning rate of 0. which is divided by 10 at the 75th and 90th epoch (100 epochs in total). The training and testing attackers are PGD 2P with random start, maximum perturbation e 0.031 and step size a 0.007.
SVHN & STL-10 setup. All settings are the same to CIFAR-10 & CIFAR-100, except that the initial learning rate is 0.01.
ImageNet setup. Following Shafahi et al. (2019) and Wong et al. (2019). we also use ResNet-50 et al. 2016) as the backbone. Specificalll. in the training phase, we use the SGD optimizer with momentum 0.9 and weight decay 2 x 10-4, A three-stage learning rate schedule is used as the same With/Wong et al. (209)). The training attacker FGSM (Goodfellow al. (015)) with random start, 0.007, and the testing attacker is PGD with random start, maximum
maximum perturbation perturbation e 0.007 and step size a = 0.003.
For ImageNet dataset, t4th epoch model is used to evaluate robustness as it did in Wong et al. (019)). For other datasets, a 75th epoch model used like did in Madry et al. 2018)). These settings fixed for all experiments unless otherwise stated. This paper mainly focuses the adversarial robustness of the model, but the comparisons of the class performance between the robust model and standard model are highlighted in Appendix C for saving space.
3

---

Under review as a conference paper at ICLR 2021
3.1 THE RELATIONS AMONG DIFFERENT CLASSES
We first systematically investigate the relation of different classes under robust models. Figure 2 shows the confusion matrices of robustness between classes on all the six datasets. The X-axis and Y-axis represent the predicted classes and the ground truth classes, respectively. The grids on the diagonal line represent the robustness of each class, while the grids on the off-diagonal line represent the non-robustness on one class (Y-axis) to be misclassified to another class (X-axis).
Ground truhh dasses (annes(uuperraa darssasss... 100000
M2
0.8
00
03
64
00
Ground truth caaves
80
12
02
80
2
-45
(uupers
3
66
Ground truis
-30
20
10
40
17
15
14
02
50
440
1/5
33
-15
10
-20
13
23
114
98
67
33
15
60
M5
6
8
10
Predicted classes
Predicted classes
Predicted classoopsperrcins index)
(a) MNIST
(b) CIFAR-10
(c) CIFAR-100
RG
53
46
41
24
24
61
45
1.5
25
33
AA
66
34
11
16
14
18
18. ITA
200
20
Ground truit claves
1.5
13
311
-20
Ground truth classes
200
80
30
Ground truits
13
400
60
6
600
14
0.5
8
800
20
163
05
26
2
5
6
7
8
200
400
600
800
Prodicted classes
Prodicted
Prodicted classes
(d) SVHN
(e) STL-10
(f) ImageNet
Figure 2: Confusion matrix of robustness in test set
Results Analysis. All the confusion matrices in Figure roughly demonstrate one kind of symme- try, indicating that similar classes could easily be maliciously flipped to each other. Specifically, for more likely to be flipped to each other, e.g., the number 6 and number 8 are similar in shape and the non-robustness between them (number is misclassified to be number 8 or vice versa) is very high as shown in Figure 2(d) For CIFAR-10 10 and STL-10, Figures 2(b) and 2(e) clearly show that the classes belonging to the same superclass have high probability to be craftily misclassified to each other, for example, both class 3 (cat) and class (dog) in CIFAR- 10 belong to the superclass Animals, the non-robustness between them is very high in Figure 2(b) In contrast, there are few flips between superclasses, since the classes belonging to different super- classes are dissimilar with few mutual influence. For example, in STL-10, the class 5 (dog) belongs to superclass Animals, while class (truck) belongs to Transportation, and their non-robustness is almost O as shown in figure 2 A t-SNE (Maaten& & Hinton] 2008) visualization of CIFAR- 10 is reported in Appendix Diswws for further supporting our above analysis and conclusions on CIFAR- 10. For CIFAR- 100 and ImageNet, we can also observe symmetry properties of confusion matrix in Figure 2 and Figure 2()) indicating that some similar classes cloud easily be misclassified to each other. Overall, Figure 2 demonstrates that the classes with similar semantic would be easier misclassified (with higher non-robustness) to each other than those with different semantic (e.g.. the classes belong to different superclasses) in adversarial training.
SVHN, digits with similar shapes
Removing the confound class. Inspired by the relations among different classes as shown in Figure 2, in this subsection, we further investigate the class-wise properties of adversarial examples by removing the confound class. Specifically, for the example x from class i is attacked to the confound
As we introduced in Table class 0,1,8,9 belong to superclass Transportation and class 2,3,4,5,6,7 belong to superclass Animals in 110. Similarly, class 0,2,8,9 helong to Transportation and class 1,3,4,5,6,7 helong to Animals in STL--10.
4

---

Under review as a conference paper at ICLR 2021
class are curious if we remove confound class (i.e., remove all examples of ground truth class j in the training set) and re-trai re-train the model, will example x become a robust example WITHOUT being maliciously flipped to a new confound class?
Definition 1. (Confound Class) The oulput class of the model to adversarial example x' is defined as the confound class of this example x'. (This class must be different from ground truth class)
Definition (Robust Example) An example is defined as a robust example if it does not exist adversarial counterpart in bounded c-ball, saying it would be correctly classified by the model.
Definition 3. (Homing Property) Given an adversarial example x' from class i which is misclassi- fied as the confound class j by model, this example satisfies homing property if becomes a robust example after re-train the model via removing confound class j.
To explore the above question, we conduct extensive experiments on the popular dataset CIFAR-10 10 as the case study. The results reported in Figure Figure 3(a) and Figure 2(b) are similar, and the difference is that the values in Figure (aa) represent the number of examples instead of percentage, and the diagonal elements (the number of examples correctly classified) are hidden for better visualization and comparison. Thus this figure is called as the Misclassified confusion matrix. To check the homing property, we alternatively remove each confound class re-train the model and plot the results in Figure 3(b) where the element in the ith row and jth column (indexed by the classes starting from 0) indicates how many adversarial examples with ground truth class and confound class j that satisfy homing property (i.e., these examples will become robust examples after removing the confound class j), so this figure defined as the Homing confusion matrix.
S7
24
56
28
45
200
34
128
125
12
15
25
Ground truthh classes
71
X9
165
75
12%
29
160
Ground truth classes
2
19
15
6
15
22
100
34
14
200
D62
40
45
3
19
13
119
48
SS
18S
133
1:5
120
4
64
16
7X
74
75
HS
I81
DUN
19
5
16
92
24
20
6
13
92
75
143
53
20
27
15
26
17
71
10
7
10
50
8
73
39
16
33
7
12
45
27
10
-5
96
43
33
F
22
14
17
LG
38
66
18
13
49
147
15
s0
7
s5
23
21
73
16
116
26
2
3
4
5
6
7
8
2
3
4
6
8
Predicted (confound) classes
Removed classes
(a) Misclassified confusion matrix
(b) Homing confusion matrix
Figure 3: Misclassified and Homing Confusion matrix of CIFAR-10
Results Analysis. Figure shows homing property is widely observed in many misclassified examples. For example, we can focus on the 9th row and the 1st column of Figure 3(a) and 3(b) 147 in Figure ((a) means that 147 examples of class are misclassified class 1, and T16 in Figure 3(b) means that if we remove class 1 and re-train the model, 116 of 147 examples will home to
the correct class (i.e., become robust examples). This suggests that improving the robustness of class 9 only needs to carefully handle the relation with class 1 and has nothing to do with other classes. Interestingly, these group-based relations are commonly observed CIFAR-10, e.g., class 0 (airplane)-class 8 (ship) and class (cat)-class 5 (dog).
Conclusion and Suggestion. Classes can be divided into several groups, and intra-group classes are easily affected by each other. More importantly, many examples from a certain class are only attacked to some specific classes (e.g.. most misclassified examples of class are attacked to class 1), which will become robust examples if re-train the model by removing those specific classes. This discovery inspires us that the robustness of some classes can be handled in groups, e.g., group by superclass one-to-one grouping like class 1-elass in figure ((b)
5

---

Under review as a conference paper at ICLR 2021
3.2 THE ROBUSTNESS CLASSES V.S. THE WEIGHT OF CLASSIFIER
In this section, we attempt to explore whether some factors in the model are related to the unbalanced robustness among different classes in adversarial training. In fact, unbalanced performance is widely studied in the field of long-tailed data research (Wang et al.] 2017). that is, the number of training examples in each class varies greatly, and the class with small number of examples achieve lower accuracy since it cannot be sufficiently trained. Recently, Kang et al. (2019) find there strong correlation between model parameters and the cardinality of classes, and they propose the state-of-ther algorithm in long-tailed tailed data research according to this property.
usually
Inspired by their work, naturally curious whether the robustness of each class in adversarial training is related to the model parameters. For a better demonstration, we assume that the feature dimension of the penultimate layer d and the total number of classes C. Then the parameter of the last classifier can be represented by classifier weight W = {w;] E RdxC and classifier bias {b:} E 1x where w; Rd is the classífier weight corresponding to class i. Similar to Kang et al. (2019). we calculate the l--orm of the classifier weight ||w/|]a corresponding to each class 2 ( E C). To clearly show the relation between the robustness of a class and its corresponding classifier weight W,, respectively normalize its robustness and &2 22-morm of wi, and report the
results in Figure
10
1.0
10
08
0.8-
OR
06-
6-6-
Nomnitaed
06-
0.4
2.4
0.4-
021
6.2
02-
0.9
0.0
0.0
Chacos
Classs
(a) MNIST
(b) CIFAR-10
(c) CIFAR-100
1.0
0.8
0.8
0.8
Nomaited vatoo
0.5-
Namalied
0.5-
0.6
0.4
0.4
04
0.2
0.2
0.2
0.0
0.0
0.9
127
185 900
553 M02 BAN 109
Clacs
Classe
Classs
(d) SVHN
(e) STL-10
(f) ImageNet
Figure The correlation between the robustness each class and its norm classifier weight
Results Analysis. From Figure we can find that for most classes, their robustness is positively correlated with their norm of classifier weight, i.e., higher (lower) robustness corresponds to a higher (lower) norm of classifier weight. For example, the robustness of class reduced as decreasing of the norm of classifier weight across classes in CIFAR- 100 as shown in 4(c) We also check this kind of correlation in standard training, but the experimental results show no significant correlation between the accuracy of each class and its corresponding norm of classifier weight in standard training. The main reason might be that these datasets standard training are sufficient for most classes to well trained, while adversarial training always requires abundant data for training (Schmidt et al. (2018), ) hence the insufficient adversarial data cannot guarantee the classifier is well trained and lead to the above experimental observation results.
Conclusion and Suggestion. The robustness of each class near monotonically related to its norm of classifier weight. Inspired by this property, we believe that balancing the norm of classifier weight of each class is a possible way to alleviate the different robustness among classes, thereby improving overall model robustness.

---

Under review as a conference paper at ICLR 2021
3.3 THE CLASS-WIL WISE ROBUSTNESS UNDER DIFFERENT ATTACKS
The above analysis mainly concentrates on the performance under PGD attack. In this section, we investigate the class-wise robustness of state-of-the-art robust models against various popular attacks in the CIFAR-10 10 dataset.
The defense methods we chose include Madry training (Madry al. 2018). TRADES (Zhang al.
2019), MART (Wang et al. 2019b) and RST (Carmon et al. 2019). We re-train WideResNet-33 Ne--eeeeet---- 32-10
Zagoruyko & Komodakis 2016) following Madry al. (018). Other defense methods directly
use the models released by the authors. White-box attacks include FGSM (Goodfellow et al.. 2015)
PGD (Madry et al., 2018) and CW (Carlini & Wagner 2017). and the implementation of CW
follows (Carmon et al. (2019). Black-box attacks include transfer-based and query-based attack
(Dong et al. 2020). The former uses a standard trained WideResNet-32-10 as the substitute model
to craft adversarial examples, and the latter uses N atacck (Li et al. 2019). See Appendix B for the
hyperparameters of all attacks.
Table 2: Adversarial robustness (%) (under popular attacks) on CIFAR-10.
Defonses(AAtise
Tot.
0
4
5
9
Madry(FGSM)
65.5
73.7
81.21
51.9
41.5²
54.2
49.4
73.9
72.5
78.5
78.5
TRADES(FGSM)
66.9
77.5
85.9
49.7
41.9
55.8
52.8
73.0
76.8
80.7
75.3
MART(FGSM)
67.4
73.7
84.9
45.7
50.1
51.6
76.9
77.7
RST(PGSM)
74.7
83.4
88.9
61.2
51.7
67.7
61.4
80.6
80.8
85.7
85.6
Madry(PGD)
52.2
63.8
71.6
39.1
25.3
36.7
38.6
57.4
59.5
63.1
66.8
TRADES(PCI)
56.4
67.8
80.6
37.8
29.4
40.6
43.9
59.3
66.9
71.8
65.6
MART(PGD)
58.3
64.5
78.0
35.4
37.7
43.5
63.3
67.5
76.3
69.5
RSTPGD)
62.9
74.5
81.8
46.7
37,2
46.8
51.0
66.6
71.9
74.4
78.0
Madry(CW
57.1
67.5
79.5
43.0
32.5
41.5
41.0
59.5
60.0
71.5
75.0
IRADES/CW. m))
59.4
70.5
85.5
38.0
34.5
43.0
46.5
57.0
67.0
80.5
71.5
MART(CW...
58.9
65.5
84.5
43.0
31.5
33.0
40.0
63.0
70.5
83.5
74.0
RSTCCW,..)
67.6
77.5
87.0
54.0
44.5
$4.0
51.0
69.5
69.5
82.5
86.0
attack)
80.3
84.5
87.7
71.0
68.3
78.9
69.2
86.3
82.9
87.6
86.4
TRADES TRADES/TTanSer-bee Trander--beeed attack)
82.0
87.7
92.3
70.9
68.0
78.2
70.0
87.8
87.6
90.8
86.8
TransSr bused attack)
82.9
87.4
94.7
74.0
66.7
76.0
68.8
89.9
88.0
93.7
90.2
RST Transffee based altack)
88.9
92.5
95.8
82.4
75.4
88.8
79.4
94.1
91.8
95.2
93.5
Mudry(N atacck)
56.1
67.5
77.7
43.7
31.4
42.7
49.0
53.7
60.1
64.4
71.1
TRADESCA atacck)
73.1
87.4
49.1
61.7
56.9
74.1
MART(N atacck)
67.5
72.3
83.4
55.3
49.0
54.1
61.2
67.1
72.9
82.3
77.6
RSTAV stacck)
69.3
80.8
86.9
55.8
52.3
56.2
59.1
64.9
72.6
81.0
83.0
The undorscore indicates the most robast class. The bold indicates the most vulnerable class.
Results Analysis. As shown in Table 2 in all models and attacks, there are remarkable robustness gaps between different classes, which further verifies our discovery in Figure [ Then we try to compare different attacks from a class-wise perspective. Interestingly, stronger attacks in white- box settings usually more effective for vulnerable classes, e.g., comparing FGSM and PGD, the robustness reduction the vulnerable classes (e.g., class 3) is obviously larger than that of robust classes (e.g., class 1). In black-box settings, the main advantage of the query-based attack over the transfer-based attack is also concentrated in vulnerable classes. Additionally, we also notice that class and class 3 are always the most robust and vuinerable class in all settings, which suggests the relative robustness of each class may have a strong correlation with the dataset itself.
Conclusion and Suggestion. Unbalanced robustness is commonly observed in the defenses of state-of-the-arr models against popular attacks, and stronger attacks are usually more powerful for vulnerable classes, e.g., class 3. We hope that future attack or defense works can report the results of class-wise robustness to better understand the proposed methods.
4 TEMPERATURE PGD ATTACK
Some previous work has improved significantly in the vulnerable classes, such as MART Madry al. 2018 in Table 2 and HE (Pang et al.) 2020 in Table 5 One simple explanation is that
they implicily use instance-level (a more fine- tine-grained level than class-level) information to improve model robustness. Specifically, MART achieves this by re-weighting misclassified examples (mainly
7

---

Under review as a conference paper at ICLR 2021
TRAD85 MATT
MAKE 2014
MARE V7-1
MAKI 13-1
Ad6
Gasadmmaadaa
(b)
(c)
(d)
Ht
ICLE
17-10
0B: 17-11
1h-
soau
(c)
(I)
(h)
Figure Analysis of class-wise predicted probability distribution: (a) Class-wise variance of predicted probability of Madry, TRADES and MART. (b)-(d) The output probability distribution (Madry: 1/T=1 and MART: 1/T=1) change of image 127 (ground truth class 3) in the process of generating adversarial examples (iteration step 1, 10 and 20). (e) Class-wise variance of predicted probability of HE, HE-LIS and HE-CMP. (f)-(h) The output probability distribution (HE: 1/T=1 and HE: 1/T=10) change of image 46 (ground truth class 3) in the process of generating adversarial examples (iteration step 1, 10 and 20).
comes from vulnerable classes) in the regular term. HE uses the re-scaled coefficient (s in their paper) make the misclassified examples provide a larger gradient for the model (more details can be found in Appendix
Since the misclassified examples have received higher weight during the training process, the classi- fication boundary of each class will be more complex, and the variance of the predicted probability of example x in each class (i C)) will be smaller, that is, the output distribution will be more smooth as shown in Figure (bb) ((d) (MART). The vanilla PGD attacker may not be able to find the most effective direction in such smooth probability distribution.
Figure ((a) and Figure 5(e) represents the mean value of the variance of the output probability distribution for each class. Specifically, the calculation process of class in the Madry's model is as follows: First, we calculate the variance of the prediction distribution of each examples with ground truth class in CIFAR-I0, and then use these variance to obtain the mean value. This value can well reflect the smoothness of the output distribution of each class. Combine information from Table 2 Figure 5 and Table 5 the stronger model has a lower variance than the weak model, and the vulnerable class has lower variance than the robust class. (HE-LIS and HE-CMP improved methods based on HE which can boost robustness of vulnerable classes under vanilla PGD attack, see Appendix G for details). In order find an effective direction in the extremely smooth distribution, we propose to use a temperature factor to change the smooth probability distribution, so to create virtual power in the possible adversarial direction.
For better formulation, we assume that the DNN is f, the input sample is the number of classes in the dataset is C, then the softmax probability of this sample x corresponding class (i C) is
(())
S((())); = D el(aaa/
(1)
Using this improved softmax function, the adversarial perturbation crafted at thh step is g4+1 = 16' + a sign( (VCor(S( + +
(2)
Where II. is the projection operation, which ensures that is in c-ball. Leg is the cross-entropy loss. is the step size. is the ground truth class.
Figure 5(1)(()
((h is a good example of how Temperature-Per PGD works. This method allows us to arbitrarily control the absolute value of the gradient, and the idea of creating virtual power may
8

---

Under review as a conference paper at ICLR 2021
Table 3: Relative robustness (%) between Temperature-PGI- and vanilla PGD
Defense
1/T
Tot.
0
2
4
7
8
Madry
-0.2(52.0)
-0.4'
+0.4
-0.3
+0.4
-2.8
+0.6
-0.8
+0.3
-0.1
+0.1
TRADES
-1.8(54.6)
-1.0
-0.6
-1.1
-3.2
-5.0
-0,9
-3.3
-0.9
-1.0
-0.7
MART
-3.9(54.4)
-1.8
-0.9
-3.6
-9.1
-10.2
-2.0
-4.5
-1.4
-3,5
-2.2
RST
-2.0(60.9)
-1.0
-0.6
-2.2
-2.1
-4.2
-1.5
-5.1
-1.1
-1.2
-0.8
12" represents the robustness reduction compared with corresponding element PGD attack in table
"*" represents the robustness improvement compared with the corresponding element PGD attack in table
also be used in targeted attack (Gowal et al. 2019). that is, manual re-scale gradients may reach the adversarial point faster in a limited number of iterations.
The data in Table 3 is the result of comparing the performance of model robustness under Temperature-PCG PGD attack with the vanilla PGD data in Table 2 We find this method can reduce the overall robustness of the state-of-the-ae art models by 1.8%-3.9% where the robustmess of the vul- nerable class (i.e., class 4) can be reduced by 4.2%-11..... 10.2%, which is consistent with our previous findings. Since the model output of Madry's model is relatively more certain (Figure 5(b) 5(d). the effectiveness of Temperature-PGD is not obvious. See Appendix [ for more ablation studies.
In general, Temperature-PGr is powerful tool for evaluating the defense which explicitly im- plicitly use instance-wise information to improve model robustness. More importantly, it can give researchers new perspective of how variants of Madry's model work. We speculate that the ro- bustness improvement of many current state-of-the-ar models may be due to this phenomenon. For example, label-smoothing-bee based defenses (Goibert & Dohmatob 2019 Cheng et al.) 2020) may not be able to defense Temperature-PGr attack, since these methods explicityy flat the distribution of predicted probabilities.
CONCLUSION
In this paper, we conduct class-wise investigation adversarial training based on the observation that robustness between each class has recognizable gap, and reveal three intriguing properties among classes the robust model: 1) Group-based relations between classes are commonly ob- served, and many adversarial examples satisfy the homing property. 2) The robustness of each class is positively correlated with its norm of classifier weight. 3) Stronger attacks are usually more effective for vulnerable classes, and propose attack to better understand the defense mech- anism some state-of-the-ee models from the class-wise perspective. Based on these properties, propose three promising guidelines for future improvements in model robustness: Deal with different classes by groups. 2) Balance the norm of classifier weight corresponding to each class. 3) Pay more attention classes suffering from poor robustness. We believe these findings can promote the progress of adversarial training and help to build the sate-of-the-an art robustness model.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, pp. 274-283, 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICMI,, pp. 41 41-48, 2009.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. S&P, pp. 39-57. IEEE, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In NeurlPS, pp. 1 11192-11203, 2019.
Minhao Cheng, Qi Lei, Pin-Yu Yu Chen, Inderjit Dhillon, and Cho-Jui Hsieh. Cat: Customized adver- sarial training for improved robustness. arXiv preprint arXiv:2002.06789, 2020.
Adam Coates, Andrew Ng. and Honglak Lee, An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215 223, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, Ieee, 2009.
Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu. Bench- marking adversarial robustness on image classification. In CVPR, pp. 321-331, 2020.
Morgane Goibert and Elvis Dohmatob. Adversarial robustness via adversarial label-smoothing
arXiv preprint arXiv: 11567, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang, Timothy Mann, and Pushmeet Kohli.
An alternative surrogate loss for pgd-based adversarial testing. arXiv preprint arXiv: (9X.v:9910009336: 2019.
Yandong Guo and Lei Zhang. One-shot face recognition by promoting underrepresented classes. arXiv preprint arXiv: T 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-2) -2244, 1998.
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. Nattack: Learning the dis-
tributions of adversarial examples for an improved black-box attack on deep neural networks. In ICML, pp. 3866-3876, 2019.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In ICML, volume 2, pp. 7, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, ((Nov):2579-2600 2008
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.
10

---

Under review as a conference paper at ICLR 2021
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli; and Pascal Frossard. Optimism in the face of adversity: Understanding and improving deep learning through adversarial robustness. arXiv preprint arXiv:2010.09624 2020.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Kun Xu, Hang Su, and Jun Zhu. Boosting adversarial training with hypersphere embedding. NeurIPS, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, pp. 8026-8037, 2019.
M.J.D. Powell. Nonlinear Programming. New York: Academic Press, 1970.
Rajeev Ranjan, Carlos Castillo, and Rama Chellappa. 1.2-constrained softmax loss for discrimi- native face verification. arXiv preprint arXiv: 2017.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning reweight examples for robust deep learning. arXiv preprint arXiv: 1.X3iv1803.060 09050,00000.0 2018.
versarially robust generalization requires more data. In NewrIPS, pp. 5014-5026, 2018. Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad-
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer. Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NeurlPS, pp. 3358-3369, 2019.
Chuanbiao Song, Kun He, Jiadong Lin, Liwei Wang, and John E Hoperoft. Robust local features for improving the generalization adversarial training. In ICLR, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. ICLR, 2014.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. TPAMI, 30(1 19958897788 2008
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, number 2019.
Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli. Are labels required for improving adversarial robustness? arXiv preprint
13755 2019.
Yisen Wang. Xingjun Ma. Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In CVPR, pp. 322-330. 2019a.
Yisen Wang, Din Zou, Jinfeng Yi, James Bailey, Xingjun Ma. and Quanquan Gu. Improving adver- sarial robustness requires revisiting misclassified examples. In ICLR, 2019b.
Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning model the tail. In NeurlPS, pp- 7039,7033,,0 2017.
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2019.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. ICLR, 2019.
Sergey Zagoruyko and Nikos Komodakis. 160.00160 2016.
Wide residual networks. arXiv preprint
Runtian Zhai, Tianle Cai. Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint 1906:/990600050 2019.
11

---

Under review as a conference paper at ICLR 2021
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In JCML, 2019.
Tianyuan Zhang and Zhanxing Zhu, Interpreting adversarially trained convolutional neural net- works. In ICML, pp. 7502-7511, 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. 7PAMI,
2017.
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. Parallelized stochastic gradient descent. NeurIPS, pp. 2595-2603, 2010.
12

---

Under review as a conference paper at ICLR 2021
A THE NUMBER OF CLASSES IN EACH ROBUSTNESS INTERVAL OF CIFAR- AND IMAGENET
25
100
Advemanial lod
Adversal
160
140
Class manbbes
Class mambers
120-
05
800
10-
[1.9] (10,2) (2UR) (2RA) (4,90) (T. (90ngpppp))
0U1 (P020p N: (MM W: (SL0])PPX] (hrog) (Ptsr]ee.10 Robusmess interval
Robustncss interval
(a) CIFAR-100(75-th epoch)
(b) ImageNet( (44-th epoch)
Figure 6: Number of classes per robustness interval
Due to the large number of classes in CIFAR- 100 and ImageNet, we randomly sample 12 classes for analysis in the above paper. For the sake of experimental completeness, the number of classes in different robustness intervals is shown in Figure O Obviously, the robustness of the classes is distributed at multiple intervals, which is consistent with the results shown in Figure [
B INTRODUCTION TO THE DATASETS USED IN THE EXPERIMENT
variety of datasets are used for research on adversarial training. Here, we introduce in detail the six datasets used in our experiment.
MNIST. MNIST LeCun
1998) is handwritten digit dataset, containing numbers 0 to 9. The dataset consists of 60,000 training images and 10,000 test images, with 6,000 and 1,000 images per digit. All images are fixed size (28x28 pixels) with a value of 0 to 1, and these digits are located in the center of the image. This dataset is widely used in adversarial Training(Made) et al. 2018
Zhang et al. 20199 Wang et al. 20196 Carmon al. 2019).
CIFAR-10 & CIFAR-100. CIFAR-10 & CIFAR-100 (Krizhevsky et al. 2009) are labeled subsets of the 80 million tiny images dataset (Torralba et al. 2008). CIFAR-IO 10 consists of 50,000 training images and 10,000 test images in 10 classes, with 5,000 and 1,000 images per class. CIFAR-100 has the same total number of images as CIFAR-10, but has 100 classes. Thus CIFAR- 100 has only 500 training images and 100 test images per class. All images in these two datasets are 32x32 three- channel color images. As mentioned in Section 3 CIFAR- 10 can be grouped into 2 superclasses and CIFAR- 100 be grouped into 20 superclasses. CIFAR-10 is the most popular dataset for adversar- ial training Dong et al. (020) and all proposed methods are evaluated in this dataset. CIFAR- 100 is more challenging than CIFAR-10, Shafahi al. (2019) and Song al. (2019) evaluate their defense methods in CIFAR-100.
SVHN. SVHN (Netzer et 2011 is similar in flavor to MNIST and both of them contain 10 digits. SVHN contains 73,257 labeled digits for training. 26,032 labeled digits for testing and over 600,000 unlabeled digits images for semi-supervised or unsupervised training. In order to maintain MNIST-like style, SVHN crops the image to size of 32x32. As a result, many of the images do contain some distractors the sides. Due to SVHN is obtained from house numbers in the real world, its data distribution is more complicated than MNIST. Uesato et al. (2019) and Zhai et al. (2019) use this dataset for cemi-supervised adversarial training.
STL-10. STL-10 (Coates al. 2011 is inspired by the CIFAR-10 and sampled from images in the ImageNet (Deng et al.) 2009). STL-10 also contains 10 classes and each class has only 500 training images and 800 test images. The size of all images is 96 96. As mentioned in Section E STL-10 can be grouped into superclasses. Song et al. (2019) use this dataset to evaluate their proposed method.
13

---

Under review as a conference paper at ICLR 2021
ImageNet. ImageNet (Deng al., 2009) is a high-resolution image dataset with 1000 classes. It contains 1,281,167 training images, 50,000 validation images and 100,000 test images. Since the test set has no labels, the validation set is often used to evaluate model robustness. The speed of training a robust model on ImageNet is intolerable, so using this dataset to evaluate model robustness usually requires some accelerated training techniques (Shafahi et al. 2019 Wong et al." 2019).
C COMPARE CLASS-WISE GAPS BETWEEN ADVERSARIAL TRAINING AND STANDARD TRAINING
Clase 59-Clan 50)
Parcangofii
ParcatigefSi)
Snteett tramiigg
Robrsttess/MMilr training?
20
M.s Clas 4.din
20
Class Cl
20
M2iiMMas Chars OMdnn Class 9)
Chbs 31
0
AI
79
50
Class
Clasc
Class
(a) MNIST
(b) 10
(c) CIFAR-100
5)
6l. Claas
S1 Cls C. 5)
40-
74.7% 22- 890)
Pancstupet))
50-
40-
80-
40-
40-
40-
20
C45c S7 1-3haw
20-
Classcs
(d) SVHN
(e) STL-10
(f) ImageNet
Figure 7: Class wise accuracy in standard training and wlssswwise robustness in adversarial training
In the above paper, we mainly focus on the robustness gap in the model obtained by adversarial training. Since Wang et al. (2019a) also report the emergence of unbalanced accuracy in the standard model, we compare this phenomenon with that in the robust model to highlight the differences between adversarial training and standard training.
Experimental setup. We use the pre-trained model of Pytorch (Paszke et al. 2019 as the standard model for ImageNet. For other datasets, the experimental settings of standard models are the same as the robust models mentioned in Section 3 but adversarial examples are not added to the training set.
Figure 7 shows the class-wise accuracy in standard training and class-wiss robustness in adversarial training. The slashed part in each sub-figure represents the largest gap in accuracy/robusttess among different classes, and the classes in the bracket represent the highest and lowest accuracy/robustness class. Note that natural test images are evaluated for standard training, while adversarial test images are used for adversarial training.
Results Analysis. As illustrated in Figure 7 the relative order of accuracy/robustness of different classes is almost the same in standard training and adversarial training, but the class-wise gap is enlarged adversarial training. For example, CIFAR-10 10 and SVHN datasets, their largest class- wise accuracy gap in standard training are (Figure 7( and (Figure (7d)). but these indicators are enlarged to 52.8%(Figur 7 and 37 (Figure 7 in adversarial training. In more complex datasets, such as STL10, CIFAR- 100 and ImageNet, although standard models also have imbalanced accuracy between classes, these gaps in adversarial training are still larger.
Conclusion. The performance discrepancies among classes in the robust model are larger than that of the standard model. e.g., the largest gap in CIFAR- 10 is enlarged from 10.7% in the standard model to 52.8% in the robust model.
14

---

Under review as a conference paper at ICLR 2021
D T-SNE VISUALIZATION OF CIFAR-10
Class 0
Class
Class
Class 5
Class ]
Class
Class
Class 6
Class
Class
Class
Class 7
Class
Class
Class
Class
Class
Class
(a) Natural test images
(b) Adversarial test images
Figure 8: t-SNE visualization of CIFAR-10
In the evaluation phase, we show the t-SNE (Maaten & Hinton 2008) visualization CIFAR-10 to further demonstrate the property of the two superclasses Transportation and Animals in this dataset. The results of Figure [(a] and Figure 8(b) show that the datapoints between Transportation (class 0,1,8,9) and Animals (class 2,3,4,5,6,77 can be clearly distinguished, while the datapoints between the inner classes in Animals are completely confused, which confirms our analysis Section 3.1
E HYPERPARAMETERS FOR POPULAR ATTACKS ON THE STATE-OF-THE MODELS
FGSM setup. Random start, maximum perturbation 0.031.
PGD setup. Random start, maximum perturbation e = 0.031. For RST model, step size = 0.01 and steps 40, following Carmon al. (2019). For other models, step size 0.003 and steps
20.
maximum perturbation times n 1000, learning rate Ir = 0.005, initial constant Co = 0.01, 7 decrease factor 0.9. Similar to Carmon et al. (2019). we randomly sample 2000 images evaluate model robustness, and 200 images per class.
CW 00 setup. Binary search steps
Transfer-based attack setup. All settings are the same to PGD for the substitute standard model.
0.031, population size Npop 300, noise standard deviation 0.1 and learning rate Ir = 0.02. Similar to Li et al. (2019). we randomly sample 2000 images to evaluate model robustness, and 200 images per class.
N attack setup. Random start, maximum perturbation
F ABLATION EXPERIMENT OF TEMPERATURE-PGI ATTACK
The data in Table similar to Table 3 except that 1/T is different. Combined with the results of T we can find the robustness of vulnerable classes (e.g., class 4) in TRADES, MART and RST has significant decrease in all 1/T settings. When 1/T is set to 2, the overall robustness of the model trained by Madry et al.) (2018) is no exception reduced by 0.26%, with the most significant decrease by 2.8% in class 4. Furthermore, the decline the overall robustness Madry's 's model indeed lower than that in other models, One possible explanation is that these improved robust models may obfuscate gradients AAthalye et al. (2018) in vulnerable classes, and theoretical analysis left to the future.
15

---

Under review as a conference paper at ICLR 2021
Table 4: Relative robustness (%) between Temperature-PGDD and vanilla PGD
Defense
Tot.
0
2
5
7
Madry(1 /T
-0.26
-0.40
+0.40
-0.30
+0.40
-2.80
+0.60
-0.80
+0.30
-0.10
+0.11 11
TRADES(1 /T = 2)
-1.24
-0.70
-0.40
-0.79
-2.30
-2.90
-0.60
-2.10
-0.90 -0.80
-1.00
-0.80
MART(1/T /T
-2.53
-1.30
-0.91
-2.10
-5.40
-6.30
-1.20
-3.00
-0.80
-2.40
RST11/T = 2)
-1.72
-0.80
-0.50
-2.30
-2.40
-3.50
-1.40
-3.40
-0.99
-1.00
-0.90
Madry(1/7 /T 3)
+0.40
+0.10
+1.00
-0.10
+0.10
-2.60
+1.10
-0.20
+2.00
+1.50
+1.11
TRADES(1 /T =)
-1.68
-0.90
-0.50
-1.09
-3.20
-4.30
-1.10
-2.70
-1.11
-1.20
-0.70
MARR(/ /T
-3.28
-1.90
-1.00
-3.00
-7.40
88.00
-1.60
-3.60
-1.21
-2900
-2.00
RST1/TT = =3)
-1.96
-1.10
-0.70
-2.10
-2.00
-4.20
-1.30
-5.10
-1.09
-1.10
-0.90
Madry(1/7 /T
+1.16
+0.80
+1.60
+0,40
+1.00
-1.61
+1.40
+0.50
+3.30
+2.10
+2.11
TRADES(1 /T
-1.77
-1.01
-0.50
-1.09
-3.10
-4.70
-1.21
-3.00
-1.00
-1.10
-1.00
MARRI /T
-3.73
-1.80
-1.00
-3.50
-8.40
9.70
-1.90
-4.00
-1.40
-3.30
-2.30
RST(1 /T 4)
-2.04
-1.00
-0.60
-2.41
-2.20
-4.20
-1.60
-5.20
-1.09
-1.30
-0.80
Madry(1/1 /T"
+2.97
+2.40
+2.10
+1.60
+2.90
+0.20
+3.70
+4,20
+5.30
+4,30
+3.01
TRADES(1 /T
-1.69
-0.70
+0.00
-1.19
-5.10
-0.90
MART(/ T 77)
-3.85
-1.80
-0.70
-3,70
-8.90
-10.90
-2.10
-4.60
-1.21
-3,20
-1.40
RSTU/T = =7)
-1.84
-0.60
-0.40
-1.90
-2.00
-4.30
-1.50
-5.00
-0.79
-1.00
-0.90
Madry(1/1 / 10)
+4.96
+3.20
+3.20
+3.70
+5.50
+3.30
+6.60
+8.20
+6.20
+5.70
+4.01
TRADES(1/2 /T 100)
-1.15
+0.30
+0.50
.4700
-2.90
+0.00
MARTU/ T 10)
-3.29
-1.30
+0.60
-3.50
-9.00
-10.30
-1.80
-4.70
-0.10
-2.30
-0.50
RST(//T- = 10)
-1.60
-0.70
+0.00
-1.70
-1.90
-4.00
-1.30
-4.80
-0.49
-0.30
-0.80
G IMPROVE THE DEFENSE OF PANG ET AL. (2020)
Recently, Pang et al. 2020 combine feature normalization (FFN (Ranjan et al. 2017). weight nor- malization (N) (Guo & Zhang 2017) and angular margins(AM) (Liu et al. 2016) to propose defense method that can boost model robustness. In their paper, they believe that using FN and WN to limit embeddings on the hypersphere is the key to robustmess improvement, so they call their method hypersphere embedding(HE) However, we find that the factor (temperature factor in our paper) for scaling WN is the real key to improve robustness. We try to understand this from a point of view they overlooked.
Our analysis mainly focuses on FN and WN, following Pang et al. (2020). For a better formulation, we assume the input sample is the extracted feature of an example in the penultimate layer is classes in the dataset C, the scale factor 8 (corresponds to the 1/T in our paper) and the parameter of the last classifier is W {w:} E RdxC" where w; E Rd is the classifier weight corresponding to class i. Then FN operation is 2 = 2/||=||2, WN operation is wi w 2|2 and the probability P. of this sample corresponding to class i (i C) after the softmax function S is
z Rd, the number
E//T
P. = S(2)
(3)
e 1//TT
k=
k=
The gradient of cross-entropy loss Lce corresponding to wi is
{
(P, 1) i= P.
V Vw, LCE
(4)
i i
Equation 3 and Equation (4) suggest we can change P; by adjusting T and finally control V w., Log- e.g., the maximum probability for an example is class 1, then P. = 1 if 1/T - 00 and P. = 1/C 1/T 0.
For better demonstration, suppose that there is three classification task (Figure The top three figures represent equation (3) and the bottom three figures represent the corresponding equation (4). The light-colored bars in the top three figures indicate the sum of the probabilities should be equal to 1. The bars corresponding to the pink arrows in Figure 9b) and Figure (c) have the same length, and Figure and Figure ( are similar.
Specifically, we consider an adversarial example, i.e., the highest probability of this example is usually the incorrect class, Figures (aa) and (d) represent the probability and gradient of this example after softmax. When 1/T > (Figures (9)) and (())), this example is considered by the model as harder example (the probability of the adversarial class becomes larger), so that the
16

---

Under review as a conference paper at ICLR 2021
Ground Truth Class
Adversarial Class
Other Class
Probubility
Probability
Probability
Class
Class
Class
(a) Probability (1/T=
(b) Probability (1/T > 1)
(c) Probability (1/T 1)
Gradient
Gradient
Gradient
Class
(d) Gradient (1/1'= = 1)
(e) Gradient (1/11 > 1)
(1) Gradient (L/"l 1)
Figure 9: The relation between the probability and gradient for adversarial example
model can update more gradients based on this example. Similarly, when 1/T < (Figures (c mn)) this example becomes simpler for the model and obtains smaller gradient. Overall, these figures clearly show that we can adjust the gradient through the temperature factor 1/T.
According the above analysis and learn from the ideas of curriculum learning (Bengio er al,) 2009). we believe the gradient should be gradually enlarged (by increasing 1/TT) during adversar- aal training, instead of using a fixed 1/T like Pang et al. (2020). Specifically, the following two schedules to adjust I/ are proposed:
Linear Interpolation Schedule(LIS), We use a simple linear interpolation schedule to adjust 1/T. Therefore, the 1/T of th epoch is
1 Tn
(5)
TIFT
TFI
To
our implementation, the initial temperature factor 1/To and the final temperature factor of the interpolation 1/TFr 75, where the subscript F/ is short for final interpolation. The final epoch of the interpolation is equal to the total training epoch nfI = Nuot = 100. The training pipeline follows Panlows/aangaaa et al. (2020), but we do not use angular margins. Other hyperparameters are the same to Section 3
Control Maximum ProbabilityCCMPP) At each epoch, 1/T according to equation ( to control the maximum probability. Since equation (3) is a nonlinear function, Powell's dogleg method (Powell 1970) is used to solve this function. Therefore, the maximum probability pmiee of all examples in epoch is
accurately calculate the required
pmaz pmace
(pmax -mmaa)
S < RFI
(6)
pmaz
nFI Root
In our implementation, the initial maximum probability pr 0.2 and the final maximum proba- bility of the interpolation Pmax 1. The final epoch of the interpolation nrr = 90, while the total training epoch Niat 100. For the last 10 epochs, the maximum probability always controlled at 1. Other settings are the same to Linear Interpolation Schedule.
We use the above methods to train ResNet-18 on CIFAR-10 and choose vanilla 2GD attack to evaluate model robustness. For fair comparison, we also re-train ResNet18 following Madry et al. (2018) and Pang et al. (220)). As shown in Table our methods HE-LIS and HE-CMP have further boosted the model robustness. Especially, the improvement of vulnerable classes (e.g.. class 3 and class 4) is impressive. This seems to be an exciting result because the robustness gaps between classes are largely reduced.
17

---

Under review as a conference paper at ICLR 2021
Table 5: Adversarial robustness (accuracy (%) on vanilla 2GD on CIFAR-10.
Defense
Nat.
Adv.
0
2
3
4
5
8
9
Madry
84.0
51.7
57.6
74.9
37.6
22.1
30.7
41.2
55.6
64.0
71.0
62.0
HE
83.1
60.7
+7.2
+2.1
43.2
+12.2
+99.1
+10.4
+11.8
+6.6
+6.2
+11.2
Ours(HE-LIS)
84.6
63.9
49.7
+2.3
+10.6
+20.6
+31.5
+10.1
+15.8
+3.7
+6,2
+12.1
CMPP
84.4
69.9
+14.3
+4.0
+14.6
+29.5
+41.1
+18.1
+22.7
+6.7
+13.0
+18.7
"+" represents robustness improvement compared with the corresoonding element the Madry's model.
Unfortunately, these improved methods cannot defend the Temperature-PGD attack. As shown in Table 6 the numbers the first row indicate the different 1/T.. The improvement disappears when 1/T in Temperature-PGin is large enough. We have analyzed it in Section 4
Table 6: Adversarial robustness on Temperature- PGr of different 1/T
Defense
5
10
50
75
100
125
150
200
300
Madry
51.7
50.8
55.2
68.1
71.0
73.1
74.4
75.6
77.4
79.3
HE
60.7
57.2
54.1
50.4
50.7
51.2
51.7
52.3
53.2
55.3
Owrs(HE-MMP)
69.9
67.8
65.1
54.1
52.2
51.2
50.7
50.4
50,3
50.9
18


Reviews:
[{"review_id": "sMHWEm3Tuwq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a detailed analysis of adversarial training on several datasets (including CIFAR-10/100 and ImageNet), and observes three interesting (class-wise) properties: (1) some classes are extremely vulnerable to adversarial attacks; (2) for a certain class, its robustness is positively correlated to the norm of its FC layer's weight; (3) stronger attacks usually hurt vulnerable classes more. These results could be helpful for future works on furthering model robustness.\n\n\nPros:\n\n(1) This paper is very clearly written and easy to follow.\n\n(2) To the reviewer's best knowledge, this is the first attempt to analyze how adversarially trained models behave at the class level (rather than just reporting the overall performance on the whole dataset). And the observations are kind of surprising, e.g., there indeed exists certain classes that are much more vulnerable than others, and the observed confusion patterns (e.g., \"class 9\" and \"class 1\" in CIFAR-10 are easily confused with each other) are pretty consistent among different defense methods. \n\n\nCons:\n\nThe reviewer believes some conclusions of this paper can be further enhanced if analysis can be provided for the following parts: \n\n(1) In Section 3.1, this paper finds certain (pairs of) classes are extremely vulnerable to adversarial attacks. As there are already some works that begin to take care of vulnerable samples (a more fine-grained level than vulnerable class) in adversarial training [1,2], it will be good to analyze if such customized strategies successfully reduce class-wise confusions under adversarial attacks.\n\n(2) In Section 3.2, this paper observes that the robustness is correlated to the norm of weight, and hypothesis such phenomenon is caused by insufficient data in adversarial training. As there are already several works [3,4,5] on augmenting extra data for improving adversarial training, the authors can use these models to validate the hypothesis above. i.e., with more training data, will the robustness and the norm of weight become less related?\n\nBesides, the reviewer is a little bit confused about the proposed attack in Section 4.  What is the motivation for adding the temperature parameter in PGD attack? Why it can hurt certain classes more? It seems that only empirical results are provided, but no motivation or explanations of why it can work are provided.  (which makes this section less related to the core parts of this paper and easily confuse readers)\n\nThe authors should carefully address these concerns during the rebuttal.\n\n\n[1] Cheng M, Lei Q, Chen P Y, et al. Cat: Customized adversarial training for improved robustness[J]. arXiv preprint arXiv:2002.06789, 2020.\n\n[2] Balaji Y, Goldstein T, Hoffman J. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets[J]. arXiv preprint arXiv:1910.08051, 2019.\n\n[3] Zhai R, Cai T, He D, et al. Adversarially robust generalization just requires more unlabeled data[J]. arXiv preprint arXiv:1906.00555, 2019.\n\n[4] Carmon Y, Raghunathan A, Schmidt L, et al. Unlabeled data improves adversarial robustness[C]//Advances in Neural Information Processing Systems. 2019: 11192-11203.\n\n[5] Uesato J, Alayrac J B, Huang P S, et al. Are labels required for improving adversarial robustness?[J]. arXiv preprint arXiv:1905.13725, 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "almCEh0y9E", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically studies the class-wise properties of classification models produced by existing adversarial training methods on benchmark image datasets. In particular, it demonstrates the following observations: 1) robustness varies for different seed-target class pairs; 2) for certain class, retraining the model without other semantic-similar class improves its robustness; 3) class-wise robustness relates to the norm of the model weights; 4) stronger attacks are more powerful for vulnerable classes.\n\n\nPros:\n\n+ The paper conducts extensive experiments, including six image benchmarks.\n\n+ Heatmaps of class-wise robustness is visually good.\n\n+ The fine-grained analysis on class-wise properties help understand the model performance to some degree.\n\n\nCons:\n\n- The motivation of this paper seems somewhat unclear. The paper motivates its main research question based on the observation that most existing works in adversarial community focused on overall robustness. However, it is unclear to me why focusing on overall robustness is inferior. From my perspective, it is important to clarify the following questions in the introduction: 1) If you aim to improve the overall robustness using class-wise properties, what are the potential solutions? 2) If you think overall robustness is not the best evaluation criteria for adversarial robustness, what are the reasons and alternative criteria?\n\n- The related work section simply listed many existing works related to adversarial training. The connections between these works and your work are not well-explained. More specifically, how will the understanding of class-wise properties contribute to the improvement of existing works is unclear.\n\n- There is a lack of coherence in Sections 3 and 4. The subsections demonstrated in Section 3 are somewhat disconnected with each other. For instance, the transition from Section 3.1 to Section 3.2 is not well-explained. In addition, Section 4 really confuses me, i.e., the proposed Temperature PGD Attack seems not related to the previous sections and does not have an advantage over the vanilla PGD attack.\n\n\nMinor comments:\n\n- It is hard for me to see a strong positive correlation from Figure 4. I would recommend the authors to compute the correlation coefficient or conduct a correlation test to support the argument.\n\n- The paper will be much stronger if the authors can improve the performance of existing robust models by leveraging the class-wise properties discovered in Section 3.\n\n======= POST-RESPONSE UPDATE ========\n\nI appreciate the author's efforts for responding my questions and providing additional results and I do find the empirical observations of class-wise properties interesting. However, I still feel that the contribution of the current form of the paper is not strong enough to reach the bar of ICLR, so I remain my previous rating. Beyond exploratory analysis, the paper would be much stronger if it can go deeper with the observed class-wise properties of robust models. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7wnqQNOSogA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the robustness of adversially-trained models across different classes. They find that classes tend to have largely non-uniform robust accuracy--i.e., some are less robust then others. Moreover, certain datapoints can only be misclassified as specific classes and removing these classes during training can make these datapoints robust. Next, the authors investigate how this robustness discrepancy across classes relates to the norm of the last fully-connected layer of the model, as well as to the strength of the attack used. Finally, the authors propose a new adversarial attack that they evaluate against existing models.\n\nThe observation that robust models can have widely varying robust accuracy across classes is quite interesting. At the same time, the insights gained from the current study are not particularly impactful. Specifically:\n- **Robust accuracy disparity.** Before drawing conclusions, it is important to perform the same study on a standard model. Are standard class accuracies on these datasets closer to uniform? Note that this will require accounting for the variance introduced by the overall lower accuracy of robust models.\n- **Certain examples can only be flipped to specific classes.** Why is this surprising? It is natural to expect that classes that are similar to each other will be more likely to be flipped on multiple examples. As a result, removing one of these classes completely from the dataset will result in the other class becoming more robust.\n- **Class robustness is tied to classifier norm.** First of all, the correlation of these quantities needs to be more formally investigated. Clearly, this is not a perfect correlation so we need at least some quantitative analysis to support this conclusion. Nevertheless, it is not clear what conclusion we can draw from this finding. Is the larger norm a cause or a symptom of brittleness? Can we make certain classes more robust by reducing their weight norm? Does this allow us to understand something fundamentally new about robust models?\n- **Temperature scaling attack.** It is not immediately clear what the intuition behind this attack is. From what I understand, it will cause PGD to focus more on classes with higher (original) softmax probabilities. However this is not a fundamentally new attack. Moreover, if this viewpoint is correct, its performance should be lower than that of an attack that exhaustively considers all classes and tries to produce a targeted misclassification (e.g., MultiTargeted https://arxiv.org/abs/1910.09338).\n\nOverall, while the topic and initial exploration is interesting, the findings do not provide us with a fundamentally new understanding of robust models. \n\n======= POST-RESPONSE UPDATE ========\n\nI appreciate the authors' response and additional experiments. Unfortunately, my criticism still stands:\n- **Comparison with standard models.** Based on Appendix C, it is still unclear if the class-wise disparity is a unique property of robust models. As we can see in Figure 7, standard models also have disparate accuracies between classes. More importantly, comparing the **absolute difference** between class accuracies is misleading, since robust models have overall lower robust accuracy. A better comparison would be to measure the **relative error** between classes. While it is hard to draw conclusions by just inspecting the graph, it seems that the discrepancy is significantly milder based on this metric.\n- **Classifier norm.** The additional experiments still do not demonstrate any causal link between classifier norm and robustness. It is thus still unclear what this metric conveys.\n- **Temp-PGD.** While I appreciate the effort to provide additional intuition about the attack, I still do not find the method fundamentally new when compared to other attacks optimizing combinations of logits (e.g., Carlini-Wagner, Multi-Targeted).\n\nOverall, I still think that the original observation is intriguing, yet requires a deeper and more systematic study.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4mR5jaKW5gp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper examines the robustness of adversarially robust models at the class-level.  Specifically, they note a disparity in the class-wise robustness of models for standard datasets. Furthermore, they suggest that many of these class-level vulnerabilities are eliminated if the model is trained without the corresponding confounding class. Finally, they propose a temperature based attack to further degrade accuracy of vulnerable classes.\n\nComments: I think that fine-grained studies of existing robust models can be valuable to identify ways to improve their performance.  While the paper is based on an interesting finding, the analysis is not sufficiently novel or extensive, and does not lead to a better understanding of why this phenomenon occurs or actionable insights to improve the performance of robust models.\n\nNovelty: This paper is not the first to note the disparity in class-wise robustness---for instance, a fairly extensive study has been performed in https://arxiv.org/abs/2006.12621. Even if one were to consider both works as concurrent, the paper under review lacks depth and it is not clear what, if any, the impact of its findings are (see below). Relatedly, there is a line of work on adapting adversarial training to account for instance-level disparities in robustness---for instance, https://arxiv.org/abs/1910.08051.\n\nTakeaways: The authors do not provide any conclusive evidence for the cause of class-wise disparities in robustness. They claim that this is linked to the norm of the weights, however they do not establish a causal link between the weights and per-class robustness (Fig. 4 could simply be correlation). Moreover, I expect that a similar relationship would be observed between standard and robust per-class accuracies (from Appendix Figure 6)---instead of the weight norm and robust accuracy---if both these quantities were normalized. In general, it is also tricky to make inferences based on the norm of the weights (without proper normalization) as the outputs of individual neurons in the pre-final layer could have vastly different scales. Additionally, the authors do not demonstrate that their findings can help to improve model robustness.  At the end of every section, the authors do provide some suggestions, however these are vague and unsubstantiated.\n\nOverall, I think in its current form, this paper lacks depth and novelty. The observation of class-wise disparities alone is not substantial enough a contribution. As I mentioned above, the authors do not supplement this finding with a clear explanation of the underlying cause, or any evidence that this finding could help improve model robustness.\n\nAdditional comments:\n\n* Are the confusion matrices based on targeted or untargeted attacks? If it is the later, then these numbers do not provide a complete picture of class-wise robustness. For an example of class i, it is possible that the adv. example that maximizes loss within an eps ball is from class j, though adversarial examples for many other examples exist within the ball (and are even closer to the original sample). The authors should repeat the evaluation using pairwise-targeted attacks.\n\n* Many of the experimental choices are not sufficiently justified. For instance, why FGSM is used for ImageNet training, and why the best model is not picked based on cross validation. Further, the authors do not once mention what kind of adversary they consider.\n\n* It is clear from Fig. 6 that, for most datasets, the same classes with lower robust accuracy have lower standard accuracy. Are the confusion matrices in Fig 2 based on all test set examples or just the ones that are correctly classified in the non-adversarial setting? What do the standard confusion matrices for these datasets look like?\n\n* It is odd that the second element in the first row in Figure 3b (18) is larger than the corresponding element in Fig 3a. From what I understand, 3a should be an upper bound on every element in 3b.\n\n* When evaluating the temperature based attack, do the authors apply the final perturbation to the original model (without the temperature scaling) or to the scaled model? If the numbers reported in Table 3 are based on the latter, then they do not indicate the robustness of the original model as the inference process has been modified.\n\n#### Post-rebuttal Update ####\n\nI thank the authors for their detailed response and edits to the paper. However, even after reading the rebuttal some of my original concerns stand:\n\n[Novelty] As I mentioned in my original review, the discovery of disparities in class-wise robustness is not new to this paper. In the rebuttal the authors mention that their finding is different from [1] because in [1] the measure of class-wise robustness is distance to the decision boundary with respect to every other class. However, this is in my view, is just an alternative and well-established measure of robustness---i.e., distance to the decision boundary and robust accuracy are fairly correlated and not fundamentally different.\n\n[Takeaways] \n- The authors do not perform sufficient quantitative analysis to justify the link between robustness and weight norm. Moreover, without establishing the causality of this link, it is unclear to me how this observation provides any new insight to understand robust models. \n- Figure 7 (in the revised manuscript) shows that a similar class-wise disparity is present even in the *standard accuracy* of *standard models*. Thus, although I find the observation of disparities in class-wise robustness interesting, I believe further investigation is needed to understand whether this is just an inherent property of the data distribution that hurts both standard and robust models or is specifically tied to robustness. \n- The authors' comment about improved robustness based on methods adapted from [3] is misleading. As mentioned in Appendix G (I believe there is no Appendix H), this seems to be the case only for a specific attack. The authors themselves demonstrate a different attack under which the improved robustness of vulnerable classes disappears.\n\n[Other comments] There is no evidence to suggest that untargeted attacks will find the *closest* adversarial example within an eps ball. The optimization problem for untargeted attacks is set up to maximize the loss (w.r.t. the ground truth label) and not to find the nearest misclassification. Thus, I still assert that to get a better picture of per-class robustness, the authors need to measure the targeted confusion matrix (or distance to per-class decision boundaries as in [1]).\n\nDue to these concerns, I am unable to raise my score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Analyzing class-wise adversarial vulnerability of models is an interesting direction to pursue. However, the authors should consult the references pointed out in the reviews to put their contributions in the right perspective. Overall, the lines of inquiry explored in this paper are of interest but, as some of the reviewers point out, there are improvement in the methodology that still need to be addressed before this paper is ready for publication. (I very much recommend that the authors do build on this feedback and revise the paper, as it will be a valuable contribution then.) 
Decision: Reject