Clustering Effect of Adversarial Robust Models
Authors: Yang Bai
Xin Yan
Yong Jiang
Shu-Tao Xia
Yisen Wang
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Adversarial robust models
hierarchical clustering effect
domain adaption tasks

Abstract: Adversarial robustness has received increasing attention along with the study of adversarial examples. So far, existing works show that robust models not only obtain robustness against various adversarial attacks but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear. In this paper, we interpret adversarial robustness from the perspective of linear components, and find that there exist some statistical properties for comprehensively robust models. Specifically, robust models show obvious hierarchical clustering effect on their linearized sub-networks, when removing or replacing all non-linear components (e.g., batch normalization, maximum pooling, or activation layers). Based on these observations, we propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy. Our code is available at https://github.com/bymavis/Adv_Weight_NeurIPS2021.

Clustering Effect of (Linearized) Adversarial Robust Models
Yang Bai Xin Yan Yong Jiang!25 Shu-Tao Xia Yisen Wang
'Tsinghua Berkeley Shenzhen Institute, Tsinghua University
Tsinghua Shenzhen International Graduate School, Tsinghua University
3 Key Lab. of Machine Perception, School of Artificial Intelligence, Peking University *Institute for Artificial Intelligence, Peking University SPCL Research Center of Networks and Communications, Peng Cheng Laboratory, China
Abstract
Adversarial robustness has received increasing attention along with the study of adversarial examples. So far, existing works show that robust models not only obtain robustness against various adversarial attacks but also boost the performance in some downstream tasks. However, the underlying mechanism of adversarial robustness is still not clear. In this paper, interpret adversarial robustness from the perspective of linear components, and find that there exist some statis- tical properties for comprehensively robust models. Specifically, robust models show obvious hierarchical clustering effect on their linearized sub-networks, when removing replacing all non-linear components (e.g., batch normalization, max- imum pooling, or activation layers). Based on these observations, we propose a novel understanding of adversarial robustness and apply on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy. Our code is available at httpss//gitubb.co/byyaawsss Laaeaaiis//dyWeegihhht NeurlPS20211
1 Introduction
Nowadays, deep neural networks (DNNs) have shown a strong learning capacity through a huge number of parameters and diverse structures [6 [. Mcanwhile, adversary has raised increasing security concerns on DNNs due to the observation of adversarial examples (i.e., the examples mislead classifier when crafted with human imperceptible but carefully designed perturbations) [K3 Adversarial robustness and defense techniques have thus become crucial for deep learning, yet many adversarial defense techniques found with deficiencies such as obfuscated gradient [2]. Up to now, the widely accepted techniques to improve adversarial robustness are adversarial training variants 35 36 by training DNN after data augmentation with the worst-case adversarial examples, Recent study has found that not only do robust models show moderate robustness under vast attacks. but also they can surprisingly boost the downstream tasks (e.g., domain adaption task with subpopulation shift) 29
A
However, the underlying mechanism of adversarial robustness is still not clear. Existing studies have explored adversary on some specific components of DNNs, such as batch normalization [12), skip connection [2] or activation layers 14 4 which yield some enlightening understanding. On the other hand, despite variation in DNNs' structure, the components can be roughly divided into linear and non-linear ones. The non-linear components tend be instance-wise. For example, adversarial
'Equal contribution. "Corespondence to: Yisen Wang
and Shu-Tao Xia (xiast@s.ssinghuu.cdu..o)
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

CIFAR-10
Non-animal
Animal
(a) Class Hierarchy (b) VGGI6-STD
(c) VGG16-AT
(d) VGGI6-AT+C
Figure I: Correlation matrices of VGG-16 on CIFAR-10: b) 'STD' and c) 'AT' indicate standard and adversarial training, d) 'A++C' indicates applying hierarchical clustering in adversarial training. Robust models tend show clustering effect aligned with a) the class hierarchy of CIFAR-10, which is enhanced in d) our clustering adversarially trained model.
and benign examples tend to present different patterns activation components, once the optimization done and model evaluation mode is activated, they are shared and fixed independently from inputs. Nowadays, the intrinsic relationship of adversary with linearity has been studied. Some ascribes the adversary to linear accumulation of perturbations from inputs to final outputs, dubbed 'accidental steganography" [K] Some show that linear backward propagation enhances adversarial transferability [4 [4] Nevertheless, such mentioned linearity analyses are all on the original non-linear models, which hinders deeper understanding on linearity directly. Further
4 In contrast, for those linear
essential exploration is encouraged.
In this paper, we thus study adversarial robustmess the statistical regularity of linear components. We provide a novel insight on adversarial robustness by showing the clustering effect well aligned with class hierarchy. As shown in Fig. [d), given data set, superclasses defined by grouping semantically similar subclasses together to form a hierarchy, egg., in CIFAR-10, 'Cat' and 'Dog' represent different subclasses/fine labels, despite both being the same superclassesccoarse labels 'Animal'. Based on this, we conduct backward propagation on linearized sub-networks of robust and non-robust DNNs (through adversarial/standard training respectively) to extract the corresponding implicit linear expression. In this fashion, we could obtain Dimpt X Doutput linear matrix W. which extracted from inputs to predictive outputs. D Dmpptt represents for the dimension of one input data, specifically defined as width x height x channel. Doutput represents for the dimension of one output vector, which is also the number of classes. We further study the correlation of vectors in W by matrix C, whose shape is Doutput x Doutput- The details of W and C are introduced in Sec. 3 Then we find that robust models tend show hierarchical clustering effect in correlation matrix C. which is surprisingly consistent with class hierarchy. For example, in Fig. on CIFAR- 10, the fine labels (0,1,8,9) belong to 'Non-animal', others belong to 'Animal.. The matrix C' of robust model shows block clustering: values are close to '+1' in the two superclasses 1,.9, and (2,3,4,5,6,77 7), and values are close to '-1' across these two superclasses. Such clustering effect enhanced by our clustering regularization penalty in Fig. [(d] This phenomenon could be semantically explained as subclasses from the same superclass share more feature similarities. Such hierarchical clustering effect can give novel and insightful explanation on the superiority of robust models in some observed downstream tasks, as they can extract more semantic and representative features. To further confirm our findings, we enhance the hierarchical clustering effect in both adversarial robustness and downstream tasks, e.g., domain adaption proposed in 29 22). The improvements of experimental results demonstrate that the superiority of robust models is closely related to their hierarchical classification capability. Our contributions are summarized as follows:
To the best of our knowledge, we are the first to systematically analyze the statistical regularity of adversarially robust models (through adversarial training) compared to non- robust models (through standard training) on their linearized sub-networks.
We present an intriguing phenomenon of hierarchical clustering effect in robust models, and provide a novel yet insightful understanding of adversarial robustness. The clustering effect aligned with class hierarchy demonstrates more semantic and representative feature extraction capacity of robust models, which benefits a lot in various tasks.
Based on the observations, we propose a plugged-in hierarchical clustering training strategy to generally enhance adversarial robustness and investigate some intriguing adversarial attack findings. Besides adversarial-relatee study, we further explore some downstream tasks with

---

the understanding of hierarchical clustering, e.g., domain adaption with subpopulation shift. Experimental results show that the clustering effect and hierarchical classification learned by robust model benefits the task as well.
2 Related Work
2.1 Adversarial Attack and Defense
Attack Given a clean example æ, its true label y, a DNN model F with parameters 0 and its loss function L. the goal of adversarial attack is to find adversarial example z' that is close clean example on pixel level but can fool F to give an incorrect prediction. Fast Gradient Sign Method (FGSM) | adds perturbations clean example z by one step with step size 0), 0),
(1) 24 adds perturbations on clean example by steps with smaller step size a. After every step (k-th) attack, adversarial example is projected into the e-ball (x)) around clean example (function II):
x²
x + essinninnv
Projected Gradient Descent (PGD)
[I -~ + asign (
.0).y)). (0))))))
There are other attack techniques, such as white-box CW attack 6 black-box attacks 21 3 and
adaptive AutoAttack 9
Defense Many adversarial defense techniques have been proposed since then, such as input pre- processing [19], defensive distillation 24 model compression [TC), and adversarial training [23).
Among them, adversarial training variants are assumed to be most effective in comprehensive attack evaluations. They solve min-max problem as:
min max (FF(x'.).y)) (x)
(3)
The inner maximum problem often generates adversarial examples by FGSM or PGD, while the outer minimum problem optimizes the worst-case loss. Some defense technique improves robustness through wiss feature clustering clustering.
[| 30
which is different from our instance-wise feature
2.2 Lincarity Exploration in Adversary
Beside attack or defense techniques, the cause and understanding of adversary studied. The lin- earity is one essential perspective with no doubt. Previous studies attribute the existing of adversarial examples to the linearity of DNNs [K] and observe the adversarial transferability enhancement on more linear models 34 14] Some works study linearity from another perspective by focusing on the properties of weight layers, e.g., their norms, variances and orthogonality. Lp norm regularization works by pulling examples far away from the decision boundary, which comes as the side-effect of adversarial robustness 377 [8] Spectral norm 7 is useful in adversarial defense, which is
defined as a, = || and is used to constrain the Lipschiz constant a DNN F on x (assuming æ||2
Lipschiz continuous) [ 8) F(x)]l2 S a||z||2 < o; |2 The weight scale shifting issue is also discussed in adversary [2]. that is, scale weights can be shifted between layers without changing the input-output function specified by the network, which could affect the capacity to regularize models. Then one weight scale shift invariant regularization is proposed and improves adversarial robustness. Moreover, the orthogonality could help improve generalization and adversarial robustness [5] by inducing uncorrelated features. However, these experiments are all conducted on the original non-linear models instead of the linear components directly, which hinders deeper understanding on linearity.
Observations on Linear Components
The robust models perform completely differently from non-robust ones especially under compre- hensive adversarial attacks. However, the statistical regularity on linear components of these robust models is yet under little exploration. Different from related works, we explore linear components directly. To be specific, we extract weight matrix expression of linearized sub-networks, estimating the linear propagation from inputs outputs.
3

---

3.1 Extracting a Linear Weight Matrix
Given DNN FF composed of L linear layers and non-linear components, the output y F(x) could be expressed as
y = (w x 91-1(W1-1 ... (1(1 x a + bb) bb-1) by):
where 9i is series of non-linear components (e.g., activation function ReLUs), w; and b; the expressions of linear ones. Then a corresponding linear sub-network of this original one could be extracted with the same weights and architectures, yet it removes activation function (e.g., ReLU layers) and batch normalization layers, and further replaces maximum pooling layers with average pooling layers. This output of linear network ylinear Flinear (a)) is expressed
ylieear wL X (WL-1 x (.:
bj)... + b + PL-
(5)
Thus in Finear, each input a multiplied by an instance-agnostic total weight
W wt x WL-1 ...WI Σ bi... x Wj...
(6)
However, such total weight expression W in DNNs is difficult compute directly, because the multiplication of convolution layers is infeasible. Instead, we propose to compute W by applying backward propagation on the linear sub-network. To be specific, given a random input and a pre-trained DNN F. W is computed as the gradient propagated on linear sub-network Fiinear following Algorithm ! For non-linear model, as the activation part is example-wise, is difficult to analyse the original network with non-lineat components. In contrast, the linear component is generally applied on all examples and example-agnos
So extract linear expression from the original non-linear model to approximate and analyse the model performance. Intuitively, although the linearized network has limited capacity, we think that the class-wise directions (the extracted linearized weight vectors) should still represent some class-wise directions, which holds a connection with the feature space of the non-linear models.
Algorithm 1 Extracting the Linear Weight Matrix W
Input: A random input æ, pre-trained DNN model F with D Doutput classes
Output: Weight matrix W
Get the corresponding linear sub-network Flinear from F
Conduct forward propagation as: Viinear Flinear (2)
for i range(Dnutpin) do
Vlincar backward()
W|: = x. æ.grad
end for return W
3.2 Observation on Weight Clustering Effect
As indicated, though instance-wise performance varies on one specific DNN with non-linear com- ponents, it shares the same linear sub-network (i.e. parameters and linear structures) and thus the same implicit weight matrix W. As the implicit matrix W could estimate linear output scores Vlinear of any input o when forward propagating on the linear sub-network, hints some class-wise linear amplifications. After extracting W with shape Dmputt x Doutput, we further explore the correlation across classes by normalizing weight vectors in W (to avoid the class-wise scale variance) and computing correlation matrix C' following
WT
W
C1,5
1l2
|IW
In such fashion, the matrix C. whose shape is D Doutput x D Doutput, has all elements in [-1, 1]. C(i.)) represents for cosine value of two weight vectors corresponding to class and j on linear sub-networks, which also represents for the correlation of class-wise weight vectors. If the element C(i.j) is close 1, it means that i-th and j-th class-wise weight vectors are strongly positive correlated the linear weight space, which also demonstrates that class i and j are more likely to be positively related with

---

PISTIEITE FERESSAAIIOO
(a) STL-AT
(b) CIFARIO-AT
(c) CIFAR20-AT
(d) ImageNer10-AT
(e) STL-STD
(f) CIFARIO-STD (g) CIFAR20-STD (h) ImageNetIO-STD.
Figure 2: Correlation weight matrix C' of ResNet-18 on different data sets. 'AT' indicates adversarial training. 'STD' indicates standard training. Robust models tend show weight clustering effect aligned with class hierarchy.
each other. In contrast, for the element C(i.j) close to -1, it means that i-th and j-th class-wise weight vectors are strongly negative correlated in the linear weight space, which also demonstrates that class and j are more likely to be negatively related with each other.
Datasets and Setups Beside Fig. [ we show more clustering effect across data sets using STL-10 [ and two reconstructed data sets CIFAR-20 and ImageNet-10, composed of 20/10 subclasses from 4/2 superclasses in CIFAR- -EFAR--10 and TinylmageNet (27l The details of constructed data sets are in Appendix A For robust models, we adversarially train ResNet-18 18 [15] using PGD-10 (c = 8/255 and step size 2/ /255) with random start. The non-robust models are standard trained. Both models trained for 200 epochs using SGD with momentum 0.9, weight decay 2e-4, and initial learning rate 0.1 which is divided by 10 the 75-th and 90-th epoch.
Result Analysis We plot the correlation matrix C' in Fig. 2 Similar to Fig. D on CIFAR-10, 10, the similar block clustering effect is observed robust models and aligns well with class hierarchy. To be specific, in Fig. 2(a) and 2(e) on STL-10, fine labels (0,2,8.9) belong to 'Non-animal" others belong to 'Animal', and the matrix of robust model shows block clustering: values close '+1' in the two superclasses (0,2,8.9) 9) and (1,3,4,5,6,7), values close to '-1' across these two superclasses. Such phenomena hint at the fundamental connection of adversarial robustness with weight clustering effect. That is, subclasses in the same superclass are positively related to each other, which is in contrast with those in different superclasses. This hierarchical classification property is more semantic with data sets, demonstrating that robust model has a better representative feature extraction capacity and thus outperforms standard model various tasks.
Discussion order to analyse the connection of weight clustering effect on linearized models with their original models, specifically take the linear model model is itself. We additionally train two linear models with initial learning rate 0.01 and 100 epochs by standard training PGD-10 adversarial training. The detailed architecture of the model shown Appendix [F] The final accuracies are listed in Table D Though the model capacity restricts their accuracies, the robust model still shows clustering effect. Fig. 3 indicates that the clustering effect of linear weight matrix generally holds even on a simplified linear model.
example whose linearized
Table 1: Accuracy (%) on CIFAR-10 two linear models. 'ST' indicate adversarial//tanddar training. The model capacity restricts their accuracies.
Linear Model
(CIFAR-10) 10)
AT
STD
CIFARIO-AT
(b) CIFARIO-STI
Clean ACC
29.21% 20.80%
39.53%
Figure 3: Correlation weight matrix C' of linear models on CIFAR-10. 'AT' model shows clustering effect.
PGD-20 ACC
4.36%
5

---

Explorations with Clustering Effect
simple and direct idea is to apply the linear weight clustering effect as a penalty. However, the time cost and computational cost are too expensive. Based on our understanding that robust models learn better class hierarchy of one data set and thus better semantic features, we further add the penalty in feature space by evaluating class-wise feature distance matrix D of original non- non-linear DNNs on CIFAR-I0. Given data pairs (ar,, Vi), their features Zi, we compute the class-wise feature centers Zj for class j following
Zj mean 2; ( = j)
F(a. b) represents the distance of class-wise feature centers Zu and Zz which is normalized with the largest class-wise distances for better visualization.
Zolla
F(a
Zn
2el22
c,cpa
Result Analysis We adopt the same basic setting as Sec. 3.2 and find a similar clustering effect on CIFAR-10 in Fig- The models are the same with those in Fig. Robust models consistently show the better clustering effect compared with non-robust models. That is, the centers of subclasses the same superclass lic closer to each other in feature space. The feature clustering effect also aligns well with class hierarchy. Such effect further confirms our understanding that robust models could extract more semantic and hierarchical features even in their original non-linear feature space.
Further Clustering Enhancement Given any unknown data set, we first extract its class hierarchy from pre-trained robust model, e.g., trained by adversarial training. To be specific, we compute matrices W and C following Sec. 3.2 We then formulate C' to an approximate matrix Cop with all items set to '-1" or '+1', i.e., in Cop. subclasses in the same superclass block have values '+1', else have values '-1', With the extracted class hierarchy, a regularization loss is designed to minimize outputs within the same superclass. Take the data set CIFAR- 10 for example, subclasses are divided into (,,1, and (2,3,4.5,6,7) in Cop. The regularization loss defined in Eq. I0. The hierarchical instance wise clustering training strategy is proposed following Algorithm
Fu = F,+(( [ 0, 1,8,9
Fi = E,F(,) 1(j 2,3,4,5,6,7
(10)
(x))(a) Σ [F(x); Foll2
Σ j=266,5,66,7
[F(x), Fill2.
Algorithm 2 Enhancing the Hierarchical Clustering Effect
Input:
A random input æ, pre-trained robust DNN model FF (e.g., CIFAR-10)
Output: Robustness Enhanced Model FF
Compute the weight correlation matrix C following Algorithm | and Eq.
Compute an approximate matrix Cop with all items '+1/-1' based on C and extract class hierarchy
Compute regularization loss Creg (z) with feature clusters following Eq. 10
with Lreg Retrain F from scratch using the original robust loss L(x, y) (e.g., adversarial training loss) added
return F
4.1 Domain Adaption Case
Recent study shows that robust models tend to perform better in downstream tasks, especially domain adaption with subpopulation shift 20 However, further analyses are insufficient. We study from two perspectives, 1) verifying how robust models outperform in domain adaption with different settings and 2) exploring a further improvement by enhancing the clustering effect. Specifically, we randomly divide a hierarchical data set into source data and target data according to their subclasses. That is, source and target data share the same superclass yet different subclasses, dubbed subpopulation shift, In the training phase, we use fine labels of source data and map fine labels to coarse ones. In such a fashion, the model returns fine and coarse labels. Only coarse label evaluated in the test phase as the target test data has different fine labels from source data. Both coarse and fine accuracies

---

(a) STL-AT
(b) CIFARIO-AT
(c) CIFAR20-AT
(d) ImageNetlO-AT
(e) STL-STD
(f) CIFARIO-STD
(g) CIFAR20-STD (h) ImageNet1O-STDD
Figure 4: Feature distance matrix F of ResNet-18 on different data sets. 'AT" indicates ad advessaiaal training. 'STD' indicates standard training. Robust models tend show feature clustering effect aligned with class hierarchy, which is similar to C.
evaluated source domain. We enhance the feature clustering effect by penalty loss defined in Eq. [0 To better utilize hierarchical features extracted by pre-trained models, we also finetune parameters on target data to achieve further improvement. Different models, different constructed data sets and different adversarial training settings (i.e. adversarial robustness levels) are evaluated and analyzed.
4.1.1 Different Models
Datasets and Setups The data set CIFAR-100 is composed of 20 superclasses, and each superclass composed of 5 subclasses. Based on that, we construct our own data sets by randomly sampling from CIFAR- 100, e.g., our default CIFAR-22 20 is composed of 20 subclasses from 4 superclasses. The details of all constructed data sets are shown Appendix A We adversarially train ResNet- 18, DenseNet-122 121 [17] and AlexNet 20 with PGD. If not specifically mentioned, PGD-5 is adopted (5 steps, step size 1/255, e=2/255). The robust/non-robust ResNet- 18 and DenseNet- 121 are trained with SGD 25 and initial learning rate 0.1. The robust/non-robus AlexNet
trained with SGD) and a initial learning rate 0.01. The learning rates decay with a factor of 0.1 at the 75 and 90 epoch. Other hyper-parameters
the same with Sec. 3.22 Moreover, finetune the pre-trained models on
target data for 20 epochs with a learning rate 0.01.
Results Analysis We evaluate the models on our default CIFAR-20. As shown in Table when comparing the coarse accuracies on target and source data, robust models tend to decline less than non-robust ones. Take pre-trained ResNet-18 18 for example, the coarse accuracy robust model directly declines by 5.44% (from 87.94% to 82.50%) but declines by 10. 10.18% (from 90.93% to 80.75%) on non-robust model. When considering finetuned models, the superiority of robust models outstands. For example, the coarse accuracy on finetuned robust ResNet-18 increases by 6.46% (from 87.94% 94.40%) while increases by 3.32% (from 90.93% 94.25%) finetuned non-robust ResNet- 18. When applying the clustering penalty defined in Eq. 10 coarse accuracy is improved by 2.25% on robust R8sNet-18 (from 82.50% 84.75%) even without finetuning. Experimental results demonstrate that features extracted by robust models have non non-trivial impact on domain adaptation. Such superiority keeps consistent when finetuning with the feature centers of target training data, Thus our understanding is confirmed that a better hierarchical classification learned by robust models benefits a lot in downstream domain adaption task.
4.1.2 Different Data Sets
Datasets and Setups For CIFAR- 100, we randomly select 4 to 6 superclasses, select 4 subclasses from each superclass for training and apply the left one for test. We also select 5 superclasses, select 3 subclasses from each superclass for training and apply the left 2 subclasses for test. When selecting 4 superclasses and subclasses, we typically design two extreme data set construction methods, i.e. the most different and similar data set. The 'different" 4 superclasses are selected index 1, 2, 5, 14 (fish, flowers, household electrical devices, people). The 'similar" 4 superclasses are selected as index 8, 11, 12, 16, which are all animals. The reconstructed data set ImageNet-20 is composed of
7

---

Table 2: Accuracy (%) CIFAR-20 across various models. 'Coarse' and 'Fine' indicate the ground- truth labels are coarse or fine respectively. 'FT' indicates the pre-trained models are finctuned on target data. 'R' indicates robust models (adversarial training). 'NR' indicates non-robust models (standard training). '+C" indicates robust models with an enhanced clustering effect defined in Eq.
10
Source Domain Fine
Target Domain
Coarse
Coarse
Coarse-FT
AlexNet
NR
85,50
65.37
71.75
86.70
R
84.75
64.50
72.75
87.50
R+C
90.68
65.56
75.25
89.25
R8sNet-18
NR
90.93
70.62
80.75
94.25
R
87.94
68.75
82.50
94.40
R+C
88.25
69.62
84.75
94.75
DenseNet-123 121
NR
92.31
73.56
77.25
95,50
R
90.18
72.87
81.00
95.75
R+C
91.37
73.87
85.25
96.50
00.5
15.5
Dage Minco
Thipe
Sours Accamoon
S0uce ACOUROOIM
Souic Acc 10777
(a) Different Data Sets
(b) Different CIFAR-4*4
Figure 5: Comparisons of target accuracy across different data sets. Different colors indicate different data set construction methods. Different shapes indicate different training methods, i.e. adversarial or standard training. Circle indicates non-robust model (NR), square and start indicate robust model without/with an enhanced clustering effect (R/R+C) respectively. Pentagon, diamond and triangle indicate finetuning NR/R/R+C models respectively. The legend in format A * B: A indicates the number of selected superclasses, B indicates the number of selected subclasses from each superclass for training. The points located in left and upper represent for better performances they have lower source accuracies yet higher target accuracies.
20 subclasses from 4 superclasses in TinylmageNet. The details conduct experiments ResNet-18 18 in this section.
of datasets are in Appendix A We
Results Analysis As shown Fig. 5 we find that experimental results on different data sets show different yet consistent performances: robust models show great superiority on those more hicrarchical data sets, which can be boosted with an enhanced clustering effect. In Fig. [5(a] our constructed data sets from CIFAR-100 and TinylmageNet show superiority with robustness and an enhanced clustering. Moreover, a better class hierarchy of data set enhances domain adaption tasks. For example, orange points (i.e. the 'different' data set) in both two subfigures in Fig. 5(b) tend to present lower source accuracies yet higher target accuracies, especially compared with lightseagreen points (i.e. the 'random' data set). For 'similar' data sets, because the class hierarchy is not obvious, target accuracy drops lot. However, the royalblue start point (i.e. 'R+C' version model trained on 'similar' data set) still achieves the highest target accuracies among three models trained 'similar' data set, We show the specific numerical results in Appendix
4.1.3 Different Robustness Settings
Datasets and Setups We evaluate 18 with different robustness settings (i.e. different PGDs in adversarial training) on random CIFAR-4x4 defined in Appendix The PGD details are given in Appendix B
Results Analysis As shown Fig. we could observe a trade-off between adversarial robustness and accuracies on target data. For example, in the polylines of both two subfigures, purple points show higher accuracies on target data than brown ones. However, brown points are of better robustness as their accuracies source data much lower. Royalblue points show the relatively bad performance
8

---

on target data yet the best performance on source data. The phenomena demonstrate trade-off between robustness and domain adaption performances. The better performance on target domain could only be achieved with both robustness and moderate accuracies on source data. Furthermore, robust model with an enhanced clustering effect has significantly results than the original robust model. We show the specific numerical results in Appendix In addition, the time costs results with different robustness settings are shown in Appendix D
750 115 738 arwaayy
753
733
Figure 6: Comparisons of target accuracy across different adversarial robustness levels (i.e. different adversarial training settings). Different colors indicate different PGD attack settings in adversarial training. Circle and square indicate robust model without/with an enhanced clustering effect (R/R+C), Pentagon and diamond indicate finetuning R/R+C model respectively. The points located in left and upper represent for better performances as they have lower source accuracies yet higher target accuracies.
4.2 Adversarial Case
We further conduct extensive experiments explore such intriguing clustering effect along with other adversarial phenomena. The experiments are designed from two perspectives, 1) adversarial robustness enhancement and 2) attack confusion matrix study. The results show that adversarial robustness could be improved by enhancing the hierarchical clustering effect, and attack success rates highly related to class hierarchy well.
4.2.1 Adversarial Robustness Enhancement
Datasets and Setups We adversarially train robust ResNet-18 CIFAR-10. The basic training setting is the same with Sec. 3.2 The PGD-20 attack applied with c = 8/255 and step size 0.003.
Results Analysis Although the main exploration of clustering effect is studied not only in robustness improvement, we conduct comprehensive robustness evaluations following the statc-of-the- attack strategy with adaptive AutoAttack [9] We find that the clustering enhancement actually boosts adversarial robustness in Table 3 for both best and last checkpoints. For black-box attacks, the surrogate model is standard trained VGG-16. A'tttack is conducted 1,000 random test data and with a maximum query 20,000. Note that, the improvement against Auto/Atack is 1.31% 1.3,% which shows that the clustering alignment with coarse labels learned by robust models could indeed boost adversarial robustness. An interesting phenomenon is that when enhancing clustering effect, both natural and robust accuracies are improved because the clustering effect improves feature representation. The natural accuracy by standard training improves as well (from 92.75% to 93.54 3445)) with clustering effect. Therefore, the trade-off between adversarial robustness and natural accuracy still exists. In addition, we count the training time of two models (AT and AT+C). The AT model costs 191.20s per epoch, while the AT+C model costs 193.66s. using one GPU 1080X with batch size 128 of ResNet-18 18 on CIFAR-10 10. The time cost slightly increases with the clustering regularization, which is negligible. More results are shown in Appendix E
Table 3: Robustness (accuracy (%) on various attacks) on CIFAR- 10, based on the best/last checkpoint of ResNet-18. 'AT' indicates adversarial training. '+C' indicates applying our hierarchical instance- wise clustering training strategy. The best results are boldfaced.
White-box
Black-box CW
Adaptive
Defense
Natural
FGSM
PGD-20
CW
PGD-20
NAttack
AutoAttack
Best Last
84,20
63.32
50.12
50.97
81.80
84.80
48.21
46.58
AT
84.27
60.46
46.50
48.97
79.13
79.87
45.05
41.90
Best
85.43 85.53
64.11
52.54
52.72
82.27
85.22
50.51
47.21
AT+C
Last
62.11
47.78
49.95
80.21
78.98
47.51
43.21

---

Smebbrice (b) UNTARGET-STD
UNTARGET-AT
TARGET-AT
(d) TARGET-STD
Figure Attack confusion matrix M under different settings. 'AT'STD'' indicate adversar-
ial/standard training. Robust models tend to show clustering effect aligned with class hierarchy.
4.2.2 Attack Confusion Matrix
To further analyze how class hierarchy matters in adversary, evaluate the attack confusion matrix M of adversarial examples in Fig. whose shape is Doutput x Doutput- M(, j) is defined as attack success rate (ASR) of adversarial examples being attacked from the ground-trun truth label i to the target label j.
Datasets and Setups We train robust and non-robust ResNet-18 on CIFAR-10. The basic setting is the same with Sec. 3.2 The PGD-20 attack is applied with 8/ 8/255 and step size 0.003. For target attack, target labels are set the other 9 classes except for the ground-truth one.
Results Analysis Experimental results show that the attack confusion matrix M performs in another blocking pattern especially on robust models, which is well-aligned with class hierarchy as well as weight correlation matrix and the feature distance matrix F. That is, the difficulty attacking robust models shows distinguishing performance for subclasses in or out of the same superclass. For example, against untarget PGD-20 attack in Fig. a). the ASRs on adversarially trained ResNet- 18 show that classes in (2,,,4,,,6,7) could decline to almost zero from one superclass (0.1,8,9) to the other (2,3,4,5,6,7). Different superclasses are harder to attack especially robust models, demonstrating that the hierarchical classification capacity is enhanced by adversarial robustness. Such phenomenon is also consistent with our hierarchical clustering observations on robust models.
easier to be attacked from one other. However, the ASRs
Conclusions
In this paper, we provide novel view of linear model exploration by extracting an implicit linear matrix expression. Then we surprisingly find an intriguing clustering effect adversarially robust models, which is well-aligned with class hicrarchy. Based on such observations, we first give an insightful explanation and understanding of robust models, which have a better capacity in extracting more representative features and semantic information. Extensive experiments show that enhancing the hicrarchical clustering effect not only improves model robustness, but also benefits other downstream domain adaption task. Overall, our observations and findings motivate a deeper understanding on adversarial robustness.
Acknowledgement
Yisen Wang partially supported by the National Natural Science Foundation China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund. Shu-Tao Xia is supported in part by the National Key Research and Development Program of China under Grant 2018YFB 1800204, the National Natural Science Foundation of China under Grant 62171248, the R&D Program of Shenzhen under Grant JCYJ201 80508 52204044.
References
[1] Motasem Alfarra, Juan C Pérez, Adel Bibi, Ali Thabet, Pablo Arbeláez, and Bernard Ghanem. Clustr: Clustering training for robustness. arXiv e-prints, 2020.
[2] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2019.
10

---

[3] Yang Bai, Yuyuan Zeng, Yong Jiang, Yisen Wang, Shu-Tao Xia, and Weiwei Guo. Improving query efficiency of black-box adversarial attack. In ECCV, 2020.
[4] Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustmess via channel-wise activation suppressing. In ICLR, 2021.
[5] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? 2018.
[6] Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint 1607.0411
[7] Moustapha Cissé, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness adversarial examples. In ICML, 2017.
[8] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks unsuper- vised feature learning. In Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, 2011.
[9] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. 2020.
[10] Nilaksh Das, Madhuri Shanbhogue, Shang- Tse Chen, Fred Hohman, Siwei Li, Li Chen, Michael E Kounavis, and Duen Horng Chau. Compression to the rescue: Defending from adversarial attacks across modalities. In KDD, 2018.
[11] Jacob Devlin, Ming-Wor Wiig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[12] Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normalization is a cause of adversarial vulnerability, 2019.
[13] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In ICLR, 2015.
[14] Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating linearly improves transferability of adversarial examples. 2020.
[15) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv: 1512.03385, 2015.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[18) Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In ECCV, 2018.
[19] Xiaojun Jia, Xingxing Wei, Xiaochun Cao, and Hassan. Foroosh. Comdefend: An efficient image compression model to defend adversarial examples. In CVPR, 2019.
[20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NewrIPS, 2012.
[21] Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. Nattack: Learning the distributions of adversarial examples for improved black-box attack on deep neural networks. 2019.
[22] Ziquan Liu, Yufei Cui, and Antoni B Chan. Improve generalization and robustness of neural networks via weight scale shifting invariant regularizations. arXiv preprint arXiv:2008. 02965, 2020.
11

---

[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. 2018.
[24] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In S&P, 2016.
[25] Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 15,,, 1999.
[26] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. Adversarial training generalizes data- dependent spectral norm regularization. 2019.
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. arXiv preprint arXiv: 1409.0575, 2014.
[28] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? arXiv preprint arXtv:2007.08489 2020.
[29] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopu-
lation shift. arXiv preprint arXiv:2008. 04859, 2020.
[30] Qi Tian, Kun Kuang, Kelu Jiang, Fei Wu, and Yisen Wang. Analysis and applications of class-wise robustness in adversarial training. In KDD, 2021.
[31] Yisen Wang, Xuejiao Deng, Songbai Pu, and Zhiheng Huang. Residual convolutional networks for automatic speech recognition. arXiv preprint 1702.0777 2017.
[32] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness adversarial training. In ICML, 2019.
[33] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[34] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
[35] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurlPS, 2020.
[36] Jia Xu, Yiming Li, Yong Jiang, and Shu-Tao Xia. Adversarial defense via local flatness regularization. In ICIP, 2020.
[37] Ziang Yan, Yiwen Guo, and Changshui Zhang. Deep defense: Training dons with improved adversarial robustness. 2018
[38] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy, 2019.
12


Reviews:
[{"review_id": "_h34CUHWCpT", "reviewer": "Reviewer_HkhD", "summary": "The paper proposed a novel method of linear model exploration by extracting their implicit linear matrix expression and find the weight clustering effect. The paper also claimed that the adversarial robustness shows close relationship with such clustering effect, which enables the further understanding and improvement of various tasks in the experimental parts. Overall, I like the analysis in the paper, it provides an interesting and reasonable way to understand adversarial robustness via linearity of weights.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The goal of the paper is to contribute to an explanation of the underlying mechanism of adversarial robustness in DNN, and to exploit this explanation to improve both robustness and domain adaptation. Thus, the subject of paper is clearly of interest for the adversarial community.\n- The general approach of the paper is to build two summary descriptors of the linear component of a DNN in the form of square matrices, each entry being associated with a pair of classes (Section 3). In the next section, these descriptors are exploited to create a regularizer for ordinary classification, and for domain adaptation tasks. The approach presented here is novel and convincing with a detailed explanation. \n- Section 4 provides a large number of good quality experiments. The penalty used for regularization is based on an inferred hierarchical clustering, itself build through an approximation of the “correlation” matrix. These experiments show moderate improvement. Other than adversarial robustness, the authors also applied the understanding of weight clustering on domain adaption study, showing the method has a general usage. \n\nWeakness:\n- In Algorithm 2, why do authors need to retrain F from scratch? Would the fine-tuning with regularization loss work?\n- The experiments on attacking confusion matrix is somehow confusing. Is the hierarchical classification enhanced or not? And is it enhanced by robustness, or is robustness enhanced by the use of the hierarchical classification penalty?\n- The explanations for figures 5 and 6 should be more clear. Currently, it is not so easy to follow when reading the figures. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see the above weakness.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "F6rZJHWPHG", "reviewer": "Reviewer_psEt", "summary": "This paper explores adversarial robustness via focusing on linear sub-networks. Through back propagation on linear sub-networks of a pre-trained network, an implicit weight expression which represents approximated class-wise responses can be obtained. With this matrix, the correlation among classes can be computed for exploring the hierarchy of classes. Similarly, the feature distance for different classes aligns well with class hierarchy through empirical observations. Based on these observations, the authors propose to impose feature distance regularization based on the computed class hierarchy.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n* This paper is well-organized and easy to follow.\n* The motivation seems reasonable and natural from sufficient observations. \n* Extensive experiments are conducted on various tasks and datasets. The comparison with other baselines show superiority. \n\nCons:\n\n* There exist some confusing descriptions. For example, the authors mention that labels (0,2,8,9) belongs to ‘Non-animal’ in Line 158, however, it seems labels (0,1,8,9) are actually highly- correlated in Figure 3(b).\n\n* The authors propose to extract a linear sub-network from a DNN through removing all the non-linear operations, and introduce a simple algorithm to obtain the weight expression of DNNs. However, the influence of this approximation is not discussed since most DNNs contains non-linear operations, such as ResNet-18 evaluated in experiments.\n\n* Lack of a large number of defense baselines, such TRADES, etc. in Table 1.\n\n* The approximate matrix C_{op} are binary. What if there exist a complicate class hierarchy in larger datasets, such as ImageNet?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DeqKD2OD7Nt", "reviewer": "Reviewer_aQh1", "summary": "This paper analyzes the statistical regularity of robust models and non-robust models on their linear sub-networks and show that the robust models tend to show clustering effect aligned with the class hierarchy. A plugged-in hierarchical clustering training strategy is proposed to enhance adversarial robustness.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The observations and findings are interesting, which gives me a lot of insights.\n\nMy main concerns:\n1. In Eq. 9, how did you compute the features z_i?\n2. [1][2] show that regularization terms can benefit AT. It is not clear your robustness improvement comes from the regularization term itself or the class hierarchy. Can you conduct more ablation studies to explain this issue?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations:\n1. The observations are based on CIFAR-10 dataset. The class hierarchy might not be clear on other datasets.\n2. The robustness improvement is limited.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper targets the problem of how to understand the mechanism of adversarial robustness, which is a hot research topic. The authors presented that adversarial robust models have a clustering effect. Specifically, robust models show a clear hierarchical clustering effect on their linear sub-networks. By exploiting the effect, the authors illustrated how to further boost adversarial robustness. We think the paper will interest many readers.
Decision: Accept (Spotlight)